An Ancient Argument

By Alec Meer on June 18th, 2010 at 12:22 am.

I’ve given myself 37 minutes to write this post in. You won’t learn anything new from it.

Kieron’s covered the problems with MMO reviews in a reliably masterful way, and there really isn’t much to be said beyond it. All I can do is repeat some of what others have touched on a thousand times previous, with different words at a different time, and with my own sense of awkwardness and upset. I don’t have an answer. That doesn’t make it any less relevant.

Today’s drama has been unpleasant, both personally and in terms of feeling really awful about how the two fundamental sides of the games criticism divide – the creators and the commentators – see each other. That’s (almost) all I’m going to say on that specific matter.

What I will offer is comment on why that kind of situation can ever develop, which unavoidably becomes the eternal bete noir of how games journalists can possibly review MMOs.

They can’t. They just can’t. Not to anyone’s satisfaction, at least. They have to please day-1 buyers, they have to please beta testers who pledged undying allegiance months ago, they have to please people typing “[game name] + review” into Google. They have to please people with £30 to spend, and please people with £300 to spend over a period of months or years.

That investment is why a company might ask that a reviewer give an online game more time, a chance to bed in and see how it takes shape once a community forms around it. That’s also why someone might turn up on a site on a Monday morning, wondering whether they’re about to drop their cash in the right place, then wander off when they don’t get their answer. Developers need fair representation, but customers need timely buying advice.

21 minutes.

Over the last year, there have been multiple cases of MMO reviews being pulled or re-done because server logs suggest enough time wasn’t given. There’s an awful lot of validity in that kind of response, because MMOs do genuinely require more time, and because most publications don’t provide enough expenses or time to do the job that’s required. Trouble is, publishers still want positive reviews up in that sales-critical release week – and positive reviews of an MMO can be exactly as short-sighted as negative reviews, so awkward compromises are reached. Whether positive or negative, such reviews look at the potential of what the community might bring to bear, because the actuality of it simply doesn’t exist yet. A prediction is never anything more than a prediction.

There is a trust issue, though. A presumption that journalists are looking for the easy route, that they won’t appreciate what a game could be and, unstated but implicit, that they just don’t understand. Unless they’ve played for whatever is deemed long enough, their experience and opinions can be seen as invalid, making judgment calls based upon what they do see isn’t good enough, and accusations will inevitably ring that nothing can be said or appreciated until hour or day X. If a judgment is issued before then, it’ll be called cruel, lazy and irrelevant. Such a response can be absolutely right – a lot of MMOs don’t come into their own until the high-level stuff’s experienced, and most of all until someone has built up and invested in both a character and a community. The trouble is presuming/deciding that that’s the case, yet expecting that a journalist on a short deadline and the same fee as reviewing an eight-hour nothingy shooter can somehow get to that point immediately. That’s why we get post-release embargoes.

16 minutes.

There have been enough mishaps and misjudgments to, perhaps, warrant this mistrust. Bad eggs exist in every field. But how do we earn that trust back, when the status quo remains snatching time on intermittent beta servers or private QA areas, cobbling together a sense of what might be rather than what is? The double-bind of doing an online game justice in the eyes of its creators and most passionate players, but also to provide useful buying advice to an audience on the day of a game’s release, remains. Post-release coverage can be arranged, if budgets and deadlines are flexible enough, but so much hangs on that week-one coverage, for consumers and creators alike.

9 minutes.

Of course journalists want to do a good job, and celebrate what an MMO does well. Mandating that they do it under specific conditions and time periods isn’t going to magically ensure that, though. There needs to be conversation, not accusatory server logs and absurd embargoes. Obviously that raises all manner of issues in and of itself, but in my experience, the question of how to review an MMO has never been adequately tackled by either side. Magazines and websites have deadlines, publishers and developers have server and population and privacy issues. So they butt heads and reach mutually uncomfortable compromises, and the net result is always that the game has to be reviewed by someone who hasn’t been able to have enough access to the game.

The most fascinating element of today’s fallout, to me, is that Euro journalists can now play APB’s beta on US servers from the 26th, two days earlier than previously. Why wasn’t that the case to start with? Because someone didn’t want it to be, and someone else didn’t want to pay for a reviewer to take that much time.

No-one wants to do a bad job. No-one wants to tear something down unduly. But I fear that, because of mistrust and limited access and painfully short deadlines, we’re headed towards a point where that’s the presumption. So instead of conversation, we get “you will do this.” How is that possibly going to lead to excitement and betterment?

1 minute. Perfect.

, .

93 Comments »

  1. Will says:

    This post is a travesty of justice, I looked at the server logs and it says he posted it at 12:22am and that’s just not good enough. If only Alec Meer had spent 38 minutes and 27 seconds writing then he would understand properly, but no-one can possibly write a post about MMO reviewing in only 37 minutes. I’m going to write an angry post on Eurogamer now, and they’ll make Kieron do a proper review which will say something similar.

    I burn with rage!

    • Will says:

      This comment is a travesty of justice, I see it was posted only 12 minutes after the article was posted. 12 minutes just isn’t enough time to formulate a good response, if only this ‘Will’ person had spent 13 + n minutes reading and thinking then he could have written a better response. I demand Kieron write a reply!

      Raagghhhh!

    • Chris D says:

      This reply to the first comment is a travesty of justice. Posted only 2 minutes after he did! You can’t possibly do it justice in that lenghth of time! Grrrrr!

    • Premium User Badge

      El_MUERkO says:

      All this cause of APB?!

      It’s simple, it sucks. Looks good in screenshots, looks a bit meh in motion and plays like a turd.

  2. Eightball says:

    Why I think reviews of MMOs will suck/Why I think reviews of MMOs won’t suck

    Alec Meer: Fair and Balanced

    :P

    Well written and thought out, though I’m not all that sympathetic to MMO devs simply because I don’t like the payment/play style. Or at least the pay for ones. Kingdom of Loathing was at least an entire year of high school year for me.

  3. Nick says:

    There doesn’t really seem to be a way to do a standard review of an MMO.. that said, the ‘oh it gets better when you hit max level’ excuse is unacceptable, an MMO should be good all the way up the ladder and then, if they want people to keep playing, have some sort of engaging “end game”, not mediocre grindy shit till you are allowed to “have fun” at max level.

    So, um, review free trial periods?

    Er. There is no answer.

  4. Fumarole says:

    This is why I like Wot I Thinks as opposed to score-based reviews.

    • The Great Wayne says:

      I second that.

    • Deston says:

      Thirdeded.

      Review scores are too inconsistent and high-level for me… I don’t care if they are there or not and I understand their purpose, but I’d rather read into the details. More importantly, it’s great to know how seasoned gamers like the RPS crew – who’s published thoughts I broadly trust to be balanced and insightful – perceive the experience as a whole.

      That’s wot I think about Wot I Think.

    • Easydog says:

      Fouthed.

  5. Metalfish says:

    The argument from MMO devs -about how you shouldn’t treat it like a standard game- falls down on one major point -you’re not just paying monthly, you pay a hefty price up front as well. If the game was free and you only paid the sub (EVE?), then they might have a point, as then the pricing reflects the continuing nature of the experience and should be treated as a long term experience. The up front price arguably means that the first few hours should be as good as any other big budget game, certainly not worse -because you paid extra for those.

    Yeah, it’s a weird argument, ain’t it?

    • karthik says:

      Wait, you have to buy MMOs (the game) before subscribing to MMOs (the service)?

      Wow. Having never played MMOs, I didn’t know this. This just makes it all the more unappealing.

    • Gorgeras says:

      From a position of ignorance Karthik, you’ve shown more wisdom than almost all long-time MMO players.

      When the first expansion for World of Warcraft was coming out, I objected very loudly on the WoW forums to it having a retail price-tag on the basis that our subscriptions had already paid for it. Subscriptions are there not just to support the infrastructure of the MMO(which is almost always under-utilised in the actual game design, which relies excessively on instancing), but the development of further content. WoW had a few months at the start where new content was coming out almost every month, then it suddenly went into snail-crawl. I figured that the resources being spent on the expansion wasn’t being spent on the additional subscription-funded content.

      Gameplay-wise, WoW has improved for me and the worst of the idiots on the design team are no longer in charge. But I’ve yet to see any justification for the additional charges in WoW.

    • cliffski says:

      I think you have hit on a damned good point there. Also, I’m sure why special slack has to be reserved for the people making an MMO. My time as a gamer is not worth less in the first 10 hours of an MMO than when I play Just Cause or World of Goo.
      My last game just got its 42nd patch yesterday. it is massively hugely changed from when it was first released and reviewed (October 09). But as a singleplayer, non-MMO game, it will not get re-reviewed at a later stage.

      That is just life, and MMO developers need to accept the fact that first imrpessions count, and make a good first impression. Singleplayer developers don’t get to whine that the game will be better in a few months, and MMO developers don’t deserve a free pass on that topic either.

      The same goes for hours played. If a game sucks for the first few hours, thats the designers fault, and the developers fault. Above all, journalists need to be honest about their imrpessions of the game. Anything else is very dangerous.

    • WildcardUK says:

      Karthik, seems like double charging doesn’t it? This is why I respect CCP for how they run EVE. ‘Free’ expansions should be the norm as far as I’m concerned.

      Bang on Cliffski! If anyone expects me to endure 10 hours of crap to get to the point where ‘it gets good’ then they can forget it. My time is better spent on a game that’s good from start to finish. I actually think there is merit to an MMO review done based on a short span of play for this reason.

  6. fnsmatt says:

    If you have to play a game for 20 or more hours before it “gets good” it deserves poor initial reviews. Mainstream game reviews should be aimed at the people who show a passing interest in a game but aren’t sure if they want to purchase it or not, not people who are planning to devote their lives to a game to get to the “good” content. Those people have community websites to help them determine if a game is worth their while.

    • iainl says:

      This.

      I’ve said it before, regarding single-player RPGs, but the same goes for MMO: if your game has a crap first 10 hours, it’s a crap game. I don’t care how good it gets 20 hours in, or whatever, because I’ll never see that far.

    • jonfitt says:

      Yup. Saying a MMO is only good when you invest a large chunk of you life in them tells me they’re not good games. Why do MMOs get this free pass?

      Take a multiplayer shooter like Battlefield 2. It is a great game, and then when you later get involved in the community and play competitively it is even better. But because it was free to play online, no one even thinks of considering the long term aspects in a review.

      You’re buying a full priced game and then paying for long term play later. The review that comes out on release should be based on immediate impressions: Are people going to want to play it for the time you expect to play a game you paid $X for?

      The question then of “has the game enough high end content to keep someone playing indefinitely” can easily later be answered by the player themselves.

      Are we expecting the reviewers to be able to divine: “The high end content in this game will definitely keep a critical mass of players playing for a long time”, because if so we are crazy. That can never be anything but a bullshit opinion. You’d be better off predicting the lottery.

  7. 7 Seas says:

    What I don’t understand is why gaming review sites have no adapted to the form that is the MMO.

    You say:


    They can’t. They just can’t. Not to anyone’s satisfaction, at least. They have to please day-1 buyers, they have to please beta testers who pledged undying allegiance months ago, they have to please people typing “[game name] + review” into Google. They have to please people with £30 to spend, and please people with £300 to spend over a period of months or years

    And I agree. MMO’s should be reviewed in the same way a television show should be reviewed. You review the premiere, which satisfies the day-1 buyers. You dip back in a month or two later, and review again. You dip back in 6 months after that and provide an update on the state of the game, and so on. The up to date reviews please the people who google game review, the people who think about returning to the game, the people with £300 who want to know whether the game is worth joining.

    Surely game reviewers have realized that their week 1 review of an MMO is essentially worthless just a few months down the line, let alone a year or two later. The MMO is an evolving beast that usually hatches ugly and unable to properly fend for itself in the world, but with the loving care of its developers and the godlike patience of its players can reach tremendous hights.

    I find this applies not just to MMOs, but also other games that require some love. Most day 1 reviews of games like Vampire: The Masquerade – Bloodlines, games which were in severe need of patch love, rightly trash the game for being a buggy mess. And that satisfies those day-1 buyers. However, that review becomes more and more irrelevant as time goes on, and yet it still stands, through the magic of the internet, years later, shining condemnation down upon a game which maybe be well worth playing. These games make top 10 lists or top 100 nostalgia lists years later, despite never otherwise being acknowledged by the journalist. It becomes the open secret that games like Stalker, Vampire Bloodlines, Deus Ex, and many more (Alpha Protocol among them!) are actually classics that just needed a bit more time to breathe. And yet this is not officially acknowledged via the reviewing medium, the original, now irrelevant reviews still stand.

    I’ve called for this before, and I’ll call for it again. Game reviewers *should* try to pass Judgement upon games, reflecting the full value and place of the game in the ecosystem of gaming. If that requires that reviews be updated, games be revieviewed, MMO’s be reanalyzed, or whatever, then so be it.

    Gaming, unlike film or books, is an oddly fluid medium, where mistakes and problems may be corrected long after the die has been cast. I think gaming journalism should reflect this.

    • Chris D says:

      Kieron covers this in more detail in his Darkfall review. (which is an excellent piece of writing) Essentially it’s just not cost effective to spend so much time reviewing a game if you get paid the same compared to say an FPS which you can blast thorugh much more quickly. If you pay more for MMO reviews then you can only afford to review the big games. If you only review games you want to spend that much time with then it’s a self selecting bias.

      There’s a place for analysing a games place in gaming history but the more pressing question for most people is “Should I spend my thirty quid on this game that just released or get something else?”

    • Archonsod says:

      I don’t think it’s up to the reviewer to adapt to the MMO format. If the game sucks on release then they should be saying it sucks, I don’t want to spend thirty quid on a game which might be worth buying in six months time, particularly not if it’s going to be fifteen quid in six months time. Similarly, I have no interest in wasting two or three months worth of a subscription to grind my way up to the point where the game is actually worth playing; I’m sure I could spend the money on three games which were worth playing from the get go instead.

      Surely it’s up to the publisher or developer to encourage the reviewers to come back and take a look when they think they’ve fixed everything complained about the first time. WoW doesn’t get as much coverage solely because it’s popular, Actard also have a superb marketing department which means if someone so much as farts in Azeroth they’re phoning everyone up to let them know. If they think the game is even better six months after the initial release then get the marketing guys to go out and create some buzz about it; if you can generate the marketing buzz then you’re giving the reviewers a reason to come back and take a second look; whether a website or magazine they’re going to cover what they think will draw in readers, if you’re hyping the game it’s going to draw interest. Thus you get a nice self – fulfilling cycle.

      Finally there’s a couple of fatal flaws with the idea of reviewing it like a TV series. As I said above, you’re basically making an initial review pointless, since all they can really say is “this sucks/doesn’t suck, but we’ll give it six months”. And like I said, most people read reviews to see if the game is worth buying there and then, not to see if the reviewer thinks it’s worth picking up in six months time. Secondly, if the devs cannot get it right after years in development, what makes you think they can turn it around in mere months after a release? Finally, there’s a word for a product released to the public that you intend to improve over the coming months. Beta. See, it’s another easy way for the MMO makers to avoid those initial bad reviews – don’t release the game. Call it a paid for beta, an early adopter trial or whatever. When you think the game is ready to stand critical scrutiny, then call it the release. The added advantage is of course that it’s a bit more honest about what you’re offering.

    • Harlander says:

      These games make top 10 lists or top 100 nostalgia lists years later, despite never otherwise being acknowledged by the journalist. It becomes the open secret that games like Stalker, Vampire Bloodlines, Deus Ex, and many more (Alpha Protocol among them!) are actually classics that just needed a bit more time to breathe.

      While what you say is true, do we really want reviews to encourage the “release something incomplete and buggy then patch it over time” mindset?

    • Premium User Badge

      Kelron says:

      In the specific case of APB, it’s definitely not a game that needs weeks to review. If you’re not enjoying the first 10 hours, you’re unlikely to enjoy the next 50. In fact, long term reviews might even work against it if you enjoy it initially (as I did) before the repetition sets in.

      In general, even for games like EVE and Darkfall that everyone will tell you are slow to start, I still don’t agree that huge amounts of time are needed to review the game. EVE had me mesmerised from the start – I didn’t know what I was doing, I didn’t know where I was going, but I wandered around the galaxy doing whatever I felt like. Even at the “clueless noob” stage I could see it was a huge, deep game that was going to keep me occupied for a long time.

      I’m playing the trial of Darkfall now, and it’s similar. It’s flawed, it’s slow, it’s hard, but I can see potential there, and I want to keep playing to learn its mysteries. Of course, I came to both of these games some time after release, and I might not have enjoyed them at all when they were first reviewed.

  8. Jambo says:

    Seems like the best way to go about it is to publish 2/3 reviews … a ‘first impressions’ review when it first comes out, then another one after a week or so, then maybe another after a month … then average out the scores

    • Jake says:

      You’d still have to spend a considerable amount of time playing the game to get to the second review – maybe several hours a day. And then in a game like Warcraft you could quite easily end up reviewing each major patch separately as they contain so much content in terms of hours you have to spend to see it all.

    • Jockie says:

      Eurogamer do re-reviews of MMO’s after they’ve been released for some time occasionally and it’s a good idea. But even with that MMO’s have never fit easily on the 10 point scale, I mean do you award points for potential, for early content (encouraging devs to make 20 levels of awesomeness then 60 of grind like AOC), or for community etc etc.

      I think because MMO’s cost more for the player, a website can more easily justify spending more on getting a journalist to review it and perhaps re-review, though not for every major content patch, because the more regularly you focus on it, the less interested the majority of the readership will be. By which I mean, a game comes out gets 7 out of 10, if in 6 months time there’s another review and it gives it an 8 or a 9, then people might be interested in the re-review. If you’re reviewing a game every month or two, on a non specialist MMO site, the casual audience of the website, isn’t going to want to read about the same game on such a regular basis.

    • Premium User Badge

      Arathain says:

      @ Jambo:

      It doesn’t overcome the fundamental problem that causes the primary conflict between the publishers and the reviewers, though. MMOs typically depend on a strong start for survival, and a strong start typically depends on good early reviews. After the first couple of weeks, further reviews will attract relatively small numbers of people to the game, even if they’re positive.

      There’s also the issue Kieron mentioned, with reviewers having to get paid enough for their copy to be able to eat and pay rent, and gaming sites having a limited budget. Super lengthy playtimes or multiple re-reviews of all but the biggest titles are a pure luxury- not cost effective.

  9. sink257 says:

    I see what you did there :D

  10. Phill Cameron says:

    I feel the only logical way to do this is with a pair of reviews. As Kieron suggested; one at release, with initial impressions, and then one 6 months to a year down the line that has a fair assessment of the community and how well it’s being supported. It’s the only way to a) be fair to the game, b) be fair to the journalists reviewing the game, c) be fair to the readers who want to know whether it’s worth it, and d) be fair to those comissioning the reviews, who can’t afford to pay huge amounts for a single review that doesn’t pull significantly (if any) more views than review of Shooter X.

    So yeah, that.

  11. littlewilly91 says:

    Couldn’t there be an arrangement made where the review that is released in that first week clearly states that it is not the ultimate one, instead describing what they can thus far and how they plan to flesh out the site’s review after that-
    Adding onto the end of the review next week, then two weeks after that, then a new review (since the old one was getting too long) a month after that, and then on the anniversaries of it’s release. Adjusting the score if necessary. Perhaps giving multiple scores aswell as the main one, for separate sections of the audience?

    Could also have another journalist on the sidelines who knows how to play and steps in to the same account when the other guy gets burnt out.

    This is obviously a massive workload so maybe only the bigger sites should do them, & RPS might review the odd one if someone has played a few months in it incidentally. + Wot i Thinks and Hiveminds of course.

  12. littlewilly91 says:

    o jeez i should read other comments innit

  13. Telke says:

    Karthik: for any ‘AAA’ MMOs, yes, you pay the upfront “buying the box” cost, even with digital downloads. Usually they cost a bit less than a normal game, and keep in mind that cost also gets you your first month of play. after that, subscriptions.

    -> Worth noting is that APB is giving players “50 hours gametime” rather than “access to the servers for 30 days following your account creation” which is an interesting mechanic.

    A large number of the non-AAA mmos have no startup cost, though.

  14. Vinraith says:

    It’s funny how much this, to me, parallels reviews of wargames and niche grand strategy games. Like MMO’s, there are games that it can take dozens, even hundreds of hours to get the most out of. Like MMO’s, their scale and complexity lends itself to buggy and problematic releases which are (in the best cases) fixed over the long haul through feedback from the community. Most of all, they tend to be reviewed by people that can’t give them the time they deserve at the moment in their life cycle that they’re most broken, so they frequently aren’t given a “fair” shake by the gaming press.

    I don’t know what the solution to this problem is in the case of MMO’s or wargames, but it’s a problem that goes back well before MMO’s were even possible.

  15. Jockie says:

    I like the Wot I Think, not so much because it does away with scores, but because it’s an unashamedly subjective single persons view of a game. Games recently like Alpha Protocol got reviews from all across the scale, because different reviewers were looking for different things in the game (some seemingly measuring it against a non-existant imaginary game in their heads about spies). With Wot I thinks we know the taste in games of the person writing it (Jim likes to shoot, John likes to cry, etc) and there’s something less authoritive (in a good way) about it than a review that’s representing a website, more than the person writing it.

    My only answer for MMOs would be to ignore the extremes (beta diehards – which in fairness I probably am for APB – the people with seemingly hate filled grudges, fanboys etc) and focus on some kind of approximate medium of a website or magazines audience for reviewing purposes. Also all MMO’s should have free trials, because then people can decide for themselves.

    • Web Cole says:

      Haha “John likes to cry”. Poor John, you’ve probably just gone and made him cry! But then… he likes to cry, so I suppose…

  16. Eamo says:

    The thing is, the MMO developers are being unfair when they publish the time played just for the negative reviews. If they were being fair they would publish the time played on all reviewer accounts, then people could see the reviewers who gave 90% despite only spending two hours in the game as well as those that gave 70% after 10 hours of play.

    I also think there is an inherent bias in using the time as some measure of effort as no reviewer is going to want to spend 50 hours playing a crap game. It is strange that nobody actually points that out, “wow, you got a free copy of this game and only played it for two hours”. The developers are certainly using this as a stick to beat the reviewers I believe. The very nature of MMOs is that they will always be able to find things the reviewer didn’t experience. They can always make the “but he didn’t see the good bits” argument.

  17. Josh says:

    I third that. Assigning an arbitrary number to a game, especially an MMO is absurd.

    • cjlr says:

      @Josh
      … But these go to eleven. It’s one higher, innit?

      I don’t think a number is an inherently bad way of rating things, at least as far as it’s quite useful for quickly and roughly comparing things – if the same reviewer gives A a 8/10 and B a 4/10 you know he liked A better. Obviously this is far from a complete assessment.

      The problem comes when people begin treating the numbers as if the source didn’t matter and as if the numerical score was the be all and end all of the review process. Because that is absurd.

  18. Premium User Badge

    drewski says:

    MMO’s are quite a bit like new TV series, really. You often can’t judge them until a fair way into the first season, sometimes even later than that, and a lot of the time they gradually improve over time as they get used to themselves.

  19. Bassism says:

    Yeah, it’s a pretty tough question.
    Personally I’d just like everybody to move to the Wot format. As long as a reviewer is upfront about how he played the game, and reports what he saw, that should suffice. I mean, what I’m looking for in an MMO review is to get an idea of what the broad scope of the game is, and what to expect as I progress. It’s certainly possible for a reviewer to get a handle on what he’s played, mention some of the things that happen in the end game (things you can learn by watching/talking to others/etc), and describe what he saw while playing.
    A review like that would let me know whether it’s worth shelling out some cash to play myself and make my own decisions on what to do.

    At the same time, 2 hours isn’t enough time to review an MMO. It’s not enough time to review any game. How long is a AAA shooter these days? 10 hours? Have your reviewer play it for that long. Especially in the case of big name MMOs where you’re paying 50 bucks to buy the game in the first place, if the reviewer doesn’t find things to like in 10 hours, something is wrong.

    At the same time devs/publishers need to grow some balls and accept that not every reviewer is going to fall in love with the game. Sure, the diehard fans of a game may disagree with the content of a review, but they’re not the audience for 0day reviews. A review written from the point of view of somebody new to the game, for people new to the game, that finds that the game isn’t much fun for people new to the game is a good thing. Note that the full disclosure thing is important here. Be clear that you only played x hours, and you met lots of level 80 players that loved the game because of y.

    If I made an MMO, released it, and had a swarm of bad reviews, while having a diehard high level fanbase, the first thing I would do is think about how to fix the game. Not blame reviewers for being unfair.

    I think that everybody could get along if everybody gave a little. Reviewers’ interests conflict directly with devs’, neither of which really have the consumer in mind. Nobody is right, so nobody should have their way.

    But if everybody is honest and open, then we could avoid things like watching devs silence the media based on fear of what they might write. If thins like this stopped happening, everybody would win.

  20. Anonymousity says:

    If you play for 8 hours then trash the game that is fair, if in 8 hours of doing something that is supposed to be fun you aren’t having fun, it’s not something worth doing. I wouldn’t watch 8 hours of movies that I didn’t enjoy, why should I play 8 hours of a game I don’t enjoy.

  21. Gabe says:

    i·ron·ic (-rnk)
    adj.
    1. Characterized by or constituting irony.
    2. Given to the use of irony. See Synonyms at sarcastic.
    3. (Stylised) usage of a famous Austalian landmark in a game
    that can’t be played in that country, due to the publisher refusing to set up a local server.

    • Malibu Stacey says:

      When wholesale bandwidth stops costing over 1000 (yes that’s one thousand) times what it does in Europe & North America, then you might have a point.

      Unless you’d be happy with paying 1000 times the subscription cost that European & North American players do. Hell lets call it 100 times just to give us something to work with. What is 1000 USD/800 UKP/900 EUR in Australian dollars these days? Happy to pay that per month? Didn’t think so.

  22. faelnor says:

    here’s my review to every MMO ever, and it’s clearly universal & objective : don’t buy

  23. TenjouUtena says:

    I notice, within the comments at Kieron’s review, that the DF apologists are sort of moving the goalposts on our beloved KG. The first review isn’t good enough because he didn’t spend enough time playing. But KG’s review is wrong because the game is so much better _now_, one month after KG’s playtime. Oh it’s patched it’s all better. There’s even a late comment there that’s like ‘It’s even better now.’ So, maybe, MMO developers should release complete games when they release. Maybe if they’re weren’t trying to play this game of monetizing what is essentially a beta, they would get better reviews.

    Also, what is waiting one week going to prove? Many people will probably only have roughly 10 – 15 hours of gameplay under their belt at that time, which MMO devs are claiming is clearly not enough. I don’t think any games writer is going to have the leisure of 40, let alone 60 or 80 hours of gameplay after a week, or even two. In my opinion this just smacks of ‘Let the wave of hype move boxes for a week.’

    Perhaps what MMO devs should do is arrange for a ride along for reviewers. A reviewer can (and should) play the game for a while. Then maybe they are given access to a high level character? Or maybe a dev or a test shows them around the world? I realize this might sully the purity of the review process somewhat

    I have to wonder if this ‘Oh the reviewer played only for X hours’ is some sort of reinforcement of conformational bias in their existing audience. Perhaps MMO devs are afraid that the existing audience will be convince that yeah, it’s not that much fun. But if they can tag the reviewer as an outsider.. One of THEM, you know what THEY say, those (carebears / non players / casual sorts / etc. ) then the userbase’s conformational bias (believing something that agrees with the views you already hold) goes ‘Yeah, what does that guy know anyway.’ Seemed to work for Darkfall’s userbase, anyway.

  24. Adventurous Putty says:

    Wait, what drama?

  25. Tim James says:

    RPS, just a friendly heads up that you might be getting too self-indulgent with the MMO review shit and XCOM defense and other nonsense. Let’s hear about more E3 games!

    • TeeJay says:

      This.

      …or just start doing loads of 10-minute articles about cute/silly indie flsh games please.

      (I’d imagine that most RPS readers who give a damn about APB have already been playing i for themselves. Those who don’t give a damn would prefer to hear about something else).

    • Rich says:

      That’s if there is actually any interesting PC news coming out of E3. So far it all looks pretty consoley.

    • Web Cole says:

      Self indulgent? Nah, this is dialogue and debate, and well reasoned at that. It’s interesting, and it’s sure as hell relevant to PC gaming as a whole. And it’s not like RPS doesn’t do this sort of stuff occasionally, and why the hell not? Who else is going to do it?

  26. malkav11 says:

    I can think of an easy solution. There would be no need to formally review MMOs if the MMOs would simply offer a free trial period of reasonable length starting from launch onwards. As opposed to maybe getting around to it a year or two down the line when boxed copies go for 5 dollars and you don’t really need the trial anymore. At that point, the critics can certainly weigh in with their impressions, but more importantly, you can see for yourself if it appeals enough to make that critical early purchase.

  27. Premium User Badge

    Tunips says:

    Possible solution: Don’t charge upfront for MMOs. Make the client and the first month free (or cheap) for everyone. This way the review doesn’t need to tell the reader whether it’s worth six months and $500 (or what-have-you). It can instead run along the lines of “It’s not fundamentally broken and has dragons/steamengines/skateboards. If you like that stuff give it a go”
    That’s an amount of investigation that can be fairly fit in 8 hours, and reduces the make-or-break aspect of reviews that is causing trouble. (Of course that runs contrary to journalists’ mad lust for power and millions of corrupt dollars)
    Now this would require an adjustment of the business model, or at least the projections of time vs revenue. I don’t know how important to an MMOs success the money from first week big box sales is. I rather suspect that if the game really is going to take off that it becomes less relevant. More important for mitigating the cost of games that fail.
    We’ve seen a number of traditional MMOs go free, and I know I’m tempted infinitely more by a free trial than a pricey box. If the business can survive doing that at launch, then we can get fair reviews, easy access to our own opinions and presumably much higher playerspikes when something interesting happens.

  28. Vandell says:

    MMOs are games like any other, and don’t deserve special treatment. In many MMOs players constantly say “the end game’s where it’s at!”. If that’s the case why the boring grind to reach it? That’s bad game design, and if you can’t entertain a player even a LITTLE after the first few hours, there’s something wrong with your game.

    • FRIENDLYUNIT says:

      I agree. The point that people have to at first play the thing at the early stages, just like the reviewer does, is a valid one. I mean, if you’ve played your share of games you can pick up the premise of a thing pretty quickly yeah?

      I dont normally say things like this (at all), but if only the high end stuff is worth playing why not just publish that?

      Also, and don’t most reviews stipulate what parts of the game and what levels they’ve played up to? I think your average review reader is going to understand that.

    • Koozer says:

      It depends what kind of person you are really. Personally I MUCH prefer levelling a character, going through all the little stories on the way and travelling across different areas. In WoW for example once you hit 80 you spend all day standing around the same city in queues.

  29. Freud says:

    With sites that compile review scores (Metacritic, Gamerankings etc) becoming more and more important it is all about the score. Not what the reviewer actually thinks of the game. So going back for a second opinion doesn’t matter as much as the first review does.

    I think the best way to handle it is to have a review close to when the game is launched. The review can partly be based on playing the beta if the beta is very close to launch. It is not ideal but a compromise that is necessary for the review to be timely and serve as consumer guidance. Besides reviews of single player games are done on review code quite often.

    I do think the reviewer should revisit the game after 3-6 months and see what happens and basically go back and add that to the review. Then the review score is adjusted as the average of the first and second review scores.

    Realistically this is only going to happen in the case of bigger MMOs but I think it would serve everyone if it was done. We all know that a MMO during launch frenzy is one thing and might be a completely different thing (for better or for worse) after a few patches and when many players moved on. How a game handles empty servers and how good they are at adding content matters quite a bit when it comes to MMOs.

  30. PjV says:

    You have to buy a cell phone and then pay for the calling plan… This buy and then subscribe pattern repeats in many areas (like cable tv as another example) so it isn’t so odd to have it in games as well.

    • Rich says:

      Phones and TVs have a unit cost, incurred by a separate company to the service provider.

  31. Bart Stewart says:

    Vandell, I disagree.

    MMORPGs have the same problem that science fiction novels have. A good SF novel has to be first a good novel, but it additionally has to be good science fiction as well. Similarly, a MMORPG has to launch as a good game… but then it *also* has to grow into a good massively multiplayer experience via the cultural norms that emerge through the interaction over time of many players.

    To my mind, that means MMORPGs can and should be reviewed as games, but in fairness to what they’re trying to accomplish they also need to be reviewed as massively multiplayer experiences. And with respect to that latter criterion, developers/publishers are correct: you can’t review the “complete” MMORPG on Day One because it doesn’t exist yet — the players haven’t created the emergent content that makes a MMORPG what it is.

    Were I running a game reviewing outfit, I think I’d institute a rule that MMORPGs, as the unique things they are, get two reviews — one on Day One, and another full review (say) six months later. The Day One review would explicitly speak only to the developer-provided mechanics and would include a caveat that the rest of the game’s content — the emergent stuff that distinguishes MMORPGs from other kinds of games — should not be pre-judged and would be reviewed later.

    And then the six-month review should review the gameworld exactly as it is.

    If every game review site took that two-review approach to MMORPGs, I don’t see that game developers/publishers would have any valid reason to complain or try to impose post hoc embargoes on reviews.

  32. Tei says:

    Maybe MMO reviews sould have a subttitle
    “The 3 first hours with Darkfall” (if you are reviewing darkfall)
    And a note on the botton (this review only shows the experience of the first 3 hours. Is also the subjetive opinion of the reviewer. The dude he is a cool cat, but your opinion of the game could be different )

    I think this could be the journalist-ish way to ‘solve’ the problem. You will still get angry people wen you make poor reviews, but that people can go to hell (where reviews are always positive, but games are horrible).

  33. Marco W. says:

    Start-up MMOs NEED to know how it works. I´am in very rage, when i see nonsense account management progress (NA Keys selling to EU, with no description and non regional constriction), support they need 6 (!!) days for a automated message, without to look into. etc. pp.

    Flagship Studios don´t was the first ship, they sink on competitive reasons.

    PS: Don´t believe the hype .. ;)

  34. Orange says:

    Problem with a first impressions is that you get mmos like Conan which polished the heck out of their starting area and first 10-20 hours, then the rest of the game had massive holes in it.

    Generally as player you are paying for one whole month of playing, so the first few hours or even 20 hours doesn’t show what you will get for your money.

    I think in this case reviewing falls to the players, if you’re interested in an mmo try the beta and scrawl the forums. Maybe an mmo review should in large part be interviewing current and ex players and surveying the forums, rather than one man playing a game and writing about his sole experience.

    • Archonsod says:

      Yeah, but the thing is you can always cancel the subscription. If I bought Conan, enjoyed those 20 hours and then decided the later game wasn’t for me I can cancel the subscription, or even not bother taking one out if it’s a month free play or whatever, and I’ve still had reasonably good entertainment for my cash.
      If on the other hand I didn’t enjoy those 20 hours, and persisted because everyone said the late game was better and it still turns out to be crap, I’ve wasted all the time and money expended to reach the late game. And I’m not going to be kindly disposed to those who insisted it got better …

    • tomsoul.grind@gmail.com says:

      @Archonsod:
      Which is why I think it’s fine for new MMO reviews to be based on only the first few hours. (more than 3hrs, clearly). That’s what most new buyers will be experiencing, and that’s what they’ll have paid for. That’s what needs to be ok out of the box.

      Many players won’t get to the mid/late level stuff for a long time, so by that time they’ll
      (a) already know if they love the game, or the community enough to stay.
      (b) not really need a review to know if they want to continue their subscription.

      The few hardcore MMO players who’ll burn through the content and max out their characters within weeks – they aren’t really who reviews are for, and they probably won’t need one.

      It does then make sense to come back at a later date and update any big changes, and maybe mention the later game, but it’s not really needed for a launch review.

  35. tomsoul.grind@gmail.com says:

    The way I see it the reviewer should be reviewing what the player will experience. That seems to imply about 3 separate reviews, based on level.

    A starting review, around launch time, that covers the first 1-10 levels (for example), with a playtime of about a single player review.
    A follow up review a few months later that covers levels 11-30, with equivalent playtime.
    A follow up review a few months later that covers 31-50 (max) with equivalent playtime.

    The problem would be getting reviewers for parts 2 and 3. It’d probably have to be the same reviewer for all 3 parts, so a new guy wasn’t having to play through all the previous parts. The website/mag would just have to have a 3-review deal with the reviewer that they’d deliver the parts over a prolonged period.

    That’d be a more interesting review anyway, as it’d be following the experience from newbie to veteran, and cover any patches or improvements in that time. It’d be more like the multi-part commentaries RPS sometimes does (which i love).

  36. mihor_fego says:

    The biggest problem with reviews in general is the public itself. People that will just skip to the score at the end shouldn’t define how gaming journalism works. Parallel that to music reviewing for a minute: People that would buy (or pirate) any big-name mainstream release are of the same mindset with those that would buy AAA titles without even playing a demo or reading any reviews.
    Opposite that are those that (perhaps) give a bit more thought into their entertainment; they have developed personal tastes over the years and know what suits them. By reading a large number of reviews, one can finally ascertain which journalists share the same tastes concerning the music or games. In dealing with musical reviews, I had eventually sorted out some journalists whose reviews I only read out of interest, as their score would reflect how I would judge the subject myself.
    Perhaps many would object by supporting the notion that there’s too many different voices over the web and this would be a time-consuming process; on the contrary, I believe this is an excuse out of laziness. As an example, most people reading RPS devote a lot of time gaming but also enjoy the meta-game discussions and gaming culture.
    Of course, a gaming blog and a game review website ain’t the same. Sites like Eurogamer, Gamespot or GamesRadar all try to appeal to more casual gamers. And by casual I don’t mean those that play Farmville, but those that would buy any glossy and over-advertised release without second thought. Well, to put it bluntly, journalists shouldn’t care to appeal to them as much as they shouldn’t have to please the publishers.
    For MMOs especially, no-one should ever dictate how reviewers will do their job. If a journalist I trust that shares my tastes trashes a game even after 4 hours, I’m pretty sure I’d have the same appeal to me. It’s unfair for reviewers to expect them to spend a month in a game to check out the end-game content if they hate it in the first place.
    If publishers were really proud of their final products, they could bypass those “lazy reviewers” by giving out 15-day free trials for their games. Who wouldn’t be able to form his/her own opinion within such time? Especially for subscription-based MMOs, if the game is good enough, it should be able to hook its target audience into buying and expanding their subscription. Oh, I forgot, they don’t really care for the end product as long as they get a big opening sale…

  37. Octaeder says:

    @Tei: I think from now on all reviews should include a note telling the reader is the reviewer is a cool cat or not.

  38. Octaeder says:

    @Tei: I think from now on all reviews should include a not telling the reader if the reviewer is a cool cat or not.

    • Tei says:

      I dunno. I hate wen people feel the need to announce what is (Really) the default. Like water is wet, fire burns, and the clouds are white. But the world is not perfect, so you may end needing to explain things that normally people sould understand.

      Also, I am open to be hired by MMO companies to review his games in a fair way, at 60$/hour. I have not problem playing his game for a week before making a ultimate decission, and I am sure I will find in that week enough fun to fill 2 pages of text.

  39. Mr Gray says:

    I’d just like to thank RPS for covering this, seeing as no one else has.

  40. Premium User Badge

    Sagan says:

    I don’t think reviewing MMO’s would be a problem if the reviewer admitted that this is merely about the first ten hours or so. He should explain everything that the MMO aims to do, then he should say that he could only play it for ten hours for the review, and then he should write about those first ten hours. If MMO reviews were done like that, then every WoW-clone ever (and WoW itself) would have gotten the 4/10 they deserve. Seriously, who thought it was a good idea to send me on quests to kill ten wolves or collect twenty spider-claws or whatever for the first half of the game?

    A review like that would also help buyer’s decision on the first day. It would pretty much say “if you are going to buy the game today, this is the experience you are going to have for the evening. You aren’t going to experience the exciting stuff on the box for a while.” And the people would say “I don’t want to spend my evening collecting basilisk feathers, so I’m going to buy a different game.”
    MMO-reviews that only explain the big selling point of an MMO, and who only mention what you actually do most of the time in one paragraph, fail the players I think.

    If MMO reviews would have been done like that ever since WoW came out, we would have much better MMOs nowadays, and a lot of people would have realized that there are better games that they can spend their time on.

    I know the counter argument is going to be “but later on the game is amazing. You risk condemning a brilliant game.” To which my reply is: No, the game is not amazing, simply because the first half sucks. People haven’t realized how much what they play is not a good game because nobody has ever said it. If the game is brilliant later on, and you give it a great score for that, you are actually doing the people a disservice because they are going to collect golem pieces for however long while they could have exciting experiences in other games.

  41. Garg says:

    I remember that PC Gamer decided to combat this a while ago by saying that they would always revist an MMO after a sizeable update or active community presence presented itself and write about it in the “Long Play” section of the magazine. That is a system works really well for the informed person; the initial review will tell you if the MMO will provide a good game experience from the off and with minimal time investment, and the long play a few months later tells you if its worth getting into after its bedded in.

    The problem for game developers is that the all-powerful Metacritic review score (and the score at the end of the “Google + game name” review) still remains the same.

  42. Kazang says:

    This nonsense about needing a review on the release day for an mmo has to stop.

    RTW are right to insist that the game is given time. Communities are what drives and ultimately makes or breaks an mmo. Journalists and players alike have to accept this. People should not expect a number at the bottom of a page to sum up every game on the release day, this applies as much to a deep single player game as it does a mmo.
    Writing a review without having seen all the content the game has to offer is borderline negligent. It’s like selling crack (yes I know this is melodramatic but the comparison works), sure they (the customers) may think they want a review right on release date but really all that is doing is harming both their own opinion of game and the developer. This applies to bias either way having a detrimental effect.
    The bottom line is RTW is right and you cannot accurately review a mmo without giving it some time to mature. Journalists should take some responsibility and admit this.

    I’m not defending APB or going to comment on it’s quality or lack thereof, but RTW was right to make this stand. And lets face it a week is still really very short and doesn’t give the community much chance to grow. It’s not like it’s a particularly unreasonable amount of time.

    Coverage and articles about the content that is available and has been played, sure. But to call such information a review is not really on for a serious games journalist. RPS are normally very on the ball with this and I salute you guys for that. The recent Alpha Protocol “not review” is what should be presented, with a full review coming as and when the reviewer can give a full and well rounded opinion.

    • Web Cole says:

      I’m just gonna quote Mr Phil Cameron on this:

      “I feel the only logical way to do this is with a pair of reviews. As Kieron suggested; one at release, with initial impressions, and then one 6 months to a year down the line that has a fair assessment of the community and how well it’s being supported. It’s the only way to a) be fair to the game, b) be fair to the journalists reviewing the game, c) be fair to the readers who want to know whether it’s worth it, and d) be fair to those comissioning the reviews, who can’t afford to pay huge amounts for a single review that doesn’t pull significantly (if any) more views than review of Shooter X.

      So yeah, that.”

      You seem to be focusing on a, and there’s really way more to this whole thing than that.

      Also, selling crack is not ‘negligent’. No. No, the comparison does not work.

    • Mr Labbes says:

      I’d argue that because of the embargo, reviewers will not give it more time. If you want people to spend more time with your game, encourage them instead of layering embargoes.

  43. Catastrophe says:

    I think MMO’s need an ongoing Review method.

    10 Hours

    I have currently played the game for approximately 10 hours and I am enjoying blah blah blah.

    Then link the following review onto the bottom of that review for 20 Hours, etc.

    After maybe 3 Ongoing Reviews you do a “Conclusion Review” which basically Reviews your full experience with the game for how ever many hours you’ve played, stating its issues and its issues that have been rectified and stating its positives and basically rate the game on your full experience.

    People can then judge “10 hours in and hes still really enjoying it, thats where i usually realise if i’m going to like a game or not” or “I’m more interested in 20hours+ as thats where my attention usually falls with other MMO’s”.

  44. Carra says:

    The magazine I read (pc-gameplay) splits up the review of big mmorpgs. In a first issue you’ll get the first impressions. In the second issue, a month later, they’ll give the game a score and talk a bit more about it.

    It still doesn’t give a perfect picture of course. It took me a few months to get my first maximum level in WoW at which the game “starts”. But it’s a fairer way than playing for two days and judging it.

  45. Snowy says:

    The bottom line is that the people making the decisions over review embargoes and the like are marketing execs. They have no desire to enter into conversation with the games reviewers, as doing so would lead to genuine grievances being aired that they might have to tackle, and they generally avoid that like the plague in my experience.

    A precedent has been set, and they will blindly follow it, herd fashion.

    The people that are passionate about games are the players, the reviewers, and the developers. Everyone else involved in their production only cares about the bottom line.

  46. DXN says:

    Not being into MMOs I don’t have much to contribute to the debate, but: <3 Alec, and <3 RPS.

  47. The Sombrero Kid says:

    you can’t review a computer game, stop trying! Journalism is all about transparency, the simple truth is mmo developers are accusing review writers of not presenting the full truth, which honestly they are not. Every review game review should be accompanied with a break down of when and for how long the journalist played it for, otherwise the journalist is being deliberately deceitful, if a reader decides aww that was months ago and for 10 minutes that helps inform their purchasing decision more than before and it raises the bar in the ‘honesty wars’ between developers and journalists.

  48. snv says:

    It is a matter of trust, but i see the important trust-relatioship between the reader and the reviewer.

    If i have found a reviewer with which i can agree regularly, then i do not have to know the details, like how much genre-experience, or how much clocked testing time he has. Those things might only become interesting when i disagree.

    But i expect an experienced (jaded) reviewer to be able to see through the fluff and notice _if_ a game is flawed or merely cookie-cutter-mediocre. Sometimes that can happen in less than an hour, so that counterargument is flawed.

    Also, i don’t think a reviewers job is to measure the potential of a game, if it can become better with some patches/expansions or developing comunity. If it sucks now, say it. If it gets significally better later, there can be new reviews.

  49. John Peat says:

    I’m just wondering how much this whole situation is born in a world where just about anyone can setup a website and claim to be a ‘games reviewer’??

    The quality of some reviews which feed into MetaCritic/GameRankings is almost comedic – the quality of reviews on the big name sites can even be quite embarrasing and that’s all before you get the offhand opinions of people who now have a million channels through which to promote their inability to see the world from anyone else’s point-of-view.

    MMOs are unique in that if you want to play them for free, before release then it’s almost always possible to do so(*). This reduces the need for a ‘review’ of such games because most people who are curious will try the game for themselves – even at the cost of knowing their efforts will be thrown-away…

    This leaves the job of an MMO reviewer as more of a marketting exercise – a way of hooking-in people who hadn’t considered the game upto the point they see the review(s) – even if it’s on Metacritic or the BBC site or whatever…

    and y’know what – a bit of scandal like demanding people don’t review your game until well after release is probably just as effective…

    (*)There are even ways to guarantee beta access to WoW expansions – £10 in an envelope if you want to know em although I guess most people who care will be playing WoW and not reading this :)

  50. Dawngreeter says:

    I think the only sane conclusion here is that the game review busyness is defunct. Someone already stated it – Wot I Think > review. Authoritative scoring needs to go away. I approve of the IMDB-type scores because they are, essentially, an indication of what the global community thinks. But they still mean nothing and I think people who watch movies based on IMDB scores are… well. I’ll be polite. People I don’t want to be friends with.

    Who in his right mind decides to listen to an album because of a score someone had given it? Who reads a book because of a score? No one, that’s who. If there was ever an indicator that video games are still nowhere near some imagined maturity level one would expect from an entertainment industry (and a form of expression) it is this pointless, utterly ridiculous fixation with goddamn scores.

    I want a public conspiracy of professional game reviewers. Can you guys all get together and agree to give a solid 8.5 to every game you review professionally, no matter the content of the review?