He Shoots, He Scores

By Jim Rossignol on June 9th, 2008 at 10:42 am.


There’s been some hubbub lately over the popular topic of videogame review scores, thanks to a recent column by Simon Parkin. In his GameSetWatch thought-piece Mr Parkin makes some rather astute observations about the troubling numbers, but he also comes up with a rather contentious assertion.

The average reader (even if they don’t know it) is after a complete objective, scientific comparison between game x and game y with data and statistics and, finally, a numerical point on a linear scale by which they can compare, for example, Mass Effect with Rock Band and see which one is empirically better.

Oof.

I’d argue for something quite different. I think that the median of readers actually wants a subjective opinion, even if he doesn’t know it. In fact, I get the feeling that Mr Parkin actually goes some way to saying that in his article when he addresses the subject of hype, which he thinks conditions readers’ expectations.

Scores then become a reference to a game’s preceding hype. An 8/10 for a game that was hugely hyped to hobbyist gamers is a punch in the stomach for excited fans (see the anguish exhibited in the MGS4 comments thread). Conversely, an 8/10 for a game nobody cares about is viewed a gross over-generosity.

What the hype topic does is raise the question of how much someone has been exposed to this or that marketing ecology, and to what degree they are susceptible to its influences. And anyway, haven’t we all kind of agreed that word of mouth is what really carries the most weight? It’s this that leads me to the idea that in fact all gamers want is a subjective description, even if that description is simply a number attached to the game. And let’s be clear about this, the number is a description, in some sense, because it’s trying to attribute some kind of quality to the game. (A “seven” kind of game.) I remember seeing one Zero to Ten score system given verbal equivalent. “0 = Unplayable, not a game. 5 = Okay, but boring or badly made, 10 = Amazing, brilliant in many ways,” that kind of thing. And I think that’s an honest way of looking at them. There’s a reason why most people look at that score at the end of a review before reading: it’s the gist of the review, the most general description we can give. (Perhaps attributing that 8/10 becomes more perceptual than anything else, like saying it’s a red-brown coloured game, when you thought it was more rust coloured.)

Readers want a description, starting with a number, because it allows them to better define their own thoughts on a game, whether or not they’re in-line with the conclusion of the review. If undecided it might nudge their feelings one way or the other. Or, if they’ve already made up their mind, it allows them to express their thoroughly ingrained opinion – strikingly illustrated by Oli Welsh’s Metal Gear Solid Review. Readers get to argue why the review is wrong, or why their description is more suitable. It’s not exactly dialogue, but the result is similar: we get to make up our minds about something, either by changing our description, or confirming that what we thought was right all along. And there’s not much that’s objective about that.

__________________

« | »

, , , .

80 Comments »

  1. cliffski says:

    What I want from a game review is information. I want to know stuff that I cannot tell from screenshots, movies or the demo. In other words:

    How long is the game?
    How replayable is the game?
    How varied is it?
    Is it stable?
    What’s the performance like?
    To what extent is the demo a reflection of the full game?
    What sort of stuff happens later, not in the demo?

    I wish more reviews had that sort of information in them.

  2. The Hammer says:

    I used to be a slave for scores, and always want them at the end of reviews. But now – well, I think not having a score would encourage me to read the actual review in the first place, but not only that, but defeat a lot of these anal arguments about MGS4 being rubbish because it only got an 8/10, and how conversely in Eurogamer’s eyes it MUST be of the same quality as Halo, but two bars below Halo 3.

    It’s a bit arbitrary, I think, and I think people pay more attention to the score than the actual review itself, which is a shame. That’s why I like RPS reviews, because the only score is whether or not you’d buy it.

  3. Jeremy says:

    I agree with your take on this. The number is important and probably is meant more to fit out expectations than anything else.

    One thing we should all recognize is that reviews are subjective, and if we can factor that subjectivity in, we can make better buying decisions. Jim likes STALKER, and if he reviews a game and tells me that he likes it and it reminds him of STALKER in a good way, then I’ll know to check it out. The team at Ars Technica have a different gaming style than me, but I take it into account when they review titles, and purchase accordingly.

    The numbers are useful in their way, but actually taking the time to read the reviews is better. And another reason to keep reading the same reviewer.

  4. itsallcrap says:

    Scores are no substitute for a well-written review, but I don’t see why we shouldn’t have both.

    Those who feel they are useless can just ignore them.

  5. Jim Rossignol says:

    Cliff: I wish more games had demos! But I agree there needs to be some accuracy in the description, and some utility. That’s what people are talking about when they mention “objectivity”, I think.

  6. Sucram says:

    When somebody criticizes the score somebody gave a game, the next comment will almost inevitably be ‘read the review!’

    Every time somebody walks into a store and picks up a football game with a ’93% – ilovefifa.com’ sticker on it should we accost them and shout “DID YOU READ THE REVIEW”

    and since reviews are subjective shouldn’t they have read half a dozen reviews to get a more balanced opinion and to objectively determine which opinions deviate from the norm.

    Even those of us who post comments on blogs about computer games will probably only read a couple of reviews of a title and skip to the conclusion on a few more; and those times we do read reviews it’s often for amusement as much as to decide whether to buy the game or not.

    In semi-short:
    * not many people read reviews

    * People who think ‘I quite like this sort of game, I wonder if it’s any cop’ will skip to the score/conclusion.

    * People who think ‘I have no idea what this game is’ or ‘I love this reviewer, I want their babies’ just might read the review.

  7. Don says:

    It would be a stretch to say that you could come up with an accurate numerical comparison between games of the same basic type, say, Mass Effect and KOTOR. To try to compare Mass Effect to Rock Band is just silly and irrelevant as it’s not a choice I’m ever likely to want to make. What I want to know is whether Mass Effect is a good RPG, when I’m in the mood for another RPG, and is Rock Band a good [insert genre here] when I’m after something different. Scores are a handy shorthand to a review’s verdict but if a game is the type of thing I’m interested in it’s the reviewer’s comments, filtered through whatever I know about their own likes and dislikes, that I find useful.

  8. Cooper says:

    That Parkin’s piece was interesting – I hadn’t realised scores came from the techie side of things (though, not surprising really). I certainly don’t want or expect an objective score, but some guage is useful.

    Though basing commercial decisions on reviews seems laughable. Poor PR person, with the weight of those pay cheques on their shoulders…

    I’d rather not get rid of scores – I’ve grown up with them, and am fond of them. But for all thesee 100 point score grades, surely a 5-grade rating, as with films, is more justified. It’s that false sense accuracy that 86% or whatnot gives.

    Edit: Also; demos. more. please.

  9. Butler` says:

    His interpretation of what he perceives readers as “wanting” from a review is, in fact, exactly what they don’t want. At least from where I’m sitting.

    They want an opinion, preferably one with backing. An argument with some real weight behind it. They want information, as Cliffski says.

    For me, the review score itself serves as a (hopefully) informed reference point by which to compare similar games, as oppose to Simon’s strange example of Mass Effect vs Rock Band…

    And all this from the guy that brought us the painful Rez HD review.

  10. ryn says:

    Slapping a number on a game is always selling it short. While I do not think a number at the end has to hurt the actual review, I immediately get the suspicion that the review is less criticism and more buyer’s advice. And I don’t want that from people I don’t know personally.
    Rating any kind of experience on a scale just seems ridiculous to me.

  11. Slappeh says:

    I say we ban scores. Zero Punctuation is great because of this very reason – as well as his awesome honesty and always thinking what we are thinking.

  12. Ginger Yellow says:

    I can’t speak to the average reader’s demands, but for me the score is only a starting point. If I’m flicking through a mag and a game scores 3/10 or 45%, I’ll ust ignore the review unless it’s a particularly hyped game and I want to see where it went wrong. Scores don’t really enter into my buying decisions beyond that. As Jim says, there’s nothing objective or empirical about a review score.
    I’m interested in comparing the subjective view of the reviewer with my prior knowledge of how his/her tastes compare with mine. If I know the reviewer is a big Company of Heroes person, for example, and they praise the tactical play of a new strategy game, then I’m pretty certain I’ll like it. That’s why I like places like RPS, where the tastes of an individual reviewer/poster are given more free reign.

  13. Simon Parkin says:

    Cooper: I’d like to see a game publication assume the Halliwell’s Film Guide’s 4-star scale.

    By Halliwell’s standard, all films by default receive a zero star rating. Only exceptionally interesting and important films manage to receive a one or two star rating, with a tiny handful (just over 1%) of the 23, 000 odd films covered in the guide receiving the maximum recommendation of Four Stars.

    This marking scheme, which many people disagree with or misunderstand, is a way of positively affirming interesting, well-executed and excellent films while damning all other comers with indifference.

    I think i’s a system that would translate to games very well as it clearly defines what the score is communicating and removes the idea that the review and its score is anything but a subjective critique.

  14. Ben Abraham says:

    I got the feeling that Parkin was really looking for more ‘New Games Jouranlism’ as outlined in KG’s Manifesto. Why have so few people still not realised the incredible power of writing in the style of the subjective, personlised account? Sure, like Parkin, I acknowledge there’s a place for scoring, but it’s a problematic area.

    Edit: I see Mr. Parkin has posted above. He might disagree.

  15. Simon Parkin says:

    Butler’: I agree that mature readers want well-argued and provocative opinions from their reviews but I think you’re overestimating many readerships.

    Tim Edwards, in the comments thread over at GSW challenged me on my second guessing what an ‘average’ reader wants/ understands from scores. By way of anecdotal evidence, here are two two readers talking about their score expectations in the gamespot news story in which the publication announced they were moving from a 100 point scale to something much more complex:

    “Wow Gamespot…You took the one thing that made your reviews better than every one elses, how intricate and specific they were, and dumbed it down to a system I would only expect from some 16 year olds freewebs site. This is horrible. Now I won’t know how much better a 9.5 game is from another 9.5 game.”

    “Now if i want to know how good a games graphics were compared to another game, I’ll have to read 7 paragraphs of text instead of looking at a simple, easy to understand interface that creates a well weighted average gamescore.”

    You only need to dip into any number of Eurogamer threads to see that many readers view scores as empirical indicators by which to compare any two games’ quality.

    The exaple of Rock band vs Mass Effect was intended to show how ridiculous that idea is.

  16. Gap Gen says:

    I found this with Bioshock. The release hype had been such that I found it impossible to enjoy as much as I was told I would.

    A similar thing happened with Portal – I played it immediately on release and loved it, but someone who played it some weeks after complained he couldn’t see what the hype was about. Because hype is exactly that – over-the-top love for a game that will poison the expectations of the newcomer, despite the best intentions of the hyper.

  17. Lorc says:

    Scores are a tool in the reviewer’s arsenal. They don’t necessarily have to exist outside the text of the review in any particularly useful way.

    Writing a review where most of the wordcount is picking (important) nits and then ends with an 8 or 9 will emphasise that this is a game which manages to be great /in spite/ of its many flaws.
    Or go on and on about how much you love this game, but then give a middling score and we know that there’s something to love there, but it won’t measure up to any kind of pseudo-objective standards.

    Obviously scores aren’t the /only/ tool, but they’re a useful one. Like any tool in a reviewer’s arsenal, using them relies on knowing what the reader will likely take away from any given number, and so there’s a constant balancing act saying what you want to say according to some hypothetical perfect scale, knowing that MGS4 an 8 will not go down as well as you might mean it to, and that metacritic will just add your magic number to the pile.

    And for all the problems this causes (spurring any number of efforts to find a better tool for the job like letter grades or star ratings) numerical scores do work. Every tool has its use. For all that they lack subtlety, are misinterpreted and misused, they have such brute-force impact that there’s no other tool fit to replace them.

  18. Gap Gen says:

    On the old score debate front, scoring systems are pretty arcane. In order to mean anything you need to know what a magazine rates in each band, like you say. A magazine that rates harshly in order to make the most of the whole range from 0% to 100% will be ranted at by everyone for being harsh, even though their scoring system only makes sense internally – you can’t compare two magazines with different scoring criterion, although Metacritic suggests otherwise, that scoring has become vaguely uniform across the industry.

  19. Noc says:

    The MGS4 review has 2069 comments. I don’t say this often, but . . . lol.

    (For the immense quantity, not the juvenile innuendo.)

    And Gap Gen: You don’t get much less arcane than the three-point spread.

  20. brog says:

    It’s actually pretty easy to convert a computer game to a numeric value that sums it up entirely, and contains all necessary information about the game.
    Unfortunately these numbers are often quite large; millions of digits long. Reviews do not usually print them, but you can often find them on torrent sites.

  21. Butler` says:

    @Simon

    I know exactly what you’re getting at, and I for one appreciate the gesture – there’s definitely an issue within the scoring systems currently in place throughout most major online (and offline) publications. I’m just not feeling your argument.

    The scores are there for those who wish to use them, usually those who don’t wish to read the full written review in any depth (who are, as you say, a majority whether we like it or not).

    It’s almost a case of “we put the score their just in case no one can be bothered to actually read our review.” It’s this exact predicament that Zero Punctuation bypasses with its new, entertaining method of delivery. But I honestly don’t believe Croshaw could do what he does without this innate juxtaposition between ZP and the more traditional, score-based methods.

    I see scores as an essential starting point for further reading, as well as a reference point for comparison to other similar titles – however oversimplified they may seem at times.

  22. Meat Circus says:

    Sounds like Mr Parkin is saying that, in essence, 90% of the people who read your reviews are idiots.

    And he’s right.

    Nonetheless, the whole review score thing, whilst intended to placate the idiots, actually only encourages them. You only have to read the comments thread after any slightly contentious Eurogamer review to see that.

    By omitting the review score, you force the idiots to base their idiocy on your words. And I suspect that 70% of the idiots have neither the inclination and/or the ability to read and critique that many words in one go.

    So the idiots stay away, and the discerning gamer gets considered, well-written prose on which to base their decision whether to pirate^H^H^H^H^H^Hbuy the game. Everyone’s a winner.

    Review scores: just say no.

  23. Strelok says:

    The numbers are evil.

    There are 2 basic problems with them:
    1) They are supposed to be objective (unlike the text of the review), you know – math is truth :), which they never are.
    2) They are always compared to other numbers because in the end what remains in the memory of the reader in the long term is the number. So, when a game gets an 8, it is compared to other games which got 8. And, surpriiise, chances are a lot of the other games which got an 8 are crap in comparison to this particular one. Which obviously is the case with MGS4. There is no need to be a fanboy to notice this.

  24. Dinger says:

    First, numerical scores are bullshit. Second, numerical scores are extremely useful. Third, they would be more useful if people could agree on what they meant.

    They’re BS. I’ve graded university term papers: basically, a term paper is an exercise where all the students write the same time of text on one of a few themes. They get a letter grade (US system, from F to A+). But we found that it’s much more effective to give a percentage grade: there’s higher granularity and an aura of precision. But it’s still BS. I can tell, with reasonable accuracy, what constitutes a B- and what constitutes a C+, but there’s no practical way I can distinguish between a 80 and an 81.
    Okay, you scientists will say, that’s perfect: you’re measuring to one order of magnitude beyond the gradation of your instruments. Maybe. But maybe the +/- constitutes that order of magnitude. Anyway, the point is, the hard numbers somehow make the grade they received feel more real, and give me more room to negotiate. (“Okay, so I missed that. I’ll give you an extra point”). You also have to “sell” whatever grade you give. A C+ needs a lot more red ink than a B+, even though it’s not the individual faults, but the thing as a whole that makes it a C+.

    Alright, to videogames. Yes, numbers are BS. Portal has a metacritic score of 89 and Bioshock gets a 96. AudioSurf clocks in at 85. Does that seem right to you?
    But numbers are useful. They do force the reviewer to consider the game in relation to itself and and others. The problem is that numbers do not all fall into the same review mechanism.
    Some folks I suspect are reviewing on a 1-10 scale, where a bad game scores 3 or below. Others are reviewing based on the A+ to F scale. The Onion AV club does this letter scale explicitly, which metacritic then translates into its percentage scale. Other reviewers following a percentage scale tacitly follow the A+ to F scale. So, for pedantry’s sake, here’s what those numbers mean:
    <60% = (F)ail
    (0,1,2 are -, and 7,8,9 are +)
    60-69 = D (Barely passed)
    70-79 = C (Average)
    80-89 = B (Good)
    90-99 = A (Excellent).

    You can see that this would also translate to a star system. If you want to apply this to games and criticism, you need to pick internal criteria. A game that genuinely blows, from idea to execution to packaging, gets an F. One that barely made it to market, with major sections missing, and presents something substandard in its genre, gets a D. Standard fare is a C. Something that executes well on its ideas, and presents something that sets the standard for its class, is a B (or one Michelin Star). And truly exceptional experiences get As (or two/three Michelin Stars). Games should follow a standard distribution with 79 as its centerweight.
    That’s the theory behind one approach. Numbers can’t tell you if a game is fun or not, but they can report on the craftsmanship, completeness, and relative interest of a title.
    They shouldn’t correspond to sales; who wants to read a review to know how well something will do on the market?

    But why mix numbers that score games according to different gradations? If that’s the case, then the result is a meaningless measure of anything but market hype. And at that point, all your Metacritic score is doing is closing off the market to any but the big players, who will have the resources to make sure they efficiently achieve the highest scores possible.

    And tying bonuses or online availability to metacritic score is stupid. The last thing this evaluation system needs is to be tied directly to the developers’ economic well being. Yes, critics have always had this role, but directly linking the two will guarantee that the numbers in question take even less meaning.

  25. Lorc says:

    Dinger: “First, numerical scores are bullshit. Second, numerical scores are extremely useful. Third, they would be more useful if people could agree on what they meant.”

    Best, shortest and most accurate summary of the issue I’ve ever seen.

  26. Butler` says:

    Alright, to videogames. Yes, numbers are BS. Portal has a metacritic score of 89 and Bioshock gets a 96. AudioSurf clocks in at 85. Does that seem right to you?

    Honestly: yes, it does. I couldn’t really argue any of those “scores” one way or the other.

    But yeah, I suppose the overriding issue here is a lack of consistency between publications to make things like Metacritic more useful. Which is never going to happen. :[

  27. Ben Abraham says:

    Does that seem right to you?

    Dinger, are you channelling the black assassin from Firefly? =P

  28. Butler` says:

    One thing I’d like to add is that fixed “scores” really are a major issue when it comes to MMOs, as one of RPS’ guys (can’t remember which one of you is doing the PCG review of AOC…) will probably attest to.

    A static score thrown at something that changes so dramatically from its original form really is abstract and meaningless. Funcom are cranking out patches incredibly fast and making real, tangible changes to the game – making scoring it even harder than normal.

    I’d hold KGs 8/10 WoW review up as evidence, except WoW was NEVER an 8/10 game imo! :p

    Eurogamer’s re-review system is, of course, aiming to counter this somewhat.

  29. Kris says:

    @ Dinger’s comment. Do you blog sir? You may want to consider a change in career if you dont. One of the most insightful and succinct critiques of scores I’ve seen. for my part I do pay attention to scores, but I’m more aware than ever that I can only at best refer them to scores by the same publication. Even so, changes in editorial team, style, etc… and the passge of time can still render such comparisons moot. Seen far to many end of year / all time greats lists that seek to modify or justify a score issued by the publication previously.

  30. Noc says:

    I think part of the problem has nothing to do with numbers at all: it’s that people don’t look at reviews as “Some guy’s thoughts after playing the game,” as (arguably) a reasonable person would do after reading something written by an individual human being about something they’ve seen. Maybe this is part of the system’s fault, as the writer is deemphasized in favor of their position as the mouthpiece for the publication.

    So it becomes “Eurogamer said,” or “Gamespot said,” which leads to the idea that the review score is the OFFICIAL OBJECTIVE JUDGMENT – which, by virtue of it’s ostensible position as the be-all-and-end-all measure of a game’s worth, makes it easy fodder for argument. I’d compare this to, say, the Zero Punctuation bits, which are treated as “Oh, Yahtzee didn’t like this game.” As opposed to “This game only got an 8!” The scores, I think, help facilitate that, but they aren’t the root of the problem.

    I don’t think, in the context of reviews themselves, that the numerical score is necessarily a bad thing. As mentioned above, it’s a summary, and there’s nothing really wrong with that. But I DO think that maybe some time away from scores would be good for readerships as a whole. You know, until they’ve learned their lesson.

    And that’s my completely unworkable idea for today.

  31. Rook says:

    I always find that people whining how scores should go away and everything would be better are being pretty silly. If you’re someone who doesn’t want a score on your reviews, just ignore it and read the review. If you just want a score then there it is. If you want both then that’s great too.

    This idea that I should want less information in a review is pretty silly. And the idea that I should care how other people react to it (OMG TWILOIGHT PRICNESS 8.8 FAIL!!!!!!) is similarly nonsensicle (what fanboys do in the comfort of their own home is none of my business).

  32. Naurgul says:

    I’m strongly against this whole “reviews are only opinions and are subjective” turn journalists have taken lately. Let me explain myself. An opinion, on its own, is meaningless. Anyone can have one and not matter how much of a “professional” one wants to think himself, one person’s feelings after experiencing a game do not have any real value.

    When I read a review, I want it to to be as objective as possible. I don’t want excuses like “there’s no such thing as objectivity” and “games are too complex to express with just one parameter”. With that said, I’m not asking anything impossible either.

    There is certain information that one can extract while playing a game, certain objective properties of the graphics, the gameplay, the story. A subset of those can inherently be thought as positive or negative. For the rest of them, tastes may vary.

    So, my ideal structure of a review would be something like this: First, write what the reviewer’s expectations were from the game. This would help me put into context the inevitable subjective things that will creep inside the review. Then, a break down of the game’s properties, including how they work together, preferably separating the things that are usually thought as positive, the things that are usually thought as negative and finally: everything else. The review should end with an overview of the game, with the reviewer’s opinion and feelings (the only place where he should consciously put them). I guess the review could be supplemented with a brief list of major positive and negative things, but no numerical score. For the love of hate, please spare us of that.

    So, in conclusion: Dear reviewers: This is your job. Take it seriously. If you’re not sure about something or just “felt” something, please play the thing again or think about it more or ask another gamer’s or the developers’ opinion. Don’t just put it on the untouchable pedestal of “this is my opinion” and call it a day. Frankly, that’s just sloppy.

  33. Meat Circus says:

    @Rook:

    It’s not me I’m worried about, it’s them.

    Scores feed them, nourish them. We shouldn’t be encouraging them. We should probably be sterilizing them.

  34. Ginger Yellow says:

    Nuargul, what is objective about gameplay or story? Some people love turn-based games and others wouldn’t touch them with a bargepole. Some people want their shooters to have intricate plots and others just want to shoot lots of aliens. Some people will find a particular story engrossing while others find it trite – especially in games. Even graphics aren’t particularly objective in that art style wins over technical prowess for some people and not for others. This isn’t to say that reviewers should channel Hunter S Thompson, merely that reviews are inevitably subjective and it’s best for everyone to recognise that and make the most of the medium.

    The same is true of film or music reviews, and nobody finds that objectionable. The issue is complicated in games by the fact that there has to be a consumer advice element – does the camera work properly, how intuitive is the interface etc. But for the most part it boils down to personal enjoyment in the context of other games. That’s never going to be objective.

  35. Cooper says:

    @Naurgul
    Would you expect the same from reviewers of books or film. Do you think those can be broken into constituent parts and analysed objectively? I think many people, especially those whose job it may be to quantify such things, would disagree.

    The last thing I want from my film reviews is a descent into the technicalities of camera work, editing and Sfx. Those are the most easily quantifiable parts of films. Sure, they should feature – the same way graphical fidelity, sound effects, stability can be with games. But there’s a whole lot more to visual and textual media which is not quantifiable. Even more so with games, where the nature of interactivity means people engage with the games in a variety of ways, where traditional modes of textual analysis can not be employed as the consumer cannot be considered passive. I fail to see how, for example, narrative techniques can be broken down into ‘good’ and ‘bad’, or a contimumn thereof. Cut scenes; objectively good or bad? SS2/Bioshock audio logs – good or bad? etc.

    A reviewers job, for me, is to provide an opinion – not an unbiased one – no such thing exists – but an opinion with the weight of accumulated knowledge and love for the medium and the ability to write engagingly and succinctly behind it.

  36. Pod says:

    Games should follow a standard distribution with 79 as its centerweight.

    Actually, I believe it’s 73%.

    As for reviews, I’ve always rated things like “The Big Game Hunt” in PC Gamer and the equivilent as being the indicator of the “best” games, rather than a 9/10. Though I did used to let out a giggle whenever I saw the old 9%. (South Park Racing?)

  37. Robin says:

    I think it is possible to have a scoring system that meaningfully describes the level of product quality. Gamespot (prior to the redesign and staff exodus) and PC Zone (ditto) were good examples.

    This is in effect how Halliwell’s system works, if you map stars to 8.5, 9.0, 9.5 and 10.0, and can be (and has been) applied as stringently.

    Edge’s system, where scores are fixed in perpetuity and try to gauge if a game has DONE AN ART (leading to a mass of 6-8 scores of games with wildly varying levels of quality, because most games are not developed with the intention of meeting Edge’s approval) is less useful to the reader. The top score isn’t consistently defined and is handed out so rarely as to be meaningless.

    Eurogamer’s scoring system is just random noise used to seed more random noise in the comments, of course.

  38. fluffy bunny says:

    “<60% = (F)ail
    (0,1,2 are -, and 7,8,9 are +)
    60-69 = D (Barely passed)
    70-79 = C (Average)
    80-89 = B (Good)
    90-99 = A (Excellent).”

    This is the biggest problem I have with review scores right now. I didn’t even know some sites did this kind of thing until fairly recently, but now that I know it certainly does explain why some (mostly US) sites always seemed to give games higher scores than I felt they deserved.

    The main problem happens when scores following this scale are mixed in with scores following the “proper” 1-10-scale, where 5.5 is average on metacritic and gamerankings, and there’s no way for the user to know which of the scores use which kind of scale without checking up on every single site.

    So when I see a game get 7/10 from a large number of sites, I think “hey, that must be a pretty good game”, as a seven is seen as a good score on the sites that I read. Then it turns out that half of these sevens are from people who think the game is average at best. It’s just silly.

  39. ACK says:

    @naurgul: A review is not by definition a piece of journalism, although the definition of journalism seems to be a thing in flux, one I like is “writing for presenting bare facts to describe news events”. Agreed, the motives and tastes of the reviewer should be, or become, apparent in the course of the review. But the most important role of a reviewer is disclosing and motivating his/hers opinion of the creative work he/she is reviewing. If the tastes of the reviewer is apparent I, as a reader, can judge how well that reviewer’s tastes correlates with mine, and thus if it is probable that we would agree on the quality of the work.

    Well that’s my opinion at least.

    So far I’ve only found a single publication that consistently employed writers with the ability to do just that, and that was Computer Games Strategy Plus (later renamed Computer Games Magazine), RIP. It’s slowly getting better though; the people here at RPS, some stuff over at the escapist and other places, makes reading reviews good again.

    Games journalism still seems to be a bit of a free-for-all though, with a bit too much emphasis on “the latest news first”, and not enough critique of sources etc. RPS et.al seems to do better at this too though. One could almost be led to believe that people doing it for love of the “art”-form as opposed to a career move have higher integrity. Who’da’thunk?

  40. Naurgul says:

    ACK, I agree that getting a view of the reviewer’s motives, expectations and tastes is a definite step in the right direction, but I feel that we shouldn’t stop there.

    Ginger, here’s a partial list of objective or almost objective attributes which can be thought as negative or positive, divided in broad categories (which themselves are partial as they do not fully cover everything in a game by a long shot)

    Graphics: animation, texture resolution, face-posing, UI intuitiveness, scalability, performance…
    Gameplay: pacing, intuitiveness, originality…
    Story: Writing, originality, understandability, emotion-evoking
    Misc: how well the graphical style supplements the gameplay, how well the gameplay is merged with the storytelling, modability…

    Cooper, yes, I would expect the same from book and film critics. I personally think that the critics of more mature media are purposefully obscure in their criticism to conceal the shortcomings of their subjectiveness and I’m afraid that game reviewing will end up the same. With that said, I agree with you that breaking something down in its parts is not fully representative of the whole, this is why I said that a summary with an opinion in the end is welcome. In retrospect, I think it’s better to have an objective overview of the parts than a subjective overview of the whole.

    As far as quantifying things goes, I don’t actually expect them to give out scores for these things. They can say things like “The writing was bad”, “the end of the story was original and unexpected” “the gameplay was at parts repetitive”, “the graphics work well with the gameplay”, “the UI is annoying”.

    My point is, these completely subjective comments about how the reviewer felt while playing and how awesome everything is should not be in the spotlight, although they do merit a mention. Also, I don’t say that it’s not worth anything unless the end result is completely objective, but there should be an honest attempt at that. Instead of, you know, saying “Deal with it, it’s my opinion”.

  41. John P (Katsumoto) says:

    What I hate is how most of the people on that MGS4 thread are saying things like ” -WHEN- I play this it will completely blow me away this review is shit”. None of them have even played it yet, but they’re inclined to believe the hype more than a professional review.

    I’ve just seen similar things over at EDGE’s website. Horror, really. it’s almost as bad as reading Have Your Say over on the BBC website.

  42. Thiefsie says:

    It’s funny because I have only consciously approached this topic in the last couple of years, whereas before I had an inkling of what we were all thinking but did not succinctly come to a conclusion or simply care enough to get past basic review systems. Very much so due to the blog intensity of the internet these days my connections to certain particular reviewers are a hell of a lot more important than any review score. I used to trawl gamerankings and would pick all the games above 80% for a system and set out to play or at least try them all, but realised this would still largely fail in consistently getting me games I would like – enter the subjectivity. Nowadays I find the reviewers I tend to agree with and stick with them and generally throw all other reviews away and just read them for interest’s sake. Do that enough and you can tell when a writer is easily sensationalising or pandering to hype or other pressures and generally not being honest. The Rez review is a good example. I think he wants that game to do well as it serves a nice nostalgic element of desire and would possibly bring back a more pure semblance of gaming as opposed to the current mainstream… but still Rez HD is not a 10 game in any way shape or form?

    I’m mature enough to be able to judge a game generally by its own on paper merits, appearance, and other qualities to decide whether I like it or not. Rarely is a game a surprise like of mine, which is very unfortunate. The only place this probably doesn’t hold up is in multiplayer, where I may genuinely tend to rely on reviewer’s or word of mouth opinions.

    It’s hard these days especially with how much notice I take of the games industry but the best thing to do is just ignore most of the hype and take things on face value, decide whether there might be some sort of value to you personally, and then going from there with a purchase or not. At least that way I do like most of my games, if not absolutely adore them. Can’t say I’ve had any duds lately, except maybe Audiosurf… (that being said I’m willing to take more spontaneous risks on cheaper games of course).

    Omega Five isn’t a great game, but for when me and a mate want to play together on the one xbox it is the first thing we reach for, so is fantastic for us… and no review made us decide to give it a try or not?

    If only the masses were mature enough to make these decisions themselves we wouldn’t have over 2 thousand posts to a MGS4 review, which I think anyone could bluntly say; if you like MGS you will love MGS4, and if you hate MGS you will hate it too… And if you are on the fence… well… you’re probably better off starting elsewhere in the series to whet your appetite for what you could be in for – such is the nature of heavy, story based, and self-referential game.

    I think a simple yay or nay is the best way to measure a mark for a review, and the reader can then align themselves to whether or not they have a taste for the game within the writing.

    It will never be an exact science, and unfortunately for someone to seriously rely on a number out of 10 for a purchasing decision seriously needs to think about what they are doing. (OR at least if they are hurt by a game receiving less than they think it deserve, or more… to start rambling on about the injustices of it all)

    The saddest part of this of course is the way that metacritic and gamerankings are assuming some across the board consistency with reviewing, which obviously does not exist. How many times has ign pulled a review? or lest we forget gamespot and it’s shenanigans. The hardest hit by point systems are probably those development teams relying on a points grade to secure their next investment or reward, even more so than pure sales or other more tangible quantifications.

    I read enough of the gaming media to have a pretty clear cut view of what a game is going to be like and what I can expect from it. and thus don’t need to rely on scoring. Unfortunately this is of course in the minority.

    Demo’s are of course the sure-fire way around this… but in today’s current state, many more companies are purely just too chicken-shit (and maybe rightly so?) to release any demo form of their game. Assassin’s creed for example, looked absolutely brilliant leading up to release, and frankly I think a demo could have only helped their sales even more, as a demo would never have revealed how shallow a game it actually is, conversely making everyone who played it think of the possibilities… which they genuinely didn’t follow up on past the first assassination.

    I’ve rambled on for too long and am thus sorry if you’ve made it this far.

    I do know that I like reviews that are more colourful and personal, as a form of entertainment and maybe light judgement, but nothing more. Objectivism is boring, and can be found quite easily just from reading about a game in various media. A mix of the two is of course the perfect balance… but very hard to find. Many reviewers go a bit too far into the ‘high writing’ territory, as I think can be found at Edge quite easily for instance, and others just pander to the lowest common denominator, like IGN or perhaps Gamespot. Yet others give outrageous scores just for pagehits and sensationalist outlashes.

    Most of eurogamer’s staff straddle the line quite well i think, and of course not surprisingly the staff here do it well, with the review system and Optimus’ thumbs being a great indicator of purchasability.

    Most reviews are a pissing contest and little more.

    It’s just unfortunate that the mainstream (casual?!? haha) loves to follow along with the contest. In the eurogamer posts in response to the mgs4 review, it’s almost as if that 8/10 score will stop people buying the game as opposed to if it was a 9. Wow, with metacritic… maybe it will?

  43. alco75 says:

    I wish for all games to be rated on a two-scale system:

    1. KG recommends
    or
    2. KG does not recommend

  44. Thiefsie says:

    What I hate is how most of the people on that MGS4 thread are saying things like ” -WHEN- I play this it will completely blow me away this review is shit”. None of them have even played it yet, but they’re inclined to believe the hype more than a professional review.

    But they are right! Of course they are going to be blown away, it’s MGS, a game that is so individual that they can’t not be blown away if they know they like that kind of stuff. Of course they will like it! They are showing how much the review doesn’t matter to them in liking the game or not.
    The problem is of course the need for them to stick up for a friggen number out of 10, that if removed, would probably remove 1800 of those posts, or at least promote some more healthy discussion of the actual game.
    Or the unfortunate point of view that people out there are actually thinking hmmm it’s an 8. If it were a 9 I’d get it but for now.. nope

  45. Meat Circus says:

    @Alco75:

    Makes it a rather unlevel playing field for the small majority of people who are not Kieron Gillen.

    I’m personally very happy with the OPTIMUS PRIME THUMBS OF ULTRO-TRUTH.

  46. Meat Circus says:

    @Thiefsie:

    Or the unfortunate point of view that people out there are actually thinking hmmm it’s an 8. If it were a 9 I’d get it but for now.. nope

    And still, we allow these people to breed.

    It’s political correctness gone mad.

  47. James G says:

    The best review is one where you can read it and go, ‘Well, Walker clearly hated this but, judging by what he said, I reckon I’d enjoy it,’ and be right. (Or vice versa obviously) Of course, it helps when a publication matches a reviewer to the game. Why some magazines assign a strategy game to someone who hates the genre is beyond me. While it may have some advantages in the case of the genre leaders, which might attract in outsiders, it is marred by inexperience, and by making it difficult to seperated hatred of the genre from dislike of the game. While there may be benifit to criticism of genre tropes, it is misleading to shoehorn these into a single game review when they obstruct criticism of the game itself.

    I must admit, I occasionally use metacritic if I’m wanting to get a quick feel of a game’s reception, either because its a new release, or because I’m new to the format. (For example, when I recently cot a DS, I used metacritic to get an impression of which games are work considering, and which bombed.) I’ll then follow it up be looking at reviews, forum postings and the like.

  48. Okami says:

    @Meat Circus: If we wouldn’t allow stupid people to breed, we’d have nobody to make all the stuff like houses, food, money, power, tv, cars, roads and everything else.

  49. Meat Circus says:

    @Okami:

    Nonsense. A mixture of special baby-growing jars and homicidal deathbots will do the trick.

  50. Thiefsie says:

    Pure example of journalistic puffery for the sake of ‘excellent writing’

    Metal Gear Solid has always been a story of duty in the face of obsolescence, and if this is really it for Kojima’s chapter – and who knows, maybe the entire series – his duty has been fulfilled. MGS4 is not the game it could have been; nor is it the game it would have been had the series grown with the benefit of hindsight; nor is it the game it should have been if you believed that early trailer. But it is faithful to its fans, its premise and its heart, delivering an experience that is, in so many ways, without equal. In years to come, as people stand before the grave marked ‘Tactical Espionage Action’, they’ll feel little choice but to salute.

    8/10? ha!

Comment on this story

XHTML: Allowed code: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>