By Jim Rossignol on June 9th, 2008 at 10:42 am.
There’s been some hubbub lately over the popular topic of videogame review scores, thanks to a recent column by Simon Parkin. In his GameSetWatch thought-piece Mr Parkin makes some rather astute observations about the troubling numbers, but he also comes up with a rather contentious assertion.
The average reader (even if they don’t know it) is after a complete objective, scientific comparison between game x and game y with data and statistics and, finally, a numerical point on a linear scale by which they can compare, for example, Mass Effect with Rock Band and see which one is empirically better.
Oof.
I’d argue for something quite different. I think that the median of readers actually wants a subjective opinion, even if he doesn’t know it. In fact, I get the feeling that Mr Parkin actually goes some way to saying that in his article when he addresses the subject of hype, which he thinks conditions readers’ expectations.
Scores then become a reference to a game’s preceding hype. An 8/10 for a game that was hugely hyped to hobbyist gamers is a punch in the stomach for excited fans (see the anguish exhibited in the MGS4 comments thread). Conversely, an 8/10 for a game nobody cares about is viewed a gross over-generosity.
What the hype topic does is raise the question of how much someone has been exposed to this or that marketing ecology, and to what degree they are susceptible to its influences. And anyway, haven’t we all kind of agreed that word of mouth is what really carries the most weight? It’s this that leads me to the idea that in fact all gamers want is a subjective description, even if that description is simply a number attached to the game. And let’s be clear about this, the number is a description, in some sense, because it’s trying to attribute some kind of quality to the game. (A “seven” kind of game.) I remember seeing one Zero to Ten score system given verbal equivalent. “0 = Unplayable, not a game. 5 = Okay, but boring or badly made, 10 = Amazing, brilliant in many ways,” that kind of thing. And I think that’s an honest way of looking at them. There’s a reason why most people look at that score at the end of a review before reading: it’s the gist of the review, the most general description we can give. (Perhaps attributing that 8/10 becomes more perceptual than anything else, like saying it’s a red-brown coloured game, when you thought it was more rust coloured.)
Readers want a description, starting with a number, because it allows them to better define their own thoughts on a game, whether or not they’re in-line with the conclusion of the review. If undecided it might nudge their feelings one way or the other. Or, if they’ve already made up their mind, it allows them to express their thoroughly ingrained opinion – strikingly illustrated by Oli Welsh’s Metal Gear Solid Review. Readers get to argue why the review is wrong, or why their description is more suitable. It’s not exactly dialogue, but the result is similar: we get to make up our minds about something, either by changing our description, or confirming that what we thought was right all along. And there’s not much that’s objective about that.
09/06/2008 at 11:08 cliffski says:
What I want from a game review is information. I want to know stuff that I cannot tell from screenshots, movies or the demo. In other words:
How long is the game?
How replayable is the game?
How varied is it?
Is it stable?
What’s the performance like?
To what extent is the demo a reflection of the full game?
What sort of stuff happens later, not in the demo?
I wish more reviews had that sort of information in them.
09/06/2008 at 11:09 The Hammer says:
I used to be a slave for scores, and always want them at the end of reviews. But now – well, I think not having a score would encourage me to read the actual review in the first place, but not only that, but defeat a lot of these anal arguments about MGS4 being rubbish because it only got an 8/10, and how conversely in Eurogamer’s eyes it MUST be of the same quality as Halo, but two bars below Halo 3.
It’s a bit arbitrary, I think, and I think people pay more attention to the score than the actual review itself, which is a shame. That’s why I like RPS reviews, because the only score is whether or not you’d buy it.
09/06/2008 at 11:23 Jeremy says:
I agree with your take on this. The number is important and probably is meant more to fit out expectations than anything else.
One thing we should all recognize is that reviews are subjective, and if we can factor that subjectivity in, we can make better buying decisions. Jim likes STALKER, and if he reviews a game and tells me that he likes it and it reminds him of STALKER in a good way, then I’ll know to check it out. The team at Ars Technica have a different gaming style than me, but I take it into account when they review titles, and purchase accordingly.
The numbers are useful in their way, but actually taking the time to read the reviews is better. And another reason to keep reading the same reviewer.
09/06/2008 at 11:23 itsallcrap says:
Scores are no substitute for a well-written review, but I don’t see why we shouldn’t have both.
Those who feel they are useless can just ignore them.
09/06/2008 at 11:39 Jim Rossignol says:
Cliff: I wish more games had demos! But I agree there needs to be some accuracy in the description, and some utility. That’s what people are talking about when they mention “objectivity”, I think.
09/06/2008 at 11:43 Sucram says:
When somebody criticizes the score somebody gave a game, the next comment will almost inevitably be ‘read the review!’
Every time somebody walks into a store and picks up a football game with a ’93% – ilovefifa.com’ sticker on it should we accost them and shout “DID YOU READ THE REVIEW”
and since reviews are subjective shouldn’t they have read half a dozen reviews to get a more balanced opinion and to objectively determine which opinions deviate from the norm.
Even those of us who post comments on blogs about computer games will probably only read a couple of reviews of a title and skip to the conclusion on a few more; and those times we do read reviews it’s often for amusement as much as to decide whether to buy the game or not.
In semi-short:
* not many people read reviews
* People who think ‘I quite like this sort of game, I wonder if it’s any cop’ will skip to the score/conclusion.
* People who think ‘I have no idea what this game is’ or ‘I love this reviewer, I want their babies’ just might read the review.
09/06/2008 at 11:45 Don says:
It would be a stretch to say that you could come up with an accurate numerical comparison between games of the same basic type, say, Mass Effect and KOTOR. To try to compare Mass Effect to Rock Band is just silly and irrelevant as it’s not a choice I’m ever likely to want to make. What I want to know is whether Mass Effect is a good RPG, when I’m in the mood for another RPG, and is Rock Band a good [insert genre here] when I’m after something different. Scores are a handy shorthand to a review’s verdict but if a game is the type of thing I’m interested in it’s the reviewer’s comments, filtered through whatever I know about their own likes and dislikes, that I find useful.
09/06/2008 at 11:46 Cooper says:
That Parkin’s piece was interesting – I hadn’t realised scores came from the techie side of things (though, not surprising really). I certainly don’t want or expect an objective score, but some guage is useful.
Though basing commercial decisions on reviews seems laughable. Poor PR person, with the weight of those pay cheques on their shoulders…
I’d rather not get rid of scores – I’ve grown up with them, and am fond of them. But for all thesee 100 point score grades, surely a 5-grade rating, as with films, is more justified. It’s that false sense accuracy that 86% or whatnot gives.
Edit: Also; demos. more. please.
09/06/2008 at 11:50 Butler` says:
His interpretation of what he perceives readers as “wanting” from a review is, in fact, exactly what they don’t want. At least from where I’m sitting.
They want an opinion, preferably one with backing. An argument with some real weight behind it. They want information, as Cliffski says.
For me, the review score itself serves as a (hopefully) informed reference point by which to compare similar games, as oppose to Simon’s strange example of Mass Effect vs Rock Band…
And all this from the guy that brought us the painful Rez HD review.
09/06/2008 at 11:51 ryn says:
Slapping a number on a game is always selling it short. While I do not think a number at the end has to hurt the actual review, I immediately get the suspicion that the review is less criticism and more buyer’s advice. And I don’t want that from people I don’t know personally.
Rating any kind of experience on a scale just seems ridiculous to me.
09/06/2008 at 11:53 Slappeh says:
I say we ban scores. Zero Punctuation is great because of this very reason – as well as his awesome honesty and always thinking what we are thinking.
09/06/2008 at 11:57 Ginger Yellow says:
I can’t speak to the average reader’s demands, but for me the score is only a starting point. If I’m flicking through a mag and a game scores 3/10 or 45%, I’ll ust ignore the review unless it’s a particularly hyped game and I want to see where it went wrong. Scores don’t really enter into my buying decisions beyond that. As Jim says, there’s nothing objective or empirical about a review score.
I’m interested in comparing the subjective view of the reviewer with my prior knowledge of how his/her tastes compare with mine. If I know the reviewer is a big Company of Heroes person, for example, and they praise the tactical play of a new strategy game, then I’m pretty certain I’ll like it. That’s why I like places like RPS, where the tastes of an individual reviewer/poster are given more free reign.
09/06/2008 at 11:57 Simon Parkin says:
Cooper: I’d like to see a game publication assume the Halliwell’s Film Guide’s 4-star scale.
By Halliwell’s standard, all films by default receive a zero star rating. Only exceptionally interesting and important films manage to receive a one or two star rating, with a tiny handful (just over 1%) of the 23, 000 odd films covered in the guide receiving the maximum recommendation of Four Stars.
This marking scheme, which many people disagree with or misunderstand, is a way of positively affirming interesting, well-executed and excellent films while damning all other comers with indifference.
I think i’s a system that would translate to games very well as it clearly defines what the score is communicating and removes the idea that the review and its score is anything but a subjective critique.
09/06/2008 at 12:00 Ben Abraham says:
I got the feeling that Parkin was really looking for more ‘New Games Jouranlism’ as outlined in KG’s Manifesto. Why have so few people still not realised the incredible power of writing in the style of the subjective, personlised account? Sure, like Parkin, I acknowledge there’s a place for scoring, but it’s a problematic area.
Edit: I see Mr. Parkin has posted above. He might disagree.
09/06/2008 at 12:03 Simon Parkin says:
Butler’: I agree that mature readers want well-argued and provocative opinions from their reviews but I think you’re overestimating many readerships.
Tim Edwards, in the comments thread over at GSW challenged me on my second guessing what an ‘average’ reader wants/ understands from scores. By way of anecdotal evidence, here are two two readers talking about their score expectations in the gamespot news story in which the publication announced they were moving from a 100 point scale to something much more complex:
“Wow Gamespot…You took the one thing that made your reviews better than every one elses, how intricate and specific they were, and dumbed it down to a system I would only expect from some 16 year olds freewebs site. This is horrible. Now I won’t know how much better a 9.5 game is from another 9.5 game.”
“Now if i want to know how good a games graphics were compared to another game, I’ll have to read 7 paragraphs of text instead of looking at a simple, easy to understand interface that creates a well weighted average gamescore.”
You only need to dip into any number of Eurogamer threads to see that many readers view scores as empirical indicators by which to compare any two games’ quality.
The exaple of Rock band vs Mass Effect was intended to show how ridiculous that idea is.
09/06/2008 at 12:13 Gap Gen says:
I found this with Bioshock. The release hype had been such that I found it impossible to enjoy as much as I was told I would.
A similar thing happened with Portal – I played it immediately on release and loved it, but someone who played it some weeks after complained he couldn’t see what the hype was about. Because hype is exactly that – over-the-top love for a game that will poison the expectations of the newcomer, despite the best intentions of the hyper.
09/06/2008 at 12:17 Lorc says:
Scores are a tool in the reviewer’s arsenal. They don’t necessarily have to exist outside the text of the review in any particularly useful way.
Writing a review where most of the wordcount is picking (important) nits and then ends with an 8 or 9 will emphasise that this is a game which manages to be great /in spite/ of its many flaws.
Or go on and on about how much you love this game, but then give a middling score and we know that there’s something to love there, but it won’t measure up to any kind of pseudo-objective standards.
Obviously scores aren’t the /only/ tool, but they’re a useful one. Like any tool in a reviewer’s arsenal, using them relies on knowing what the reader will likely take away from any given number, and so there’s a constant balancing act saying what you want to say according to some hypothetical perfect scale, knowing that MGS4 an 8 will not go down as well as you might mean it to, and that metacritic will just add your magic number to the pile.
And for all the problems this causes (spurring any number of efforts to find a better tool for the job like letter grades or star ratings) numerical scores do work. Every tool has its use. For all that they lack subtlety, are misinterpreted and misused, they have such brute-force impact that there’s no other tool fit to replace them.
09/06/2008 at 12:17 Gap Gen says:
On the old score debate front, scoring systems are pretty arcane. In order to mean anything you need to know what a magazine rates in each band, like you say. A magazine that rates harshly in order to make the most of the whole range from 0% to 100% will be ranted at by everyone for being harsh, even though their scoring system only makes sense internally – you can’t compare two magazines with different scoring criterion, although Metacritic suggests otherwise, that scoring has become vaguely uniform across the industry.
09/06/2008 at 12:19 Noc says:
The MGS4 review has 2069 comments. I don’t say this often, but . . . lol.
(For the immense quantity, not the juvenile innuendo.)
And Gap Gen: You don’t get much less arcane than the three-point spread.
09/06/2008 at 12:24 brog says:
It’s actually pretty easy to convert a computer game to a numeric value that sums it up entirely, and contains all necessary information about the game.
Unfortunately these numbers are often quite large; millions of digits long. Reviews do not usually print them, but you can often find them on torrent sites.
09/06/2008 at 12:25 Butler` says:
@Simon
I know exactly what you’re getting at, and I for one appreciate the gesture – there’s definitely an issue within the scoring systems currently in place throughout most major online (and offline) publications. I’m just not feeling your argument.
The scores are there for those who wish to use them, usually those who don’t wish to read the full written review in any depth (who are, as you say, a majority whether we like it or not).
It’s almost a case of “we put the score their just in case no one can be bothered to actually read our review.” It’s this exact predicament that Zero Punctuation bypasses with its new, entertaining method of delivery. But I honestly don’t believe Croshaw could do what he does without this innate juxtaposition between ZP and the more traditional, score-based methods.
I see scores as an essential starting point for further reading, as well as a reference point for comparison to other similar titles – however oversimplified they may seem at times.
09/06/2008 at 12:29 Meat Circus says:
Sounds like Mr Parkin is saying that, in essence, 90% of the people who read your reviews are idiots.
And he’s right.
Nonetheless, the whole review score thing, whilst intended to placate the idiots, actually only encourages them. You only have to read the comments thread after any slightly contentious Eurogamer review to see that.
By omitting the review score, you force the idiots to base their idiocy on your words. And I suspect that 70% of the idiots have neither the inclination and/or the ability to read and critique that many words in one go.
So the idiots stay away, and the discerning gamer gets considered, well-written prose on which to base their decision whether to pirate^H^H^H^H^H^Hbuy the game. Everyone’s a winner.
Review scores: just say no.
09/06/2008 at 12:33 Strelok says:
The numbers are evil.
There are 2 basic problems with them:
1) They are supposed to be objective (unlike the text of the review), you know – math is truth :), which they never are.
2) They are always compared to other numbers because in the end what remains in the memory of the reader in the long term is the number. So, when a game gets an 8, it is compared to other games which got 8. And, surpriiise, chances are a lot of the other games which got an 8 are crap in comparison to this particular one. Which obviously is the case with MGS4. There is no need to be a fanboy to notice this.
09/06/2008 at 12:36 Dinger says:
First, numerical scores are bullshit. Second, numerical scores are extremely useful. Third, they would be more useful if people could agree on what they meant.
They’re BS. I’ve graded university term papers: basically, a term paper is an exercise where all the students write the same time of text on one of a few themes. They get a letter grade (US system, from F to A+). But we found that it’s much more effective to give a percentage grade: there’s higher granularity and an aura of precision. But it’s still BS. I can tell, with reasonable accuracy, what constitutes a B- and what constitutes a C+, but there’s no practical way I can distinguish between a 80 and an 81.
Okay, you scientists will say, that’s perfect: you’re measuring to one order of magnitude beyond the gradation of your instruments. Maybe. But maybe the +/- constitutes that order of magnitude. Anyway, the point is, the hard numbers somehow make the grade they received feel more real, and give me more room to negotiate. (“Okay, so I missed that. I’ll give you an extra point”). You also have to “sell” whatever grade you give. A C+ needs a lot more red ink than a B+, even though it’s not the individual faults, but the thing as a whole that makes it a C+.
Alright, to videogames. Yes, numbers are BS. Portal has a metacritic score of 89 and Bioshock gets a 96. AudioSurf clocks in at 85. Does that seem right to you?
But numbers are useful. They do force the reviewer to consider the game in relation to itself and and others. The problem is that numbers do not all fall into the same review mechanism.
Some folks I suspect are reviewing on a 1-10 scale, where a bad game scores 3 or below. Others are reviewing based on the A+ to F scale. The Onion AV club does this letter scale explicitly, which metacritic then translates into its percentage scale. Other reviewers following a percentage scale tacitly follow the A+ to F scale. So, for pedantry’s sake, here’s what those numbers mean:
<60% = (F)ail
(0,1,2 are -, and 7,8,9 are +)
60-69 = D (Barely passed)
70-79 = C (Average)
80-89 = B (Good)
90-99 = A (Excellent).
You can see that this would also translate to a star system. If you want to apply this to games and criticism, you need to pick internal criteria. A game that genuinely blows, from idea to execution to packaging, gets an F. One that barely made it to market, with major sections missing, and presents something substandard in its genre, gets a D. Standard fare is a C. Something that executes well on its ideas, and presents something that sets the standard for its class, is a B (or one Michelin Star). And truly exceptional experiences get As (or two/three Michelin Stars). Games should follow a standard distribution with 79 as its centerweight.
That’s the theory behind one approach. Numbers can’t tell you if a game is fun or not, but they can report on the craftsmanship, completeness, and relative interest of a title.
They shouldn’t correspond to sales; who wants to read a review to know how well something will do on the market?
But why mix numbers that score games according to different gradations? If that’s the case, then the result is a meaningless measure of anything but market hype. And at that point, all your Metacritic score is doing is closing off the market to any but the big players, who will have the resources to make sure they efficiently achieve the highest scores possible.
And tying bonuses or online availability to metacritic score is stupid. The last thing this evaluation system needs is to be tied directly to the developers’ economic well being. Yes, critics have always had this role, but directly linking the two will guarantee that the numbers in question take even less meaning.
09/06/2008 at 12:40 Lorc says:
Dinger: “First, numerical scores are bullshit. Second, numerical scores are extremely useful. Third, they would be more useful if people could agree on what they meant.”
Best, shortest and most accurate summary of the issue I’ve ever seen.
09/06/2008 at 12:45 Butler` says:
Alright, to videogames. Yes, numbers are BS. Portal has a metacritic score of 89 and Bioshock gets a 96. AudioSurf clocks in at 85. Does that seem right to you?
Honestly: yes, it does. I couldn’t really argue any of those “scores” one way or the other.
But yeah, I suppose the overriding issue here is a lack of consistency between publications to make things like Metacritic more useful. Which is never going to happen. :[
09/06/2008 at 12:48 Ben Abraham says:
Dinger, are you channelling the black assassin from Firefly? =P
09/06/2008 at 12:57 Butler` says:
One thing I’d like to add is that fixed “scores” really are a major issue when it comes to MMOs, as one of RPS’ guys (can’t remember which one of you is doing the PCG review of AOC…) will probably attest to.
A static score thrown at something that changes so dramatically from its original form really is abstract and meaningless. Funcom are cranking out patches incredibly fast and making real, tangible changes to the game – making scoring it even harder than normal.
I’d hold KGs 8/10 WoW review up as evidence, except WoW was NEVER an 8/10 game imo! :p
Eurogamer’s re-review system is, of course, aiming to counter this somewhat.
09/06/2008 at 12:58 Kris says:
@ Dinger’s comment. Do you blog sir? You may want to consider a change in career if you dont. One of the most insightful and succinct critiques of scores I’ve seen. for my part I do pay attention to scores, but I’m more aware than ever that I can only at best refer them to scores by the same publication. Even so, changes in editorial team, style, etc… and the passge of time can still render such comparisons moot. Seen far to many end of year / all time greats lists that seek to modify or justify a score issued by the publication previously.
09/06/2008 at 13:17 Noc says:
I think part of the problem has nothing to do with numbers at all: it’s that people don’t look at reviews as “Some guy’s thoughts after playing the game,” as (arguably) a reasonable person would do after reading something written by an individual human being about something they’ve seen. Maybe this is part of the system’s fault, as the writer is deemphasized in favor of their position as the mouthpiece for the publication.
So it becomes “Eurogamer said,” or “Gamespot said,” which leads to the idea that the review score is the OFFICIAL OBJECTIVE JUDGMENT – which, by virtue of it’s ostensible position as the be-all-and-end-all measure of a game’s worth, makes it easy fodder for argument. I’d compare this to, say, the Zero Punctuation bits, which are treated as “Oh, Yahtzee didn’t like this game.” As opposed to “This game only got an 8!” The scores, I think, help facilitate that, but they aren’t the root of the problem.
I don’t think, in the context of reviews themselves, that the numerical score is necessarily a bad thing. As mentioned above, it’s a summary, and there’s nothing really wrong with that. But I DO think that maybe some time away from scores would be good for readerships as a whole. You know, until they’ve learned their lesson.
And that’s my completely unworkable idea for today.
09/06/2008 at 13:28 Rook says:
I always find that people whining how scores should go away and everything would be better are being pretty silly. If you’re someone who doesn’t want a score on your reviews, just ignore it and read the review. If you just want a score then there it is. If you want both then that’s great too.
This idea that I should want less information in a review is pretty silly. And the idea that I should care how other people react to it (OMG TWILOIGHT PRICNESS 8.8 FAIL!!!!!!) is similarly nonsensicle (what fanboys do in the comfort of their own home is none of my business).
09/06/2008 at 13:30 Naurgul says:
I’m strongly against this whole “reviews are only opinions and are subjective” turn journalists have taken lately. Let me explain myself. An opinion, on its own, is meaningless. Anyone can have one and not matter how much of a “professional” one wants to think himself, one person’s feelings after experiencing a game do not have any real value.
When I read a review, I want it to to be as objective as possible. I don’t want excuses like “there’s no such thing as objectivity” and “games are too complex to express with just one parameter”. With that said, I’m not asking anything impossible either.
There is certain information that one can extract while playing a game, certain objective properties of the graphics, the gameplay, the story. A subset of those can inherently be thought as positive or negative. For the rest of them, tastes may vary.
So, my ideal structure of a review would be something like this: First, write what the reviewer’s expectations were from the game. This would help me put into context the inevitable subjective things that will creep inside the review. Then, a break down of the game’s properties, including how they work together, preferably separating the things that are usually thought as positive, the things that are usually thought as negative and finally: everything else. The review should end with an overview of the game, with the reviewer’s opinion and feelings (the only place where he should consciously put them). I guess the review could be supplemented with a brief list of major positive and negative things, but no numerical score. For the love of hate, please spare us of that.
So, in conclusion: Dear reviewers: This is your job. Take it seriously. If you’re not sure about something or just “felt” something, please play the thing again or think about it more or ask another gamer’s or the developers’ opinion. Don’t just put it on the untouchable pedestal of “this is my opinion” and call it a day. Frankly, that’s just sloppy.
09/06/2008 at 13:35 Meat Circus says:
@Rook:
It’s not me I’m worried about, it’s them.
Scores feed them, nourish them. We shouldn’t be encouraging them. We should probably be sterilizing them.
09/06/2008 at 13:48 Ginger Yellow says:
Nuargul, what is objective about gameplay or story? Some people love turn-based games and others wouldn’t touch them with a bargepole. Some people want their shooters to have intricate plots and others just want to shoot lots of aliens. Some people will find a particular story engrossing while others find it trite – especially in games. Even graphics aren’t particularly objective in that art style wins over technical prowess for some people and not for others. This isn’t to say that reviewers should channel Hunter S Thompson, merely that reviews are inevitably subjective and it’s best for everyone to recognise that and make the most of the medium.
The same is true of film or music reviews, and nobody finds that objectionable. The issue is complicated in games by the fact that there has to be a consumer advice element – does the camera work properly, how intuitive is the interface etc. But for the most part it boils down to personal enjoyment in the context of other games. That’s never going to be objective.
09/06/2008 at 13:54 Cooper says:
@Naurgul
Would you expect the same from reviewers of books or film. Do you think those can be broken into constituent parts and analysed objectively? I think many people, especially those whose job it may be to quantify such things, would disagree.
The last thing I want from my film reviews is a descent into the technicalities of camera work, editing and Sfx. Those are the most easily quantifiable parts of films. Sure, they should feature – the same way graphical fidelity, sound effects, stability can be with games. But there’s a whole lot more to visual and textual media which is not quantifiable. Even more so with games, where the nature of interactivity means people engage with the games in a variety of ways, where traditional modes of textual analysis can not be employed as the consumer cannot be considered passive. I fail to see how, for example, narrative techniques can be broken down into ‘good’ and ‘bad’, or a contimumn thereof. Cut scenes; objectively good or bad? SS2/Bioshock audio logs – good or bad? etc.
A reviewers job, for me, is to provide an opinion – not an unbiased one – no such thing exists – but an opinion with the weight of accumulated knowledge and love for the medium and the ability to write engagingly and succinctly behind it.
09/06/2008 at 13:55 Pod says:
Actually, I believe it’s 73%.
As for reviews, I’ve always rated things like “The Big Game Hunt” in PC Gamer and the equivilent as being the indicator of the “best” games, rather than a 9/10. Though I did used to let out a giggle whenever I saw the old 9%. (South Park Racing?)
09/06/2008 at 13:57 Robin says:
I think it is possible to have a scoring system that meaningfully describes the level of product quality. Gamespot (prior to the redesign and staff exodus) and PC Zone (ditto) were good examples.
This is in effect how Halliwell’s system works, if you map stars to 8.5, 9.0, 9.5 and 10.0, and can be (and has been) applied as stringently.
Edge’s system, where scores are fixed in perpetuity and try to gauge if a game has DONE AN ART (leading to a mass of 6-8 scores of games with wildly varying levels of quality, because most games are not developed with the intention of meeting Edge’s approval) is less useful to the reader. The top score isn’t consistently defined and is handed out so rarely as to be meaningless.
Eurogamer’s scoring system is just random noise used to seed more random noise in the comments, of course.
09/06/2008 at 14:04 fluffy bunny says:
“<60% = (F)ail
(0,1,2 are -, and 7,8,9 are +)
60-69 = D (Barely passed)
70-79 = C (Average)
80-89 = B (Good)
90-99 = A (Excellent).”
This is the biggest problem I have with review scores right now. I didn’t even know some sites did this kind of thing until fairly recently, but now that I know it certainly does explain why some (mostly US) sites always seemed to give games higher scores than I felt they deserved.
The main problem happens when scores following this scale are mixed in with scores following the “proper” 1-10-scale, where 5.5 is average on metacritic and gamerankings, and there’s no way for the user to know which of the scores use which kind of scale without checking up on every single site.
So when I see a game get 7/10 from a large number of sites, I think “hey, that must be a pretty good game”, as a seven is seen as a good score on the sites that I read. Then it turns out that half of these sevens are from people who think the game is average at best. It’s just silly.
09/06/2008 at 14:09 ACK says:
@naurgul: A review is not by definition a piece of journalism, although the definition of journalism seems to be a thing in flux, one I like is “writing for presenting bare facts to describe news events”. Agreed, the motives and tastes of the reviewer should be, or become, apparent in the course of the review. But the most important role of a reviewer is disclosing and motivating his/hers opinion of the creative work he/she is reviewing. If the tastes of the reviewer is apparent I, as a reader, can judge how well that reviewer’s tastes correlates with mine, and thus if it is probable that we would agree on the quality of the work.
Well that’s my opinion at least.
So far I’ve only found a single publication that consistently employed writers with the ability to do just that, and that was Computer Games Strategy Plus (later renamed Computer Games Magazine), RIP. It’s slowly getting better though; the people here at RPS, some stuff over at the escapist and other places, makes reading reviews good again.
Games journalism still seems to be a bit of a free-for-all though, with a bit too much emphasis on “the latest news first”, and not enough critique of sources etc. RPS et.al seems to do better at this too though. One could almost be led to believe that people doing it for love of the “art”-form as opposed to a career move have higher integrity. Who’da’thunk?
09/06/2008 at 14:25 Naurgul says:
ACK, I agree that getting a view of the reviewer’s motives, expectations and tastes is a definite step in the right direction, but I feel that we shouldn’t stop there.
Ginger, here’s a partial list of objective or almost objective attributes which can be thought as negative or positive, divided in broad categories (which themselves are partial as they do not fully cover everything in a game by a long shot)
Graphics: animation, texture resolution, face-posing, UI intuitiveness, scalability, performance…
Gameplay: pacing, intuitiveness, originality…
Story: Writing, originality, understandability, emotion-evoking
Misc: how well the graphical style supplements the gameplay, how well the gameplay is merged with the storytelling, modability…
Cooper, yes, I would expect the same from book and film critics. I personally think that the critics of more mature media are purposefully obscure in their criticism to conceal the shortcomings of their subjectiveness and I’m afraid that game reviewing will end up the same. With that said, I agree with you that breaking something down in its parts is not fully representative of the whole, this is why I said that a summary with an opinion in the end is welcome. In retrospect, I think it’s better to have an objective overview of the parts than a subjective overview of the whole.
As far as quantifying things goes, I don’t actually expect them to give out scores for these things. They can say things like “The writing was bad”, “the end of the story was original and unexpected” “the gameplay was at parts repetitive”, “the graphics work well with the gameplay”, “the UI is annoying”.
My point is, these completely subjective comments about how the reviewer felt while playing and how awesome everything is should not be in the spotlight, although they do merit a mention. Also, I don’t say that it’s not worth anything unless the end result is completely objective, but there should be an honest attempt at that. Instead of, you know, saying “Deal with it, it’s my opinion”.
09/06/2008 at 14:31 John P (Katsumoto) says:
What I hate is how most of the people on that MGS4 thread are saying things like ” -WHEN- I play this it will completely blow me away this review is shit”. None of them have even played it yet, but they’re inclined to believe the hype more than a professional review.
I’ve just seen similar things over at EDGE’s website. Horror, really. it’s almost as bad as reading Have Your Say over on the BBC website.
09/06/2008 at 14:34 Thiefsie says:
It’s funny because I have only consciously approached this topic in the last couple of years, whereas before I had an inkling of what we were all thinking but did not succinctly come to a conclusion or simply care enough to get past basic review systems. Very much so due to the blog intensity of the internet these days my connections to certain particular reviewers are a hell of a lot more important than any review score. I used to trawl gamerankings and would pick all the games above 80% for a system and set out to play or at least try them all, but realised this would still largely fail in consistently getting me games I would like – enter the subjectivity. Nowadays I find the reviewers I tend to agree with and stick with them and generally throw all other reviews away and just read them for interest’s sake. Do that enough and you can tell when a writer is easily sensationalising or pandering to hype or other pressures and generally not being honest. The Rez review is a good example. I think he wants that game to do well as it serves a nice nostalgic element of desire and would possibly bring back a more pure semblance of gaming as opposed to the current mainstream… but still Rez HD is not a 10 game in any way shape or form?
I’m mature enough to be able to judge a game generally by its own on paper merits, appearance, and other qualities to decide whether I like it or not. Rarely is a game a surprise like of mine, which is very unfortunate. The only place this probably doesn’t hold up is in multiplayer, where I may genuinely tend to rely on reviewer’s or word of mouth opinions.
It’s hard these days especially with how much notice I take of the games industry but the best thing to do is just ignore most of the hype and take things on face value, decide whether there might be some sort of value to you personally, and then going from there with a purchase or not. At least that way I do like most of my games, if not absolutely adore them. Can’t say I’ve had any duds lately, except maybe Audiosurf… (that being said I’m willing to take more spontaneous risks on cheaper games of course).
Omega Five isn’t a great game, but for when me and a mate want to play together on the one xbox it is the first thing we reach for, so is fantastic for us… and no review made us decide to give it a try or not?
If only the masses were mature enough to make these decisions themselves we wouldn’t have over 2 thousand posts to a MGS4 review, which I think anyone could bluntly say; if you like MGS you will love MGS4, and if you hate MGS you will hate it too… And if you are on the fence… well… you’re probably better off starting elsewhere in the series to whet your appetite for what you could be in for – such is the nature of heavy, story based, and self-referential game.
I think a simple yay or nay is the best way to measure a mark for a review, and the reader can then align themselves to whether or not they have a taste for the game within the writing.
It will never be an exact science, and unfortunately for someone to seriously rely on a number out of 10 for a purchasing decision seriously needs to think about what they are doing. (OR at least if they are hurt by a game receiving less than they think it deserve, or more… to start rambling on about the injustices of it all)
The saddest part of this of course is the way that metacritic and gamerankings are assuming some across the board consistency with reviewing, which obviously does not exist. How many times has ign pulled a review? or lest we forget gamespot and it’s shenanigans. The hardest hit by point systems are probably those development teams relying on a points grade to secure their next investment or reward, even more so than pure sales or other more tangible quantifications.
I read enough of the gaming media to have a pretty clear cut view of what a game is going to be like and what I can expect from it. and thus don’t need to rely on scoring. Unfortunately this is of course in the minority.
Demo’s are of course the sure-fire way around this… but in today’s current state, many more companies are purely just too chicken-shit (and maybe rightly so?) to release any demo form of their game. Assassin’s creed for example, looked absolutely brilliant leading up to release, and frankly I think a demo could have only helped their sales even more, as a demo would never have revealed how shallow a game it actually is, conversely making everyone who played it think of the possibilities… which they genuinely didn’t follow up on past the first assassination.
I’ve rambled on for too long and am thus sorry if you’ve made it this far.
I do know that I like reviews that are more colourful and personal, as a form of entertainment and maybe light judgement, but nothing more. Objectivism is boring, and can be found quite easily just from reading about a game in various media. A mix of the two is of course the perfect balance… but very hard to find. Many reviewers go a bit too far into the ‘high writing’ territory, as I think can be found at Edge quite easily for instance, and others just pander to the lowest common denominator, like IGN or perhaps Gamespot. Yet others give outrageous scores just for pagehits and sensationalist outlashes.
Most of eurogamer’s staff straddle the line quite well i think, and of course not surprisingly the staff here do it well, with the review system and Optimus’ thumbs being a great indicator of purchasability.
Most reviews are a pissing contest and little more.
It’s just unfortunate that the mainstream (casual?!? haha) loves to follow along with the contest. In the eurogamer posts in response to the mgs4 review, it’s almost as if that 8/10 score will stop people buying the game as opposed to if it was a 9. Wow, with metacritic… maybe it will?
09/06/2008 at 14:40 alco75 says:
I wish for all games to be rated on a two-scale system:
1. KG recommends
or
2. KG does not recommend
09/06/2008 at 14:46 Thiefsie says:
But they are right! Of course they are going to be blown away, it’s MGS, a game that is so individual that they can’t not be blown away if they know they like that kind of stuff. Of course they will like it! They are showing how much the review doesn’t matter to them in liking the game or not.
The problem is of course the need for them to stick up for a friggen number out of 10, that if removed, would probably remove 1800 of those posts, or at least promote some more healthy discussion of the actual game.
Or the unfortunate point of view that people out there are actually thinking hmmm it’s an 8. If it were a 9 I’d get it but for now.. nope
09/06/2008 at 14:48 Meat Circus says:
@Alco75:
Makes it a rather unlevel playing field for the small majority of people who are not Kieron Gillen.
I’m personally very happy with the OPTIMUS PRIME THUMBS OF ULTRO-TRUTH.
09/06/2008 at 14:49 Meat Circus says:
@Thiefsie:
Or the unfortunate point of view that people out there are actually thinking hmmm it’s an 8. If it were a 9 I’d get it but for now.. nope
And still, we allow these people to breed.
It’s political correctness gone mad.
09/06/2008 at 14:54 James G says:
The best review is one where you can read it and go, ‘Well, Walker clearly hated this but, judging by what he said, I reckon I’d enjoy it,’ and be right. (Or vice versa obviously) Of course, it helps when a publication matches a reviewer to the game. Why some magazines assign a strategy game to someone who hates the genre is beyond me. While it may have some advantages in the case of the genre leaders, which might attract in outsiders, it is marred by inexperience, and by making it difficult to seperated hatred of the genre from dislike of the game. While there may be benifit to criticism of genre tropes, it is misleading to shoehorn these into a single game review when they obstruct criticism of the game itself.
I must admit, I occasionally use metacritic if I’m wanting to get a quick feel of a game’s reception, either because its a new release, or because I’m new to the format. (For example, when I recently cot a DS, I used metacritic to get an impression of which games are work considering, and which bombed.) I’ll then follow it up be looking at reviews, forum postings and the like.
09/06/2008 at 14:54 Okami says:
@Meat Circus: If we wouldn’t allow stupid people to breed, we’d have nobody to make all the stuff like houses, food, money, power, tv, cars, roads and everything else.
09/06/2008 at 14:55 Meat Circus says:
@Okami:
Nonsense. A mixture of special baby-growing jars and homicidal deathbots will do the trick.
09/06/2008 at 15:07 Thiefsie says:
Pure example of journalistic puffery for the sake of ‘excellent writing’
8/10? ha!
09/06/2008 at 15:09 Dinger says:
Yes, I was thinking of Jubal Early.
objective/subjective is an easy-to-grasp concept, and is useful only as long as you don’t scrutinize its meaning too closely.
If the experience of games were entirely personal, there would be no point in talking about them. There must be some shared element that serves as the basis for discourse. I’m a fan of experiential exposition, which is why I read RPS. But if I were to quantify a game, I would do it along the lines of asking “What does this game set out to do?” and “How well does it succeed at it?” That would never be enough, so I’d have to toss in something about the overall aesthetic experience, and that’s where it gets messy. But generally, the question isn’t “Did I like the game,” but rather “How will my readership receive it?” If you write pompous, turgid prose, your readership will appreciate certain manifestations of the ludic art. But if you keep it real. It’s gonna be sumthin else.
So it doesn’t have to involve “subjective experience.” It can be an honest assessment with reference to what the game is, what it seeks to be, and who would want it.
P.S., Overusing the passive voice doesn’t constitute “Journalistic Puffery”
P.P.S., They can say things like “The writing was bad”, but if they do, they’re not doing their job. Much better to say:
“I walked into the operating room; the nurse came up, and with perfect Oxbridge enunciation said, “Mr. Doctor, the sufferer stands in the ante-bureau.” Evidently Babelfish handled the localization.”
09/06/2008 at 15:12 Ginger Yellow says:
“Edge’s system, where scores are fixed in perpetuity and try to gauge if a game has DONE AN ART (leading to a mass of 6-8 scores of games with wildly varying levels of quality, because most games are not developed with the intention of meeting Edge’s approval) is less useful to the reader.”
Yeah, but do people who read Edge pay any attention to the scores? I suspect they only include them because it’s the done thing in games publishing.
Naurgul, at least half of the things you mention in that list are far from objective, and the rest tend to be included in subjective reviews anyway (apart from texture resolution – are you taking the mickey?).
09/06/2008 at 15:39 Kieron Gillen says:
Alco75: You speak truth. We’d all be a lot happier if you just submitted to my diktats.
Naurgul: Perhaps surprisingly, I’m not entirely without sympathy to your position, but when you’ve got things like “Emotion Provoking” on your list of objective things, you have to realise you’ve gone awry somewhere. I take the position that James mentions – that a review should contain enough enough objective description of the game’s experience that you can decide for yourself whether it’s something that interests you. Even when I slag a game, I try to explain what it’s like and how it actually operates.
(In terms of technical elements, I tend to take the approach that if something is exceptional (either good or bad) it’s useful information and should get a mention. With a limited word-count, omission tends to mean “standard”.)
I also slightly bristle at the idea that people who let subjectivity inform the review aren’t taking it seriously. Because frankly, I suspect the opposite is the truth. Writing a list of the games’ qualities is fucking easy.
Thiefsie: When the game you’re reviewing has two hour-plus cut-scenes, I think a little florid ending is probably the way to go. For the record, the MGS4 review situation – specifically, the response Ollie has had to deal with, absolutely disgusts me. People are pilloried for giving the big games 10/10 and when someone doesn’t, they’re crucified.
KG
09/06/2008 at 15:46 Cooper says:
I hope you’re playing the wind up merchant, Naurgul. Half of your examples are from from easily objectifiable, the other half I would venture cannot be validated in any unbiased manner at all.
I would love to hear how emotion-evoking can be measured or be considered in an objective (and as far as I understand it, you’re taking objective as in ‘unbiased’) manner?
As far as Edge readers go, I still skip to the scores before reading the review, but only because of habit. Not that they are anymore or less meaningful than elsewhere. At least Edge wears its artsy fartsyness on it sleeve – and takes a genuine interest in games as a medium, rather than as products. I read it, knowing what kind of angle I’ll be getting, and ‘cos of the pretty pictures.
09/06/2008 at 16:18 Naurgul says:
Just to play with the specific example. I understand that emotions and logic are two separate things but that does not say anything about being able to evoke emotion not being an objective thing or a process. Don’t advertisements by design evoke emotions; desire for the advertised product to be specific? Emotions are an action-reaction thing. Certain stimuli can produce certain reactions from humans. Emotions can be produced that way. But I digress. Aren’t books and films out there that bring out emotions out of the majority of their readers or viewers? Can’t that be an objective measure of how capable something is to produce emotion? I’d go as far to say that this is one of the easiest things to measure. You know when you feel emotions and you can count how often it occurs in an audience.
To summarise the flaw in your argument, I’d say it’s a straw man. My position was “The capability of a piece of art to produce emotion is objective” (at least up to a degree, I didn’t say anything about precision or quantifying or even comparing, to be fair). To prove that wrong, all you did was say “Objective things have to do with logic, emotions and logic don’t go together; therefore the proposition is wrong”.
The examples are just examples. I have no wish to defend them on a one by one basis but I will do so if you need it so much.
I really wish, however that reviews were just a list of well thought-out, as-much-objective-as-possible values assigned to attributes. I think it’s far from easy; that’s why instead of thinking things over, reviewers just write down what they felt about it. I don’t want to know your opinion, I just need enough information to form my own.
PS: I’d love to see how things like gameplay originality and graphics scalability can be “far from easily objectifiable” at best, as Cooper seems to imply.
09/06/2008 at 16:24 Meat Circus says:
@KG
Thiefsie: When the game you’re reviewing has two hour-plus cut-scenes, I think a little florid ending is probably the way to go. For the record, the MGS4 review situation – specifically, the response Ollie has had to deal with, absolutely disgusts me. People are pilloried for giving the big games 10/10 and when someone doesn’t, they’re crucified.
To be honest, Kieron, I rather think that Eurogamer bring it on themselves.
First, EG takes to pretending that it has some ‘fair and balanced’ scoring system, and puts a few “low” scores on games like Gears of War until everybody starts believing them, and then suddenly ruins it all by spunking 10s on Oblivion, Halo 3 and any other bit of overhyped shit that comes its way.
And the situation could be completely avoided by not requiring the reviewer to offer a mark out of ten. Even the breathless fanboy frottage of Rob Fahey’s Halo 3 review was (slightly) more subtle than the OMG 10/10 LOL mark implies.
09/06/2008 at 16:35 Ergates says:
With Edge and Gamer I’ll generally look at the score first. It doesn’t really tell you anything about the game, but it does give a general impression of whether or not the reviewer liked it – sets a ‘tone of voice’ for the text if you like.
As for the response to the MGS4 review – disgusting? Yes. Surprising? Sadly not. Some games series seem to attract the worst of the rabid fanboyism (whereby people who don’t adore the game are not just wrong, they’re subhuman) – and MGS is one of them.
It seems to be an innate need of humanity to have someone to hate, preferably someone different – people who look different, people who believe different things, people who don’t enjoy hour long cutscenes, etc…
09/06/2008 at 16:39 cliffski says:
Nobody enjoys hour long cutscenes. Except the people who make them.
surely?
09/06/2008 at 17:05 Rath says:
One thing I think people are overlooking is that everyone (that I saw here, anyway) who is arguing for numerical scores also says that you have to get to know the reviewer or group to get a feel for what that score really means. That is, the score itself is – at best – meaningless without the context of the reviewer’s background.
Well, I have to agree with that assessment, but step back and assume the role of the “average consumer” for a moment. NB: If you’re reading this, you aren’t in that group.
Especially now that MetaCritic is taking over, it’s highly unlikely that the average consumer will ever get to read a given review or know where that 67.3 came from, much less build a rapport with the writer based on a series of reviews and shared opinions. Even on a given site, it’s not always a given that you will know who wrote it, what other games they have reviewed, and what direction they are approaching the game from. I’ve read hundreds of reviews myself, and I could only name a handful of reviewers. The best reviews will work hard to establish that context within the review itself, but the number, of necessity, doesn’t contain any of that.
So, without that context (which isn’t that simple to create in the first place), we have just removed the one thing that makes that number/letter meaningful to the consumer. Why, then, do we want to keep the “scores” around?
09/06/2008 at 18:15 Cooper says:
Naurgul
Taking here objectification to mean an allusion to a measurable reality of a game’s attributes, I couldn;t really disagree more.
To run with the same example; it’s not about logic, so much as irreducibility. Emotions, as much as they are biological, are not easily reducible to a set of measurable phenomena. Sure, measurable phenomena such as hormonal levels, heart rate, neural activity, etc. are indications of the affective power of cultural artefacts. I’d hardly expect (or want) game reviewers to wire themselves up to measure these phenomena when playing games. More importantly, the affective power is not reducible to those. Neither is ‘counting the number of affected poeple in an audience’ a desirable measure, given that those affects will vary widely and, anyway, what does it mean, in a form whereby all the respondants will be answering the same question, to ask (as opposed to propose) ‘are you emotional’? Comparability, largely the point of quantification in relation to game scores, becomes nigh meaningless in such a situation. Naming games, or a game’s attributes ‘Good’ and ‘bad’, or on a scale thereof becomes meaningless in the face of fanboyism, languages, personal gaming histories, genre (game and setting / narrative) likes / dislikes etc.
And, remember, we can discuss emotional engagement because we’re gamers. We, to various extents, share forms of cultural knowledge, which not only shape, but largely determine, emotional engagement in games. Without that, things would be very different.
Ok, so that’s pulling part one example, but similar arguments about cultural knowledge (which differs between gamers as much as between gamers and non-gamers) makes quantifying UI intuitiveness, pacing, ‘understandability’, coherence of narrative, immersion etc. etc. just as fraught. Moreover, it’s no small cliche to say no one plays the same game. As with film, no one watches the same film – we bring all manner of memories, affective dispositions, cultural knowledges to the cinema which shape our engagement. This is doubly so with games, whereby the interactive, feedback nature of the media is not hidden, but largely the point of it.
Ok, backpedal somewhat. There is clearly more common ground upon which critique is based than I am suggesting. If experiences were as fragmentory and disparate as I’ve made out, game reviews would be even more pointless. But the cultural knolwedges which makes us ‘gamers’ as thus able to discuss games with a certain level of familiarity – the ‘common grounds’ upon which an objective review may strive for are shaky, unfirm, nebulous things. We can allude to them, we can recall them, we can aim for them, but they cannot be fixed, written or known. Though it can be alluded to, in order to give some semblance of coherence and comparability (and to give game reviewers a job); there is no common ground between gamers stable enough to be the lode stone against which games may be objectively measured.
As for scalability, performance – those are more easily measurable in the traditonal sense. I would have no problem with frame-rate analysis of games over various system, other than it’s pretty damned dull.
Sound effects? Objectified how – by closeness to reality? What if a game isn’t striving for realism? How do you measure the ‘good or badness’ of the sound effects for peggle?
Ditto textures, level design etc.
In such instances, being, or appearing to be “as-much-objective-as-possible” is not only fallacious, but misleading. It purports a knowledge, an unbiased, non-subjective measurement – an insight – by which a reviewer may have the final word on a game. We know that’s certanly not the case.
09/06/2008 at 18:28 steve says:
I wonder if there’s this much hand-wringing over scoring movies and such at non-game publications? For some reason, people make a big deal about this as it relates to games, but don’t seem to care that almost every movie and music review includes a score.
09/06/2008 at 18:48 Kieron Gillen says:
Steve: I wonder about that too. There certainly isn’t in music where I do stuff*. Is it an audience thing (i.e. Games players are a little more anal than most consumers) or is it something about the medium (i.e. There’s more in a game that people think can be measured objectively)?
I dunno. It’s a kettle of the proverbial fish.
KG
*Admittedly, the only place I write music reviews for now doesn’t use marks full-stop.
09/06/2008 at 18:51 Ted says:
With most PC releases these days being console ports, it’s really crucial these days to have a review that does something as simple as tell whether the PC controls actually work. Lego Indy they’re totally broken like all the other Lego games, that was all the information I needed on that one. Assassin’s Creed, they’re crap, again all I needed to know.
09/06/2008 at 20:05 Simon Parkin says:
Steve and Kieron: I think some of it has to do with the fact that videogames are *our* medium. Music and films have had earlier generations who have wrestled over what makes canon; what the rules and language and boundaries are when establishing the good from the bad and the ugly. But games are so young, and there’s so much crossover between game culture and internet culture (i.e. the demographic that primarily use the internet also play games – see Digg.com where videogames are one of the eight sole categories into which ALL NEWS must be filed under) so I think everyone feels like they want to be a part of that process. Maybe. I dunno, I’m thinking out loud here.
I’d also just like to point out that the impetus for the original article was MS’s decision to use aggregated and averaged scores to decide the fate of titles on XBLA; the point being that, if we all agree different publications use scales in different ways, then surely those numbers aren’t so much the wisdom of crowds as the foolishness of random data. Nowhere is that better exemplified than in the example of the reviewer who gave Penny Arcade Adventures a 4/10 for Edge and a 68% elsewhere. I think, because of that decision, of all this discussion and more is worthy.
09/06/2008 at 22:57 Kieron Gillen says:
Simon: Yeah, but previous generations didn’t get so hung up over bloody numbers. What is it about games and gamers which reduces it to arithmetic? It’s an interesting difference about the form. Other medium got hung up on the discourse – we don’t, so much.
KG
09/06/2008 at 23:03 Noc says:
In reading through these comments, I’ve been thinking back to art school critiques, wherein the most damning thing I could say is also the most vague:
“I’m not really feeling it.”
Now, this would usually, if the critique is done right, be accompanied by other, more technical things. The colors HERE aren’t quite right. This bit’s foreshortened weirdly. The contrast between the figure and the background kind of make that guy look like a phallic silhouette. But either way, what it comes down to in the end is either “I like it” or “I don’t, really.”
Now, THIS is a real thing. The effect of a piece, on me, is real and (albeit vaguely) quantifiable. A good critique, and by extension a good review, looks at this impression and analyzes it, and tries to figure out which elements of the piece contributed to it. You might call this subjective, but I’d call it specific. The effect the piece had is specific to me, and is a result not only of the piece itself but of my own history, tastes, and prior experiences with the medium.
If I do a good job, I’m looking at that effect objectively. I’m figuring out, and then pointing out, what it is about the piece that affected me positively and negatively. I’m also pointing out what aspects of my own personality interacted with these, so the artist I’m critiquing knows how I fit into his idea of his intended audience.
But still, none of that changes the fact that the effect of the piece is specific to me. Because the effect of a piece is dependent on how it interfaces with me, and I’m the only person I can speak for. A video game, as a form of entertainment (art, whether it’s got a capital or not) is trying to create that effect . . . and I’m the only affected person I can speak for.
So, Narugul: even in a perfect, ideal world, you will never get an “Objective” review of the game’s merits. This isn’t even a matter of “Oh, everyone’s subjective to some degree” . . . the very thing that the review is trying to MEASURE is specific to the reviewer. This examination can – and should – be done as objectively and logically as possible. But all a reviewer can do is say “These elements worked for ME, and these didn’t,” because (again,) the success of the elements is contingent on them interfacing with him, and he’s the only mind he has access to.
And through that, as James G said, you get an idea of what the reviewer is bringing to the table, and how similar or different you are from them. And from that, you get a good idea if the game is one you’d enjoy or not. Granted, this sort of analysis of a review is something that requires a little bit of thinking, and it would be much, much easier if someone could just give you a judgement that you could adopt wholesale.
But until our culture becomes at least a LITTLE bit more of a dystopia, that’s not really going to be possible.
And Kieron: I think that’s what it is, actually. We, as a culture, like to share opinions. Not share as in “give to other people,” in the sense that we form a sense of community based on shared views. So we’re voracious consumers of judgements; we want to be able to say “This game was GOOD. And this game was CRAP” so that when we’re talking to someone, we’ll be able to provoke a “Yeah, totally!” response and reinforce our sense of kinship.
I think this is something, if not new, has definitely grown in scope in recent years. It came with global connectivity, and the internet; we’re in social situations where the only thing we can have in common, and the only thing that defines us, really, is the views we hold on forums and comment threads. And it’s much easier to hold a number as a view than a complex and nuanced opinion: I can get a much better reaction, both positively and negatively, by saying “Twilight Princess Deserved an X.Y” than I could by saying “Here’s the things I liked and didn’t like about the game.”
09/06/2008 at 23:13 Kieron Gillen says:
Noc: Yeah. I didn’t pick up Narugul’s points as I’m a bit too busy to go too deep into it but… well, all you can do with any story story is say whether it was emotionally moving for you. I always recall someone arguing anyone who found Dance in the Dark moving was mentally subnormal, in an entirely serious way. Conversely, that was a film I needed multiple vodkas afterwards before I became even vaguely human.
In games, I laugh at – say – MGS’s extents. They’re terribly written, subnormal rubbish. They mean the world to some people.
All a critic can do is talk about the effect a piece of culture has had on them, and try and focus no the exact elements that managed to make them feel like that. It’s based on the subjective experience, but it shouldn’t be a wishy-washy “This is how it made me feel”. It should be focused, determined, intelligent, human. HOW DID THIS THING WORK AND HOW DID IT DO WHAT IT DID TO YOU? That’s what it’s about.
I’m drinking!
KG
09/06/2008 at 23:14 The Shed says:
On huge games that are massively hyped, or for games that very little info is released about (see Alan Wake), score really don’t matter to me. What matters is the content, what looks interesting, or what would be good to play. However for games that have great gameplay mechanics, but everything else is on the surface (e.g. GRID, or Gears II), a score can help you decide whether the game is just worth it or not. The new PoP I would definately read a review or three of, but Battlefield: Bad Company I would generally glance at the review and check the score rundown of, if it got repeated good 8′s or 9′s, I would consider buying it.
Now that our blog is up, and we’ve started on reviews, I’ve found that scores seem needless and trivial once you’ve written an entire review. However- so far I’ve only written film and music reviews; in past game reviews I’ve done, a score seems much more fitting. (Byline: I’m of the opinion that an out of 20 system is the most(?) strongly balanced scoring system. It has the clarity of /10 (without decimal points), and loses the pointless unit of the percentile system (what’s the difference between 81% and 83% yadda). I usually give things an /20 if it comes to it.)
EDIT: I had the pleasure of encountering a lad on the IGN boards who claimed that reviews should NEVER be subjective, and that they should NEVER be based on opinion :D
09/06/2008 at 23:20 Calabi says:
Its that bloody DnDs fault with its D20s and D4s, and Axes of 6 thoat damage.
Pcs and programmers love numbers, but they cant handle ambiguity and they try to quantify things absolutely, but they keep ending up with a point.
09/06/2008 at 23:44 Muzman says:
Damn, that MGS4 thread is amazing. I thought I’d seen everything with Zelda.
We need some version of Poe’s Law for fanboys; No parody of extreme obtuseness and outrage in the face of disenting opinions regarding a given popular computer game series or platform can exceed, and therefore will be indistinguishable from, the real thing.
Taken at face value though what they say is sort of defensive and protective. You called my baby ugly to his face, before I could cover his ears and he’s old enough to know what it means! Bad reviews are gonna damage the platform, damage the game, damage the site for being a disenting voice etc so you shouldn’t do it for everyone’s sake.
That is generally how I look at these things; fanboy-ism plain and simple. This is a tad simplistic though, as I was the opposite about Halo 3 and, to some degree, Bioshock; I was like ‘ten for Halo 3 is like ranking Independance Day alongside The Godfather’. Which of course is horribly horribly wrong and yet perfectly fine and offers no kind of argument and is exactly what people are talking about.
The score, I suspect, is just a convenient focal point for debate and as such would probably just move somewhere else if there wasn’t one (this is me pursuing the ‘blame the fanboys’ line still). The average Eurogamer thread not devoted to beloved-fan-classic-getting-too-low-a-rating contains a good deal of debate about whether the review matches the score; “That read like a seven not a six” “Man I was sure it’d get 4 after that screed” etc etc. I would be interested to see what happens if there wasn’t one. Would that mean there’s nothing to talk about? It would make it more difficult to talk about, but I think it might still happen.
Going back in time, there was that notorious Gamesdomain review of Deus Ex which basically trashed every little annoying detail about it. As we know you can’t say bad things about Deus Ex, you just can’t. So there was…consternation. Everything the guy said was true about the game; it’s ugly, it’s buggy, it’s generally clunky, runs like a dog, has quite dodgy voice acting etc etc. All things that are bound to annoy people, and aren’t reviewers people too? The reaction generally attacked the credibility of the review for failing to acknowledge the game’s good points and, perhaps most importantly, ambition (did the positive reviews pay appropriate attention to its bad points I wonder). This was a contravenion of what reviews are supposed to do, it was mean spirited and subjective. This was a site without review scores and often used multiple reviewers (this might have even been a second opinion review. Not sure). Actually they might have had a star rating at some point and certainly had awards, but the rage was about the appropriateness of the review’s content (well, the rage as projected through nerdy rationalisation of feelings). I’d be interested to see what happens if Eurogamer stopped giving scores anyway. I dunno if I’m missing the point of the discussion somewhere, but there you go.
The part about XBLA deleting low rated games is bloody terrible though. Why bother moving into the digital age at all? It’s just like everything that was lost when film went to video and when VHS was replaced by DVD. Or the race to bottom caused by Walmart not carrying R-rated content etc. They’re determined to create Gamespot-esque PR hell around every game and review with that idea. Even if ratings were somehow objective and absolute there’s been an immeasurable contribution to culture from, for one example, really really abysmal horror films. Complete idiocy.
09/06/2008 at 23:54 The Shed says:
““That read like a seven not a six” “Man I was sure it’d get 4 after that screed” etc etc. I would be interested to see what happens if there wasn’t one.”
That’s definately a good point. If there was no score we probably wouldn’t be hearing the fanboys at all- but there is a chance a bruhaha would be raised about the review’s lack of score of course (probably from the… fanboys who want a score? Damn). Also mind that MetaCritic say “If a critic does not indicate a score, we assign a score based on the general impression given by the text of the review.”
I personally love the MGS series (playing through numero 3, Subsistence again, and loving it), and I wasn’t really disappointed with the Eurogamer or Edge score. From these critics an 8 is a fairer score; it’s really very good, but not great or revolutionary. For a lover of MGS a high 9 might be appropriate, but not too much.
The score would make more sense in my eyes if it was written as a 16/20.
09/06/2008 at 23:56 Robin says:
The kerfuffle about scores for games is nothing compared to how seriously they apply to restaurants.
10/06/2008 at 02:38 Dinger says:
multiple hour-long cutscenes? That’s no game; that’s a television miniseriies event!
10/06/2008 at 13:02 Myros says:
IMO the subjective part of game ratings is where it fails. A rating system should be created and universaly used that is based on only measurable/provable criteria … let’s call it the “Myros Scale” just for a moment ;p
ie
Game originality
Game length
Stability
Replayability
etc etc
While not all reviewers will agree and come up with the same score at least for the gamer there will be a known basis for comparing the reviewers and a common understanding of what a “5″ or a “10″ means.
Kind of the Richter Scale but for games :)
Myros
10/06/2008 at 16:53 propanol says:
The point of a numeric score for many outlets is not so much to provide an aggregation of the review as to draw traffic and increase advertising revenue, which is why you’ll occasionally see IGN (and others) giving games anything but a perfect score or somewhere there around.
Most games journalism amounts to little but PR on behalf of distributors or platform holders (main advertisers) anyway, so I wouldn’t take any of it particularly seriously.
10/06/2008 at 18:34 fearian says:
Everybody should adopt the ‘One Life Left’ school of scoring games.
also, I rate this article [7]
10/06/2008 at 18:50 Noc says:
Myros: lets look at that for a second, shall we?
- Game Originality: Umm. This is pretty subjective. Check the article on Crysostasis further up the page: We’ve got a bunch of people saying “Bioshock Rip-Off” and a bunch of people saying “No, it’s original” and a bunch of people saying “It’s drawn from these different sources entirely.” Rating this, of all things, is entirely dependent on the scope of the subject’s familiarity with possible influences. So THIS varies tremendously from person to person, even from objective, well educated person to objective, thoughtful person with similarly broad horizons.
- Game length. Alright, this can be measured pretty well, in terms of hours of content.
- Stability: This is INCREDIBLY subjective, but in a different way than usual: this time, it depends on the subjects computer. Different computers that share identical statistics can have wildly differing reactions to a game, based on any number of nonstandard cards, programs running in the background, etc. You can see this were you’ve got threads full of people saying “This game keeps crashing” and a similar amount of people saying “I’ve had absolutely no problems with it.” Giving a score for this would involve a survey across a wide variety of players, and is hardly something that’s part of a review.
- Replayability: Right. This is “How much fun do I have playing the game over again?” Now, sometimes games are the same each playthrough. Other times they’re different. Either way, “How much fun I had” is a COMPLETELY subjective issue.
Now, with your list of things that are ostensibly able to be “objectively” quantified, we end up with this:
The point I’m making here is not that reviews don’t mean anything. It’s that you cannot divorce someone’s experience from them as an individual. You have to do some THINKING here, like it or not, and draw a conclusion for yourself that is likely going to be different from that given by the reviewer. A good reviewer will give you, in the review, the tools needed to do this easily without having to make too much conjecture on your own part. (“He found that the game was too dialog heavy, but I know that I haven’t found issue with extensive dialog in the past, so I should be alright” as opposed to “He doesn’t like RPGs so he didn’t like this game, but I DO like RPGs so I . . . might?”)
But you are not going to be able to have a be-all-and-end-all number that you can stick on to a game and that is that. Or even a similar be-all-and-end-all SET of numbers.
And anyone who claims to be giving one is either bullshitting you or not thinking clearly.
11/06/2008 at 08:25 The Shed says:
Good call, Noc. It’s (in most cases) literally impossible not to be subjective, to at least some extent. However, you forget that it’s only the land of PC gaming that has an issue with subjective stability; on Console, everybody get’s the same experience. That’s one of the things I feel console gaming has one-upped PC gaming on. If they were ever at odds anyway =D
I think it was a good point that scores are more objective than the body of the review though. Your words might seem biased, but surely you have more of a handle on the numbers you tag at the end?
12/06/2008 at 03:50 Nick R says:
Given Edge’s anonymity, I’m curious as to how Gabe from Penny Arcade knew that it was the same reviewer on both magazines?
One of the first issues of Edge I ever bought, back in 2003, featured an article discussing games reviewing and scoring – how no review can be called “definitive”, etc. It still stands out in my mind as one of the most Edge-y articles they’ve ever done, and one of my favourites. In the same issue, as an experiment they removed the scores from the end of the reviews, replacing them all with question marks (but they didn’t go all the way; the scores were all listed on a single page elsewhere in the issue). Edge’s 10th anniversary edition a few months later also featured an article examining the reasoning behind the four ten out of tens that had been awarded to date.
Personally I don’t think that review scores should be removed entirely, but they also don’t need any more “precision” than 5 stars – once you get to scores out of 10 or percentages, you hit that strange “seven is average” prejudice that’s hard to break. Having said that, I do like Edge’s system of having enough marks available that the the highest score can be very prestigious and rarely awarded (well until last year it was anyway :). So someone’s suggestion earlier of a Halliwell’s Film Guide style system would be interesting to see applied to games reviews.
12/06/2008 at 16:36 iainl says:
Oh so many things:
1) Yes, the fact that we seem to have had about as many Edge 10s in the last year or so as we did in the previous decade is ludicrous.
2) Everything Simon says is right, on principle.
3) Everyone has to stick a score on the end, or you won’t get linked by Metacritic. And that’s a fete worse than death.
4) Oh dear Lord, I get offended that 7 out of 10 was average. That it’s now apparently 8 (or 79%, whatever) is even worse.
5) The real problem with the Eurogamer review, as I understand it, is that it’s pretty much the entire MGS series that’s getting a ‘mere’ 8/10. By many accounts it’s the best MGS game yet. If you like MGS games, you’ll adore it. If those problems like insane plots and feature-length cutscenes annoyed you before, they’ll do it even more now. So the reviewer is damned whichever score they give, really.
I know I won’t be buying MGS4; I never made it through MGS2 as far as the gameplay (not the most sensible purchase I’ve ever made, I admit). Even if everyone had given it 10/10.
Umm, 6 I think) The even bigger problem for reviewers with MGS4 is that a series of disasters (Heavenly Sword, Lair, Haze etc.) have left it as Sony’s Great Greyish Hope. Fanboys of any nature have too much invested in the number to care what the text says, they just want evidence that it’s “better” or “worse” than Halo 3 to shout at other teenagers with different purchasing priorities.