Rock, Paper, Shotgun

Not A Man, A Number: Metacritic Rates Devs

By Alec Meer on March 28th, 2011 at 11:27 am.

Well, that told him

If it exists, it must have a number stuck to it: this is the Metacritic way. Its voracious maw of review aggregation has now expanded to include individual developers – which means actual people are now being given a personal numerical rating. This is an average number (out of 100) based on the various reviews of games they’ve worked on. I must confess I find this concept mildly sinister, but maybe I’d feel differently if I was able to go around telling people I was worth 91%.

So, who of the many alumni of PC gaming has been treated favourably by this new system? And who’s fallen foul of it? Who is empirically proven to be the better man: Warren Spector or Ken Levine? Richard Garriott or Derek Smart? John Carmack, John Romero or American McGee? And Portal’s Erik Wolpaw or Portal’s Chet Faliszek? And which poor bastard was deemed to be worth just 8%?

Here’s who I’ve looked up so far – I am, of course, expecting you to suggest further honourable gentlebeings below. This is achieved simply by typing dudes and dudettes’ names into the main search box on Metacritic.

  • Cevat Yerli (Crytek): 91
  • Bill Roper (Blizzard/Flagship/Cryptic): 90
  • Ken Levine (Irrational): 89
  • Chet Faliszek (Valve): 89
  • Erik Wolpaw (Valve/Double Fine): 88
  • Sid Meier (Firaxis/Microprose): 86
  • American McGee (id/EA/Spicy Horse) : 86
  • Cliff Bleszinski (Epic): 86
  • Doug Church (Looking Glass/Ion Storm/EA/Valve): 84
  • Warren Spector: (Origin/Looking Glass/Ion Storm/Junction Point) 82
  • Peter Molyneux (Bullfrog/Lionhead): 82
  • Chris Avellone (Black Isle/Obsidian): 81
  • Chris Taylor (Gas Powered Games/Cavedog): 80
  • Brian Reynolds (Microprose/Firaxis/Big Huge Games/Zynga): 79
  • John Carmack (id): 78
  • Ruslan Didenko (GSC Gameworld): 77
  • Julian Gollop (Mythos/Ubisoft): 76
  • John Romero (Origin/id/Ion Storm/Midway/Monkeystone/Gazillion/Loot Drop): 75
  • Richard Garriot (Origin/NCsoft/Portalarium): 66
  • Derek Smart (3000AD): 61
  • Oleksandr Khrutskyy (Deep Shadows): 61
  • Sergey Titov (Stellar Stone): 8

Phew! Much to comment on there (Garriott and Roper make for particularly interesting discussion points in and of themselves; the former’s recent games have soured his mighty past achievements, while the latter’s Blizzard mega-scores mask troubled times such as Hellgate and Champions), but let’s use as our launching point Valve’s Wolpaw and Faliszek.

The ex-Old Man Murray pair have worked together for most of their adult lives, but have different scores. Chet is officially one better than Erik. Will he gloat and chuckle and sneer about this every day? Will Erik fall into a deep, dark depression that results in him carving ’88′ into every tree trunk, car door and human face in Seattle? Who knows.

What we do know is that the disparity results from Wolpaw having worked on one game that Faliszek didn’t – Psychonauts – while Wolpaw isn’t credited as having worked on the Left 4 Dead games. The latter was generally better-reviewed than Psychonauts: hence the mystery 1%.

My fear is that the games industry might use this system as a factor when looking to recruit people or decide on payrises. Metacritic numbers have already been known to affect the likes of retailer stock buys, publishers and studios’ public profiles (“we are/want to be a 90% Metacritic average company” is a refrain I’ve heard many times of late, especially from larger devs and publishers) and even developer bonuses. I can all too easily imagine a firm deciding it will only hire or promote staff of a certain score or higher, believing that to be the benchmark of their aptitude, and thus an indicator of how likely their work is to bring in the high scores and thus the high revenues.

“Only 88%? Not good enough, Mr Wolpaw. We’re after an 89%er at the very least.” “If you’re not worth 91% by Christmas you won’t get that payrise you need to feed your 18 children, Meier.” And pity the guys with a 60 or 70 albatross hung around their necks. What if they’ve just not been able to be attached to a big, expensive manshoot project that attracts enough breathless, drooling reviews from shooter-hungry review sites? Cutting one’s teeth on lower-key projects is no bad thing, and frankly learning hard lessons on shitty, underfundedgames is only going to increase your skills. This doesn’t meaningfully reflect any of that, or a whole lot more.

For instance, let’s try Ruslan Didenko, lead designer on the last two STALKER games. He’s a 77, which by current games industry score perception standards means mediocrity. Yet we know STALKER titles are amongst the most technically and creatively ambitious, clever and atmospheric videogames of recent years; yet their relative inaccessibility (by mainstream standards) mean they’ll never score the big numbers on some of the biggest sites/publications. Not to mention that Stalker Clear Sky was a bit of a boo-boo (at least at launch) so it’s understandable that its numbers are a bit lower than its successor, Call of Pripyat. Yet Pripyat demonstrated deftly that the devs had learned their lesson and honed their craft. Does a personal rating of 77 reflect that accurately?

Similarly Deep Shadows’ Oleksandr Khrutskyy, one of the lead designers on Boiling Point. That might be a game of legendary comedy, but in a very real way it’s an incredible technical achievement from a tiny team with a tiny budget. 61% doesn’t exactly convey how driven Khrutskyy surely is.

Then there’s poor, poor Sergey Titov, who was producer and programmer on Big Rigs: Over the Road Racing. Apparently he is only 8% good enough. There is 92% wrong with this person. How’s that going to look on the CV? Quite clearly Metacritic scores will not be the only factor involved in hiring and salary decisions, not by a long way: my feeling is simply that they perhaps shouldn’t be involved at all. I’m not sure what other purpose this new rating has, however. Theories?

Sure, it’s an entertainingly, perhaps even passingly informative idea to see and do these kinds of rankings, but I’m concerned they paint a incredibly inaccurate picture of a developer’s achievements and skills. Aggregating game review scores might be a useful touchstone but unavoidably loses nuance – do we really want that of people too?

__________________

« | »

, .

210 Comments »

  1. 28/03/2011 at 11:30 parm says:

    So, what happens now is that studios start salary negotiations on the basis of these bullshit scores.

    (don’t think it won’t happen, bonus payouts already depend on Metacritic scores for the games themselves)

    • 28/03/2011 at 11:32 Alec Meer says:

      It’s almost as if you commented before you read the story. Almost.

    • 28/03/2011 at 11:53 parm says:

      I read the story, but inadequately. My brain somehow eliminated the paragraphs between the two images. I am experiencing shame and shall hie myself away for basic reading and comprehension classes with haste.

    • 28/03/2011 at 12:30 Mike says:

      Then again, maybe this isn’t such a bad idea? It already happens in academia, where you are measured in many institutions on the basis of papers published and citations achieved, rather than the quality of the work you do.

      It’s not because people are blind to the work you’ve done, it’s because the field is too big and the people making the decisions can’t spend all year reading through work trying to decide who is better than who. So perhaps it’s unfair, but understandably unfair. The best of a bad situation?

    • 28/03/2011 at 12:43 parm says:

      Couple of issues with that though: academic papers are usually the product of maybe four or five people’s work (and often less), whereas games are usually the product of teams of tens of people, maybe even hundreds. Also, a single percentage score for a game tells you very little about whether, say, the programmer who worked on the particle effects, or the artist responsible for rock textures is any good at their job.

      It also totally fails to take into account the conditions under which a game was produced – for example, a friend of mine worked on some ropey cash-in GBA ports back in the day; one game in particular, they had 8 weeks to do it, had none of the original artwork or audio assets, nor any of the original code to work from, so they had to lash it up from ripped screenshots and audio recorded by playing the original game and sampling it; in the end they didn’t get the AI finished so there are massive, massive gameplay holes in it too. The game was, he will be the first to admit, unmitigated shit, and was quite rightly panned in the reviews; the fact that the team was understaffed, underresourced, working to a totally unreasonable deadline and so forth isn’t captured anywhere in the 3/10 scores it got everywhere. Fact is, he’s a one of the better programmers I know, and I’d happily employ/work with him, but by his Metacritic “developer score”, you’d chuck his CV in the bin before you even looked at it.

    • 28/03/2011 at 13:50 Berzee says:

      Yeah But He Didn’t Read The Post.

    • 28/03/2011 at 14:15 Ultra Superior says:

      It is a very stupid idea. Games are rarely products of their designers. (No, really.)

      Anyone who’s been in the industry knows that the executive role on the project has always the man with the cash. The stupider that man is, the more he tends to involve and ruin the game in development.

      (As in putting in all the cool features he’d seen in the latest activision game and discarding the stuff that “wouldn’t appeal to average consumer”)

    • 28/03/2011 at 14:18 battles_atlas says:

      Is it contractual that all Alec’s posts feature him being a bit of cock to the first commenter?

      @ Mike
      The system used in academia is a bad idea though, so why would you want to apply it elsewhere? As you say, it doesn’t measure quality but a combination of quantity and proximity to the mainstream. Its a shit system that limits academic freedom of enquiry. It might be acceptable if it was actually necessary, but its justification is another example of the tyranny of targets that I was under the impression was out of favour now NuLab have been given the boot. Its all predicated on a rampant distrust of professionals, and the belief that no one is capable of doing their job succesfully without half a dozen bureaucrats reifying their performance in soundbite metrics.

      Fuck the Numbers, as NWA might say if they were middle class white boys.

    • 28/03/2011 at 14:56 Jamison Dance says:

      @battles_atlas: I think Alec’s comment was justified, seeing as the article was published at 11:27, and the first comment was at 11:30, which just echoed something written in the article. I admire RPS’s efforts in dissuading those stupid “First!!!!111″ posts, and it seems like that is what Alec was trying to do.

      And yes, parm did say some more insightful things later, but maybe those were encouraged by Alec’s response.

    • 28/03/2011 at 14:58 Alec Meer says:

      Only if the first commenter does something to deserve it.

    • 28/03/2011 at 15:30 Urthman says:

      It’s a hard habit to break.

      “The first sentence of a recent Deus Ex 2 preview on pc.ign.com: ‘There’s a tendency among the press to attribute the creation of a game to a single person,’ says Warren Spector, creator of Thief and Deus Ex.” (via OldManMurray)

  2. 28/03/2011 at 11:34 Jockie says:

    I’d lke to think the interview process for game developers is a little bit more nuanced than typing someones name into metacritic, I bet Blizzard are delighted though.

    • 28/03/2011 at 15:41 Dances to Podcasts says:

      According to Metacritic, Cevat Yerli is one point better than Mike Morhaime and two better than Rob Pardo. This will not do!

  3. 28/03/2011 at 11:35 Dan Lawrence says:

    Some brilliant developers work on terrible projects and some terrible developers coast along inside successful projects. Most developers don’t get to pick the projects they work on.

    I doubt any sane companies will base their recruitment on a metacritic score.

    • 28/03/2011 at 12:00 mcwill says:

      The games industry’s management echelons are not exactly known for their sanity.

    • 28/03/2011 at 13:22 bascule42 says:

      Apropos of nothing…I read somewhere a while ago that a high Gear Score could seriously affect employment prospects.

    • 28/03/2011 at 14:36 LionsPhil says:

      It needs weightings, and badly. If they’re doing a straight mean of games-worked-on then it’s treating that bit of junior developer work for a straight-to-the-bargin-bin puzzler as every bit as important as spending years as lead developer on your groundbreaking Opus.

    • 28/03/2011 at 17:16 bob_d says:

      @ LionsPhil: Weightings wouldn’t remotely help, even if they were possible (job titles are flexible and credits aren’t terribly accurate, so usually only the team itself knows who did what, and even then there are likely differences of opinion). The whole scoring system is fundamentally meaningless. The scores are aggregates that aren’t remotely interested in the particular roles played by individuals: artists who do amazing work could have their scores dropped by providing work for games that were terrible in every other respect, for example. The publisher could insist on releasing the game unfinished, management could make some decisions about graphics and gameplay that don’t go over well with reviewers, the game could be mis-marketed and end up creating unfair expectations or causing the game to be misunderstood, etc., etc. There is no possible way that the gross Metacritic score could provide any meaningful information about the individuals in their database.

  4. 28/03/2011 at 11:35 Jackablade says:

    57% eh. Lucky they’ve only got a couple of my games on their list.

  5. 28/03/2011 at 11:35 Doesntmeananything says:

    If this would somehow affect the industry, then this is the most abominable thing ever. If not, well, then it’s just vile.

  6. 28/03/2011 at 11:36 Alexander Norris says:

    Looks like someone accidentally put a 9 in front of Cevat Yerli’s score, there.

  7. 28/03/2011 at 11:37 The Sombrero Kid says:

    any company who uses this system in their hiring policy will make very poor hiring decisions and will most likely fail as a result of that, thus I’m not too worried, I’ve been credited on games I’ve had nothing to do with & not been credited on games I’ve worked very hard on, I’ve also been credited for 2 hours work on games versus taking a game almost solely from inception to completion without any other input, to rate ME equally for these contributions is fine from a metacritic perspective but from an employer perspective is ridiculous.

    • 28/03/2011 at 14:07 Multidirectional says:

      Precisely my thoughts. Studio that would make hiring decisions based on these bullshit scores is a studio that doesn’t have a clue on how to spot talents in the first place. All they would produce is generic shit games, so it’s hard to care anyway.

  8. 28/03/2011 at 11:38 ChaosSmurf says:

    If “you must have 2 years of experience to begin working in the industry” becomes “you must have 2 years of experience and an 80 metacritic rating to begin working in the industry” you can wave goodbye to the industry.

    • 28/03/2011 at 16:55 bob_d says:

      It’s not (just) years of experience – the more important metric used in the industry is “shipped games.” (Yes, even for lower level positions.) Given how many co-workers I have had that hadn’t shipped a game in 10 years (despite constant industry work on multiple titles), it’s a completely meaningless measure of experience. My fear is that this, despite being recognized as being totally meaningless, will end up being used by the industry in some way as well.

    • 28/03/2011 at 18:18 Tacroy says:

      That’s how it works, though – the company looking to hire for a new position puts an unachievable list of requirements in the job ad*, and then when they hire you anyway they have a significant upper hand in salary negotiations (like, “oh you don’t have ten years experience in C++ (because you’re only 20), so we’d like to make your starting salary reflect that lack of experience. Don’t worry, you’ll get a raise once your skills are up to par!”)

      *unachievable either because the people who do have all those skills aren’t going to be willing to work for that salary, or because the skills are simply impossible for a mere mortal to have all at once

  9. 28/03/2011 at 11:39 sockeatsock says:

    Alec Meer (RPS): 100
    Jim Rossignol (RPS): 101
    John Walker (RPS): 678
    Quintin Smith (RPS): 87980
    Kieron Gillen (ex-RPS): 99999999

    • 28/03/2011 at 11:52 John Walker says:

      You appear to have a very weak understanding of how percentages work.

    • 28/03/2011 at 11:56 Daniel Carvalho says:

      Perhaps he is redefining how we think about percentages.

    • 28/03/2011 at 12:05 Man Raised by Puffins says:

      John should actually have a score; as I recall he did a bit of writing for the Broken Sword Director’s Cut on the DS. Brian also had a cameo in Angel of Death, so lets count that one too.
      Via the magic of maths: (78 + 73) / 2 = 75.5
      Congratulations John, you’re 0.5%* better than John Romero!

      *well, strictly 0.6666% recurring better, but lets not get ahead of ourselves here

    • 28/03/2011 at 12:09 The Tupper says:

      “You appear to have a very weak understanding of how percentages work.”

      Indeed, John. You are worth at least another thirteen thousand percent.

    • 28/03/2011 at 12:09 Simon Hawthorne says:

      Nah, it’s just you only understand them 100%. sockeatsock understands them 124%.

    • 28/03/2011 at 12:22 skurmedel says:

      He seems to understand them about as good as the games rating business frankly. For it to be a percentage it needs to be a percentage of something.

    • 28/03/2011 at 12:52 Sarlix says:

      Edit/ just woke up….nothing to see here move along…

    • 28/03/2011 at 19:11 BarneyL says:

      Perhaps Alec is the baseline and the others are scored relative to him.

    • 28/03/2011 at 19:43 DJ Phantoon says:

      Really, the fact here is that numbers are useless. Hopefully, this is Metacritic jumping the shark and people will stop paying attention to it now.

      And honestly, should the ramblings about one random blogger about one game hold the same weight as people who do this for a living?

  10. 28/03/2011 at 11:39 tomeoftom says:

    How damagingly stupid. How the hell are they supposed to numerically gauge the effect a single lead designer had on the overall product? This is ridiculous.

    • 28/03/2011 at 12:43 gorgol says:

      “How damagingly stupid”. My new favourite phrase.

    • 28/03/2011 at 12:56 gorgol says:

      But you know, its the consumers, i.e. us, that are to blame for this trend. If we were to stop using metacritic as a method of judging what games to buy, then it would stop gaining as much importance for developers and publishers.

      An alternative to using metacritic is to read sites like this which tell you what new games have come out as they come out, giving you a good idea of what they are about, so that you can decide whether to give them a try or not.

      Spread the word! ;)

    • 28/03/2011 at 15:38 Urthman says:

      What kind of idiot uses MetaCritic that way? Its only real usefulness is to find a bunch of reviews to read more easily than a Google search.

    • 28/03/2011 at 16:45 bob_d says:

      It’s even worse than that – it isn’t just for lead designers, but for everyone who gets game credits. So the artists who did amazing work on a game will have their scores brought down by the bugs and lousy design, for example. it’s worse than useless.

    • 28/03/2011 at 16:52 Wilson says:

      @gorgol – I can’t see any way you could actually tell how many people use Metacritic to make their purchasing decisions, and even where people do use it there are probably a ton of other factors as well. So I don’t think you can blame consumers for Metacritic being seen as a valuable measure of worth (whoever actually thinks that anyway), since I highly doubt they could cite any figures (certainly no reliable figures) for how many people used Metacritic in purchasing decisions.

    • 29/03/2011 at 12:24 gorgol says:

      The fact remains that publishers view metacritic score as extremely important. They do so because it reflects sales figures. Which means that there is a direct and proven correlation between review scores and sales. Only consumers can break that trend, by educating themselves.

      Obviously the people on this site don’t place much importance on review scores, after all we come to a site that doesn’t give scores, but obviously a large part of the consumer base does. That needs to change.

  11. 28/03/2011 at 11:42 icupnimpn2 says:

    You can go to advanced search for people/artists, click the check box for game, and leave the search box blank… this will give you a list of everyone. But it is unfortunately not organized in any way. Thought it would be fun if they presented a true Top 20 or Bottom 20.

    Anyway, pretty horrible considering that many people may start out working on crap games simply because they’re at the bottom of the food chain.

    • 28/03/2011 at 16:08 Dances to Podcasts says:

      What you’d probably get is top 20; bunch of people who happened to only work (or get credit) on that one good game, bottom 20: bunch of people who happened to only work (or get credit) on that one bad game.

  12. 28/03/2011 at 11:42 frenz0rz says:

    Richard Garriot is only worth 5 more than Derek Smart? Wut?!

  13. 28/03/2011 at 11:43 Teddy Leach says:

    DEREK SMART. DEREK SMART. DEREK SMART.

  14. 28/03/2011 at 11:43 Solivagant says:

    I really doubt recruiters will really use this. Project Managers, I think, are the ones that might look at Metacritic scores. But they dont handle recruitment.

  15. 28/03/2011 at 11:43 CMaster says:

    So do you get included in your score every game you are credited on? Regardless of how big or small your contribution?

    Also, not long before we see games featuring lead designer Alan Smithee, with level design by Alan Smithe, Alan Smythe and Adam Smithee

    • 28/03/2011 at 12:00 Oozo says:

      As I have written below: It’s even more arbitrary. If the cruel gods of fate have summoned up somebody to put down your name in the GameFAQs database, you will be credited for that game. If not, well, your loss.

      Needless to say that those gods are some right lazy bastards who don’t bother to summon database minions all that often.

    • 28/03/2011 at 17:40 bob_d says:

      The most minor contributions are indeed being treated the same as major roles.
      I immediately thought of the Alan Smithee phenomenon when I read about this, but there would be a couple differences from the Hollywood “Alan Smithee” mechanic, though. In Hollywood it’s reserved for those roles where there’s an expectation of creative control and that person doesn’t feel that they got it. There’s no such expectation in the game industry. In Hollywood, the Gaffer never feels the need to get credited as Alan Smithee (nor is there a system that allows that to happen). Also, in the game industry, “shipped games” is an important metric of experience that determines if you can get subsequent jobs, your job position, pay, etc. Not being in the credits for a game is equivalent to not having worked for that period of time in some ways. If you worked on a great PC game, the lousy PS3 port would count against you (even if you had nothing to do with it). If there was a system that allowed developers to remove their names from the credits, they’d have to do so before it was published; a game that was poorly marketed or had undiscovered bugs would see its score drop. Not to mention that no company would actually allow the employees to use pseudonyms in the credits – that would be an admission the game was bad before it was even published. Developers with name recognition will have their names being used by the publisher to drum up publicity, so there’s less than no chance they’d get credit control.

      In other words, if this score actually gets used for anything, developers are screwed.

  16. 28/03/2011 at 11:44 DanPryce says:

    What I’m interested in is the fact that they’re rating the individuals rather than the studio. If the studio was rated there wouldn’t be any fuss whatsoever – Valve and Blizzard sit at the top, and all is right with the world. It’s because the rating is on the people themselves that this pisses me off. It might not necessarily have been their input that caused the bad reviews – if a developer designs a decent game and the graphic department drops the ball, should he or she be held accountable? It doesn’t make sense to rate the individuals.

    • 28/03/2011 at 23:18 Inglourious Badger says:

      Very true. This whole thing would make sense if it was used on the studio as a whole rather than the actual devs. As it is it’s just a wierd joke.

      Be interesting to see how the likes of Ion Storm or Looking Glass would fair on a metacritic basis. Deus Ex – Daikatana + Anachranox = 70%? = Not good enough. Might explain why some of these studios didn’t last :(

  17. 28/03/2011 at 11:45 Grinterloper says:

    I find numerical scores for reviews are pointless (oh god I made a pun) anyway, how can you put a definitive value on something which is intrinsically subjective and nuanced? what is the unit of good?

    “using my scoratron-o-graph I can see that this game contains 84 “goods”" It just doesn’t work, but to assign them to the developers themselves?

    How do they measure this? I assume it’s purely based of the aforementioned fabricated review scores and unit sales which can be altered by the change of the winds (or fat wads of advertising cash.) So unless the metacritic people are actively snooping on the developers day to day work, hiding behind office plants and sneaking microphones into breakfast croissants, their scores are utterly baseless and can’t help but fail entirely to reflect the actual ability and worth of a developer BUNKUM I SAY BUNKUM!

    • 28/03/2011 at 20:46 Jason Moyer says:

      Metacritic isn’t measuring something that’s subjective, though. It’s measuring how people feel about a game, not whether or not a game is good.

      Videogames as a whole are past the point of giving a shit if games are good, it’s about whether people like them and spend money on them. The difference between, say, 1980-era Activision and 2011-era Activision is the same as the difference between Disney while Walt was alive and Disney after he died. In one case you have a bunch of guys trying to make good products with the idea that good products will sell enough to continue making good products, in the latter case you make things for the sole purpose of selling them so you can buy yachts and BMW’s.

  18. 28/03/2011 at 11:49 Rinox says:

    How did Roper survive the debacle that was Hellgate London? :-/

    • 28/03/2011 at 12:04 Dominic White says:

      Because, contrary to INTERNET RAGEFEST 3000, Hellgate wasn’t that bad. After patches, it was pretty fun. And yet you’d never think that when you hear people talk about it. People seriously wishing physical harm on the developers, and no shortage of ‘I hope they never find work again’ sentiment.

      Once you escape the internet echo-chamber, you find out lots of interesting things. Like how DX: Invisible War was reviewed well and how most people actually liked it, even if it wasn’t as good as the original.

    • 28/03/2011 at 12:12 TheApologist says:

      Yeah, I was annoyed about Hellgate going away, not really about the quality of the game. Couldn’t quite work out quite what the rage was for.

    • 28/03/2011 at 12:17 ReV_VAdAUL says:

      The anger was about it not being Diablo 3.

    • 28/03/2011 at 12:33 Rinox says:

      Just to be clear: I never played Hellgate London and was talking about its reception. :-) (70 on metacritic on the PC)

      And with ‘survive’ I meant ‘to get such a high score on this ranking’. I don’t wish anyone’s career to be ended by any sort of bad or good game, obviously.

    • 28/03/2011 at 12:51 jstar says:

      Deus Ex invisible war was an absolute car crash of a game.

    • 28/03/2011 at 13:13 poop says:

      I’d like to point out at this junction that dominic white bought a lifetime subscription to hellgate

    • 28/03/2011 at 13:27 TillEulenspiegel says:

      ike how DX: Invisible War was reviewed well and how most people actually liked it, even if it wasn’t as good as the original.

      It’s not about objective quality, it’s disappointment. When you make a sequel that tosses out what many people loved about the original, they’re going to hate your guts, even if it’s a decent game in its own right. See also: DA2. That’s not an “internet echo chamber”, that’s a fairly predictable result of disappointment.

      I quite enjoyed the HGL beta, but never got around to buying the full game before it went bust, mostly because I really didn’t like their two-tiered, not-quite-an-MMO business model. It made getting around on the tube much more fun when I visited London for the first time, though.

    • 28/03/2011 at 13:53 Dominic White says:

      See, when I’m disappointed, I say ‘Well, that’s disappointing’, maybe sigh, then do something else. There are people who are still raging about Hellgate and DX;IW just as strongly years after the fact. That’s not disappointment, that’s… I don’t know what it is, but it isn’t healthy or sane.

    • 28/03/2011 at 16:20 Hallgrim says:

      @Dominic White: Ain’t hard to find crazy on the internet.

      /gibbers

    • 28/03/2011 at 18:26 Archonsod says:

      “Just to be clear: I never played Hellgate London and was talking about its reception. :-) (70 on metacritic on the PC)”

      Because critical reception =/= market reception.

    • 28/03/2011 at 21:45 Rinox says:

      Well, the discussion was about designers being judged based on their metacritic averages.

      I didn’t refrain from playing Hellgate London because of reviews or anything else. I just didn’t care about it, like I didn’t care about Diablo. I hope that’s finally clear to everyone now. I was just wondering about how Roper got an average score of 92 on metacritic with a game judged at 70 on his resume.

  19. 28/03/2011 at 11:49 alexdulcianu says:

    I enjoy reading this blog, so I can’t let this hopefully mistyped statement slip: “Yet we know STALKER titles are amongst the most ambitious, clever and atmospheric videogames of recent years;”.

    I am sorry, but it is a widely accepted fact that the STALKER series ARE the most ambitious, clever and atmospheric videogames of ALL TIME.

    Meh, I’m just joking around here(I’m not), but really, no one has ever came as close to making a perfect open-world FPS as GSC did with STALKER. Not to mention they’re probably the only developers that kept improving their series over time and listening to the demands of their fans.

  20. 28/03/2011 at 11:50 Giant, fussy whingebag says:

    This reminds me of the much more interesting thing where someone used Metacritic data to show the stats of individual reviewers. (Looking for the link now – will edit in when I find it)

    • 28/03/2011 at 12:13 Giant, fussy whingebag says:

      I’m starting to think I imagined it….

    • 28/03/2011 at 12:50 Xocrates says:

      Metacritic does the stats of individual reviewers. Just click on the name of the reviewer and you get a bunch of data, including whether his reviews are usually higher or lower than the average.

      Maybe it was that you were thinking about?

    • 28/03/2011 at 13:25 Giant, fussy whingebag says:

      No, the thing that lives in my head was something else.

      (Apparently in my imagination) someone had done some data mining on Metacritic and written an article about reviewers and their scoring statistics, covering them in more depth than metacritic does. Also, I think it was about specific individuals rather than publications as a whole.

      Of course, since I can’t find a link, this all just seems like so much nonsense.

    • 28/03/2011 at 13:26 RagingLion says:

      You’re not imagining, I remember that too. Was is specific so one of the gaming sites like PC Gamer or Eurogamer I’m thinking. I remember RPS reporting on it I think.

    • 28/03/2011 at 13:30 Lambchops says:

      Yeah, it was some data mining of Eurogamer which showed the average score reviewers tended to give games (I think it was then compared to the metacritic score to show how far they deviated from consensus).

      i’ve got a bookmark too the blog on my home laptop (there were quite a few interesting articles but it has been quiet lately), can’t find it right now as calling it “The Player” makes it un-google-able.

    • 28/03/2011 at 13:54 Giant, fussy whingebag says:

      Ah ok, so I was thinking it was Metacritic, but it was actually just Eurogamer. I’m glad some people knew what I was talking about, even if I didn’t!

    • 28/03/2011 at 21:12 Lambchops says:

      As promised here’s some linkage. Actually, turns out you were right (there was a Eurogamer piece but it wasn’t related to Metacritic).

      Here’s the Metacritic one: http://www.theplayer.sekritforum.com/?p=51

      And here’s the Eurogamer one, which was actually about commenter disputing reviews with good old tiresome “read like a [inserts score here]” type comments: http://www.theplayer.sekritforum.com/?p=38

      The Eurogamer one I was initially thinking of was linked to elsewhere in teh above blog and was this years worth of analysis (which does compare to Metacritic) – http://eurogamer2009.heroku.com/

  21. 28/03/2011 at 11:51 President Weasel says:

    Bill Roper most recently oversaw Champions Online as Executive Producer. The game’s mediocrity seems fairly reflected by its metacritic score of 72: it’s not bad, it’s just not very good. great character editor, lack of content, bit of a grind-fest leading to not very much of an endgame, and a pernicious “sell the disk, and the subscriptions, AND the micropayments” business model.
    Seems like rather an indictment of metacritic’s “stick scores on people” system that he ends up being at the top of that list through being attached to some spectacularly good and well-reviewed group efforts from Blizzard, despite the game he actually helmed being rushed and unfinished, while people who have been in charge of some much better or at least more ambitious games have lower numbers semi-arbitrarily attached to them yes I know this is what you are saying with the article

    Perhaps metacritic could develop an algorithm that took into account the number of people on a given project, or the level of control a given person exerted (less weight given to being spokesgoatee for Starcraft than executive goatee of Champions, perhaps?) or a simple “add 15% for artiness” metric.
    Or just not hang arbitrary numerical scores on people.

    • 28/03/2011 at 14:58 Malibu Stacey says:

      Since when does 72/100 = mediocre? Surely it’s around 22 points too high.

      This is the problem with sticking arbitrary numbers on things like games reviews. Very few people actually understand the scale they’re using (notable exception being EDGE magazine where an average game gets a 5 not 79%).

    • 28/03/2011 at 15:26 President Weasel says:

      since metacritic includes scores in its calculations from publications that mark from 7-10, a low-70s average is pretty mediocre.
      World of Warcraft and expansions – low 90s.
      Warhammer Online – 86
      Lord of the Rings Online – 86
      City of Heros – 85
      Dungeons and Dragons Online – 74
      Champions Online – 72.

      Now I’d be the first person to agree that in a sane world, a mediocre game wouldn’t get an average rating in the low 70s. This isn’t that world though: in this world, a low 70s score tends to mean a mediocre game, a game with great ambitions that didn’t quite realise them, or a game the reviewers didn’t really understand.

  22. 28/03/2011 at 11:51 Oozo says:

    And let’s not forget the fact that they base their knowledge on who’s worked on what project on GameFAQs. Which is another horrible decision, since that database is FAR from complete. In other words: Plain stupid.

    If it really has a profound impact on the industry or enterprises who base their recruitment strategies on it, those deserve to go down, I guess.

  23. 28/03/2011 at 11:52 Daniel Carvalho says:

    John Carmack 78?! You gotta be kidding me. And judging by the most the developers above him, there was clearly very little Maths and game industry knowledge involved in Metacritic’s “calculations”.

    John Carmack should be much higher, if not at the top for his contributions to the gaming industry, and this list is retarded. But I guess, that is stating the obvious.

    • 28/03/2011 at 14:17 Stijn says:

      John Carmack is probably mentioned in the “special thanks” for about every game using an id Tech engine, without him necessarily having done anything on the game itself…

    • 28/03/2011 at 22:14 Daniel Carvalho says:

      Well, he helped formalize certain technologies such as BSP, which, is like, a foundation block for most 3D engines this century. Whether it’s his engines or not. Big deal. That’s just one thing, there’s more.

  24. 28/03/2011 at 11:53 Gassalasca says:

    The Prisoner reference!

  25. 28/03/2011 at 11:59 Jorum says:

    “Games are art”
    If we want that to be the case maybe it is a good idea not to stick numerical values on everything and everyone.
    I wonder what Kandinsky or Van Gogh’s metacritic art score would have been while they were working?

    Art involves vision not just technical craftmanship.
    As Alec points out Boiling Point was a nightmare, but at least they aimed for something bold.

    If this takes off then it will only push games industries further down the hollywood road where safe, risk-free, dependable, bankable crap is the focus

    • 28/03/2011 at 16:47 Dances to Podcasts says:

      Van Gogh is known for not selling very much at all during his life, so he likely wouldn’t even get a Metacritic rating for being such an unknown. You know, just like how many good games don’t appear on Metacritic simply because they’re not ‘big’ enough.

    • 28/03/2011 at 20:53 Jason Moyer says:

      Bach wouldn’t have a metacritic score until 200 years after his death, but it would be 100′s across the board.

  26. 28/03/2011 at 12:00 Jolly Teaparty says:

    I can’t help but knee-jerk a bit here; so what? Anyone who’s been to school, let alone on to higher education, is used to having their worth as a person presented to them in some kind of standardised grade, and rarely is the system used not hotly debated on an annual basis. It’s not like their metacritic scores are accredited by any institution. I think any use of them in a selection process would reflect worse on the employer than the developer.

  27. 28/03/2011 at 12:01 jstar says:

    Well I think this is a brilliant idea. I keep having arguments with people about who is better, Warren Spector or Ken Levine. And not just who is better but how much better. Finally we can put that age old question to rest because clearly not only is Levine better he’s a whole 7 better. So fuck you Spector fanbois. Because that’s not just any old 7 better it’s 7 in the 80s which means more than 7 in the 70s so actually it’s probably more like 14.

  28. 28/03/2011 at 12:03 mcwill says:

    Once again the quietly publisher-centric viewpoint of Metafilter is demonstrated. The quality of a final production as reviewed by critics has virtually no relationship with the quality of the people working on that production, and people working in the industry rarely get to choose the projects they work on, let alone affect most creative decisions.

    This is just another stick to beat developers with, nothing more.

  29. 28/03/2011 at 12:04 Mitza says:

    It’s a great idea, imho, but it’s implemented badly. They seem to take into account all the games a person has been credited on, which makes it quite useless (for example, Tom Hall is also rated by the voice-work he’s done on some games :)). It would be great if it would take into account the role of that person within the game team, his responsibilities and so on. Otherwise, it’s just numbers that don’t carry any weight.

    • 28/03/2011 at 12:05 jstar says:

      It’s only a good idea in the same way that sharing AIDS is a good idea.

    • 28/03/2011 at 12:13 ReV_VAdAUL says:

      The more people who have AIDS the more motivation / funding there will be to find a cure!

      Oh but all the research will be done by people who post on the New Scientist forums.

  30. 28/03/2011 at 12:07 Rond says:

    Who the hell is this mister Yerli to score higher than both Carmack and Spector?

  31. 28/03/2011 at 12:08 TXinTXe says:

    I think we should star to campaing against game reviews’ scores. And by “we” I mean “you”, of course.
    Serioulsy though, I think it’s a far more important matter than the diferent release timeframes, and I know that here in RPS you don’t do that, but I think that no one should do it.

    • 28/03/2011 at 12:24 Jorum says:

      The games industry is the only one whose vast majority of critics and reviewers think a percentage score is a) appropriate b) means anything.

      Book reviews don’t go to the absurdity of stating that Harry Potter is 4% better than Catcher in the Rye.

      I’m not sure how it started (it’s been around ever since I remember and I started with Atari’s and C64s), but its a deeply stupid idea that diminishes our medium and frankly makes us look stupid.
      The quicker we dump it the better.

    • 28/03/2011 at 12:34 skurmedel says:

      And for some reason uses a scale from 0 to 10 or 0 to 100 and sets the median at 7… Why not use a logarithmic scale then? Or radians?

    • 28/03/2011 at 13:11 stahlwerk says:

      I think it became journalistic practice because early game reviews were but a few columns in general home computer review magazines in the early 80s, which were but a few columns in office appliance review magazines in the 70s, which were read by and thus written for nerdy bureaucrats deeply in love with numbers.

    • 28/03/2011 at 14:10 Oozo says:

      As a certain Mr Wright put it so eloquently over at Gamasutra:

      “Again, it boggles the mind: of all the “art” criticism in popular culture, the reviews with the worst quality and most screwed up metrics somehow have the most power and influence over their medium.”

      Even though I’m not sure that I would fully agree with the “worst quality” thing, he certainly has a point there.

    • 29/03/2011 at 01:29 Thants says:

      “The games industry is the only one whose vast majority of critics and reviewers think a percentage score is a) appropriate b) means anything.”

      Well, the film industry definitely does as well. I mean, they tend to use stars rather than percentage but it’s the same thing.

  32. 28/03/2011 at 12:09 Turbobutts says:

    Okay, can somebody please explain to me why Cliffy “Blame it on anyone but us if our shitty console shooters don’t sell” B has a higher score than walking gods such as Peter Molyneux, Doug Church and John Carmack?

  33. 28/03/2011 at 12:11 popeguilty says:

    Wait, wait, American McGee outranks Warren Spector? Not to shit on his work at iD- his art is great- but he hasn’ done anything worth playing since, well, he left iD. How on earth does he outrank the creator of Deus Ex, Ultima Underworld, Thief, and System Shock?

    • 28/03/2011 at 12:51 diebroken says:

      Apart from American McGee’s work on the Alice series (and not including Bad Day LA!), I can see your point.

    • 28/03/2011 at 13:20 faelnor says:

      Thank god, Warren Spector is neither the creator of Ultima Underworld nor of Thief or System Shock.

    • 28/03/2011 at 14:53 BooleanBob says:

      The point stands, though. Wasn’t Bad Day LA universally panned?

      More generally, given that I’m sure Metacritic have admitted to being selective – as opposed to comprehensive – in their incorporation of review sources, the entire premise of their site, be it in ‘metricising’ individual games or more collectively their producers, always struck me as being fundamentally flawed.

    • 28/03/2011 at 16:55 Frank says:

      Bah, they gave McGee credit for his level-design on Doom and Quake, but somehow forgot to count American McGee Presents Bad Day LA (Metascore 28) and American McGee’s Crooked House (63).

  34. 28/03/2011 at 12:11 FreakyZoid says:

    I’m on 93%. According to Metacritic I am a better developer than all of those listed above. This is clearly nonsense.

  35. 28/03/2011 at 12:14 Robert says:

    As usual with them numbers and statistics.. they are only as good as the person using them. (and knowing their worth and caveats)

  36. 28/03/2011 at 12:16 terry says:

    Jeepers, the sooner everyone stops ascribing any sort of significance to Metacritic scores the better.

  37. 28/03/2011 at 12:16 ReV_VAdAUL says:

    This will be exceptionally harmful to game reviewers. Oh its directly very bad for developers no doubt but if this catches on then those developers will be less willing to associate with or talk to any kind of objective reviewer. If your livelihood depends on good reviews its only sensible to cultivate relationships with biddable yes men in the media.

  38. 28/03/2011 at 12:24 AdamK117 says:

    Completely rediculous, metacritic generally seems to give high ratings to games that reviewers love. While this isnt a bad thing (they’re professional reviewers after all) they do tend to give numbers between 70-100 frequently regardless of how good it is and additionally tend to score mainstream games really highly (because PC-Gamer doesn’t want to be seen as the only reviewer slamming Dragon Age 2 now, do they?).

    Also, Gabe Newell: No metacritic :<

  39. 28/03/2011 at 12:26 Navagon says:

    It will be a factor in salaries simply because games companies want games that sell well and bolster their reputation for quality titles which generate repeat customers, sequels and all that bullshit. They’re not going to pay much for the guy who brings home a 65 on this metascore report card. That’s not going on the fridge. No way.

    I know it’s unfortunate that metascores largely based on blatantly paid off reviews can make or break the people not involved in paying reviewers off, but that’s the way it is. Look at what happened to Alpha Protocol.

  40. 28/03/2011 at 12:29 Deano2099 says:

    Yes, for it to be any use at all there needs to be some sort of weighting system. Especially as it’s one thing doing this now, where a lot of the big names got into decent development positions pretty quickly as the industry grew around them, but in ten years’ time it’ll be ridiculous. Latest hot-property developer Mr X who has been lead designer on three brilliant 90%+ titles in a row gets an average of 60% because of the five years he spent as a QA tester on hidden object games.

    Still, it’s fun, but a huge amount of stuff is missing too.

    Dave Gaider is on 93 which beats everyone else so far. But that’s purely on Neverwinter Nights and BG2. He’s not even credited for the Dragon Age games.

    Tim Schafer: 86
    Ron Gilbert: 82 (who is also rated 38 for movies, and 27 for TV, if you were still wondering how ridiculous this is)
    Mike Stemmle: 79 (again, no mention of the Lucas Arts stuff)
    Jane Jensen: 80 (although only Gabriel Knight 3 is listed, not GK1 or 2, KQ6, or any of her casual stuff at Oberon. Or Gray Matter)

  41. 28/03/2011 at 12:32 skyturnedred says:

    Feargus Urquhart – 84

    With all the Black Isle games and NWN2: Mask of the Betrayer in his list of games, I think he should be rated at least 92.

  42. 28/03/2011 at 12:35 President Weasel says:

    I can’t tell if the people posting “such and such a person should have a higher score” are being knowing and ironic or not, and that bothers me.

  43. 28/03/2011 at 12:38 tlarn says:

    I put as much stock in aggregate scores as I do in horoscopes; interesting to look at, but I don’t put much value in them at all. I look at individual reviews after seeing those Metacritic scores; just seems silly to me to take these numbers at face value without looking at the reviews themselves.

    Still a scary thought that people could, can, and will take these numbers on developers seriously.

  44. 28/03/2011 at 12:39 Jac says:

    Numbers are dumbers.

  45. 28/03/2011 at 12:56 wisnoskij says:

    I don’t really see the point of this.

  46. 28/03/2011 at 12:58 Namos says:

    So when is Metacritic starting its selective breeding program, again?

    Ridiculous.

  47. 28/03/2011 at 13:00 Lobotomist says:

    You know its wrong when Bill Roper ends on number 2 spot

  48. 28/03/2011 at 13:06 Urael says:

    Kill Metacritic with fire! This is bloody ridiculous but it could work in our favour. Hopefully by assigning numbers to people,other people will then start to understand how bloody daft it is assigning numbers to games. Or am I being too optimistic on this one?

    “(“we are/want to be a 90% Metacritic average company” is a refrain I’ve heard many times of late, especially from larger devs and publishers)”

    I just threw up a little. Yeeeuch, What a horrible thing to aspire to.

    • 29/03/2011 at 01:41 Thants says:

      I choose to believe that this is the result of someone who works at Metacritic realizing the fundamental useless of the concept and pushing it to this ridiculous extreme in an effort to bring down the system from inside.

  49. 28/03/2011 at 13:10 bill says:

    Hey it works for movie directors and actors too!
    So now I know that James Cameron is 8% better than Ridley Scott. And mark Hamil is at least 25% better than Hayden Christiansen.

    Weirdly, i was hoping for lots of crazy examples of why this would be all wrong… but almost the people I looked up seemed to be about right.

    • 28/03/2011 at 16:47 bob_d says:

      Looking up a friend of mine (who despite his dozen or so game titles, doesn’t even appear in their database), I accidentally found some actor whose score was brought down to 20-something because he had two very minor speaking parts (where his character was not even named) in really terrible movies. It’s just one of the more obvious problems with a system like this.

  50. 28/03/2011 at 13:10 Soon says:

    I prefer the gladiatorial system.

  51. 28/03/2011 at 13:11 ix says:

    The only reason metacritic scores of games are tied to bonuses is because those scores seem to correlate well with overall sales numbers. You bring out a game that scores 90+ in all the major magazines, you’re pretty sure it’s a hit. It makes sense to give bonuses for stuff that sells well (while game developers might be in it for the games, publishers are in it for the money). If anybody has a correlation coefficient for the past year of so that would be interesting btw.

    On the other hand, Individual developer scores will not correlate at all with their ability to do their job well. It wouldn’t make sense for publishers to try to use this. I’m betting they’re at least smart enough to know that.

    Of course, past experience does matter. But that’s the kind of stuff that comes up in the interview process. I bet people who’ve worked on Big Rigs do get a couple of questions about it.

  52. 28/03/2011 at 13:13 The Magic says:

    My friend is a complete and utter Nintendo fanboy, so finding out that Shigeru Miyamoto is only 80% has made my day.
    He is officially worse than American Mcgee

    …UPDATE, he’s 9 worse than VIN DIESEL. Riddick is officially a better game developer than Shigeru Miyamoto, AND Satoru Iwata (who has 79%)

  53. 28/03/2011 at 13:37 Gary W says:

    The games journalists and the developers of AAA games are locked inside a self-sustaining feedback loop that allows them to buy cocaine and protect their own livelihoods. It’s Metacritic’s job to sit on the sidelines and document this curious process, guaranteeing a steady flow of cocaine for the employees at CBS.

    The feedback loop begins to fall apart when the games or the cocaine become too watered-down. The exact point at which this happens is currently unknown, but several industry professionals are busy working on a reliable metric.

  54. 28/03/2011 at 13:37 pyjamarama says:

    Poor Johnny Depp he only has a 62 in metacritic I’m sure he is scrambling for work now and is paycheck must simply be near zero.

    Metacritc recognizing game developers has people not just faceless corporations is a positive thing, we should recognize the people that make gaming great not just the company logos on the box. The average score is just a number not of any significant importance, any company that would hire someone just looking at that number is company that isn’t’ going to be around for long.

  55. 28/03/2011 at 13:42 Ravenger says:

    I’ve worked on over 60 games with various roles including testing, art, design, assistant producer, project lead, lead designer, etc. How many games of mine does Metacritic say I’ve worked on?

    Seven – and that includes two games I didn’t work on, and three skus of the lowest rated game I’ve ever worked on, handily dropping my score down quite a bit.

    I really hope that employers don’t start using this to gauge how good potential employees are. It’s about as accurate as the average horoscope.

    • 28/03/2011 at 15:50 myca77 says:

      I’ve got none listed, at least MobyGames have got a few of the titles I’ve worked on (granted for a short while someone with the same name seemed to be taking my credit, but I wanted some of his jobs on my list).
      Funny thing is if you only took the Bethesda and Blizzard stuff I’ve worked of I’d be at over 90%, whereas once you add some off the “other” titles I’ve worked on that score would quickly diminish, yet I’d like to think that what little part I had in each game I’ve worked on was always done with the same amount of quality.

      Edit, damn, I’ve just seen that the other me has gotten my credit for Starcraft 2 on moby games, cue an angry email going out in about 2 minutes :)

    • 28/03/2011 at 16:33 Nick says:

      Personally I’d make a complaint if I were a developer.

    • 28/03/2011 at 18:00 bob_d says:

      Not to mention that this is Metacritic data; there’s no way it’s remotely relevant to individuals even if it was accurate in its credits.

  56. 28/03/2011 at 13:56 Okami says:

    Their database sucks. Many games don’t even include credits and others are incomplete – and a lot of games (especially budget titles or games that o ly had regional releases) don’t even showing up. If they really feel the need to go on with this retarded idea, they should team up with mobygames and use their database to decide which developer worked on which games..

  57. 28/03/2011 at 14:11 Freud says:

    I am glad that the companies that care mostly about metacritic get punished a little bit when churning out lazy stuff (Mafia 2 and Dragon Age 2). At least it shows the system isn’t totally broken. Just partly broken.

    Meanwhile, the system works horribly for smaller publishers/developers, who doesn’t have the marketing/bargaining power of the big guys. The correlation between reviews and how good indie games are is very small.

  58. 28/03/2011 at 14:28 Multidirectional says:

    So, idiot Mike Laidlaw is 16 points better than wonderful Tim Cain.. Part of me wants big studios to actually base their hiring decisions on this so the game industry can collapse faster. We’ve been needing a fresh start for a while now.

  59. 28/03/2011 at 14:47 shoptroll says:

    Will Wright’s at 81. That puts him in the soggy middle with Warren and the rest of the gang.

    Not sure I’m terribly happy they’re doing this. It’s an incredibly broken system when you consider how long some of these people have been in the industry. The Metacritic listing for Wright completely lacks anything predating The Sims and I’m sure that’s the case with Molyneaux and others as well.

    I’d much rather see metacritic ratings for studios not designers. Games an are ensemble effort.

  60. 28/03/2011 at 15:05 Yargh says:

    I despise any system that attempts to apply statistics at the level of the individual.

    And I’m even more angry with the principle that anything scored below 80% is somehow below average.

    • 28/03/2011 at 15:51 Urthman says:

      Somebody’s confused about percentages. I’ve taken plenty of classes where an 80% on an exam would be below average.

      I’ve never seen a games review site where their score was supposed to mean “this game is in the 80th percentile of all games.”

      (None of which is meant as a defense of MetaCritic or the general stupidity of game scoring.)

    • 28/03/2011 at 17:38 Temple to Tei says:

      And, I’ve seen plenty that no one ever gets over 80%.
      (that is not as short and snippy as it reads, just cannot think of more to say. Something like the bar moves?)

    • 28/03/2011 at 18:14 bob_d says:

      @ Urthman: Well, in the standard (US) grading system, tests and assignments are supposed to be calibrated such that 70% is considered “average.” (This is not to say that there aren’t individual grading systems that are different, nor that a majority of students might not be above or below average in a particular class.) If 80% is considered “average” however, it does skew things in odd ways, especially if you grew up with 70% being “average,” where you end up with a combination of psychological effects and less granularity tin defining “above average,” “good” and “excellent” games.

  61. 28/03/2011 at 15:08 Acosta says:

    I think it´s quite interesting, the idea at least. The execution is absolutely horrible. For an actor/director, his role is probably going to be the same in the next project: being actor or director, not photography director, so it makes sense to see a media of their work. But in this industry, where one can pass go from QA to designer, then to lead designer and lastly to director, does this makes any sense? (That without mentioning how inaccurate many lists are at this moment.)

  62. 28/03/2011 at 15:16 Warskull says:

    This is actually kind of a good idea. They should just rate studios instead of individual developers. If a studio makes crap games, they usually continue to make crap games, while if they make good games you can reasonably assume the next game will be good.

    The only problem is individuals may work on a game, but have very little influence if it is good or bad. Measuring the studio as a whole would make more sense. Plus, the games industry is intentional horrible with credits in an attempt to hide their talent and keep them underpaid.

    • 28/03/2011 at 18:33 bob_d says:

      Actually it doesn’t work for studios for some of the same reasons it doesn’t work for individuals – it ignores the role of publishers, for example, in game quality and perception (which, let’s face it, is a big part of these aggregate scores). I’ve worked on projects that were pretty well sabotaged by the publishers.
      Even if we assume that the scores represent actual game quality for which the developers are alone responsible, that’s going to vary over time for a development studio depending on their resources and control over a project (young studios usually have very little of either). Even if you considered more recent games as being more relevant, an established company might run low on funds and have to pump out a cheap (i.e. lower rated) game or two to get back on their feet. A company like Blizzard has enough financial control over their products that they can release highly polished games (and even completely kill games that are below their expectations). Even though the gameplay is unoriginal, they’re still going to score higher than a company that makes incredibly innovative but “rough around the edges” games.

  63. 28/03/2011 at 15:18 J-Han says:

    Metacritic serves its purpose well. It’s designed by and for people who don’t think.

  64. 28/03/2011 at 15:19 Calabi says:

    So what is the point of this? Why have they done this? Are they working up to giving every facebook individual a percentage rating?

    I just cannot fathom, it is sick and it should be banned(and I’m not joking). Isnt it illegal under human rights as in you have a right, not to be a number, or given an aggregate score for your work(or some such nonsense).

  65. 28/03/2011 at 15:53 Jake says:

    Hmm, I think the high percentage developers should be paired off and bred together, they have good games developer genetic stock.

    I look forward to when they implement this rating system in Facebook – you could give your friends a percentage! High percentage people could get paired off and given the good jobs and maybe a discreet cull of all the sub-10%ers. I wonder how I sign up to give one of those TED talks…

  66. 28/03/2011 at 16:06 Stoloniferous says:

    I sent the following to Metacritic yesterday:

    I am distressed that Metacritic is now choosing to rate developers of games. Most who work on projects that flopped had little choice in the matter. If your rating system is taken seriously by anyone in the industry, which I hope and suspect it will not be, it will make life even more difficult for start-up companies, as those who are concerned about their career rating will head for the larger and more stable development houses. Companies will be less likely to take on the risk of innovative games if those involved know there is even more at stake than their current job. Please ditch this terrible idea ASAP.
    -(developer) at (studio)

    The following response was in my inbox today. Make of it what you will.

    Hey (developer),
    Thanks so much for your note. Currently, Metacritic is sharing a credits database with GameFAQs, and it’s a work in progress. We’re never going to assign “people” Metascores – we’re just listing what they worked on – in the same way we list what movies cast members, directors, and writers worked on in the movies section – and the “career” score is a simple average of those individual projects. It’s a work in progress, something that we’re going to beef up to make much more complete (with help from the gaming community). We’ll never “rank” people like we do products. You’ll never see a “top 10 producers” or anything like that.
    Interestingly, we’ve been doing this for a long time, and it seems like nobody noticed until this weekend! But I can tell you that we’re taking the reaction to it seriously, and we’re working to improve the product.
    Best,
    Marc Doyle
    metacritic.com

  67. 28/03/2011 at 16:10 Plopsworth says:

    An odd comparison:

    Jordan Mechner
    Average Movie career score: 50
    Average Game career score: 87

    Steve Purcell
    Average Movie career score: 73
    Average Game career score: 65

    Dan Houser: 90 (Rockstar)

    And some bigger names from the European action/adventure scene:
    Michel Ancel: 79 (Rayman, Beyond Good & Evil)
    Frederick Raynal: 66 (Alone in the Dark, Little Big Adventure)
    David Cage: 84 (Fahrenheit, Heavy Rain)

  68. 28/03/2011 at 16:21 Derek Smart says:

    These fuckers are just dying to be sued.

    Apart from that, the system isn’t even weighted correctly. Someone needs to go back to math 101.

    You can’t give the points of a game to a sole person when in fact it takes a LOT more people to create the game.

    Apart from that, my guess is that most of us in that list have made more money than the next person because it is widely known that a LOT of highly reviewed games LOST money.

    So wtf is this system for then?

  69. 28/03/2011 at 16:42 Dan Forever says:

    You also have to bear in mind that the credits listings are incomplete (neither Metacritic nor Mobygames have quite the ability that IMDB does when it comes to keeping credits up to date, yet).
    Cevat is only listed for Crysis 1, hence the score of 91 (clicking on many of the developers listed in the credits will also only list them as having worked on Crysis and so they’ll also have an identical score).

  70. 28/03/2011 at 16:44 Derek Smart says:

    Yeah, it’s rubbish. But that didn’t stop crazy execs (see EA and Take 2) from taking stuff like this seriously and basing a game’s success or failure on someone’s PERSONAL and aggregated metrics.

    Crazy stuff.

    • 28/03/2011 at 18:28 Deano2099 says:

      Kind of interesting though isn’t it. People generally think assessing games based on metacritic score is a horrendous practice, but it’s at least a step up from basing it entirely on sales like we used to?

      I don’t know how else you could do it, if you base it on a small sub-set of reviewers then you encourage the marketing people to focus on those publications… And you can’t just hand it to an exec and say “play it and see if you think it’s good enough to pay a bonus” either …

  71. 28/03/2011 at 17:04 champagneivy says:

    Hahaha my metacritic score is ten points higher than Peter Molyneux’s. I never realized what an industry big shot I was.

    This whole thing is completely ridiculous though. I have a high metacritic score for art credit on a single very successful game. John Carmack has a 78 despite being a mega-genius. It would seem to make a little more sense for people like Carmack and Molyneux than for artists or coders though, whose individual contributions become mashed up into a shared communal score, but again, obviously a complete failure if Carmack has a 78.

    • 28/03/2011 at 17:07 Stoloniferous says:

      Thanks goodness GameFAQ has such shoddy records of the games I have worked on. To them, I don’t exist. I would have feigned insult at that a few days ago. Now I’m relieved, because it keeps me from having a Metacritic rating.

  72. 28/03/2011 at 17:15 Malibu Stacey says:

    So how’s it work if you’ve a common name like John Smith or Dave Jones or something? How do they know simply from a credits list whether the James MacLeod listed on generic angry man shooter is or isn’t the same person as listed on indie puzzler or Space 4X game or whatever?

    • 28/03/2011 at 17:53 Stoloniferous says:

      It works poorly. One of my coworkers’ scores is all muddled up with another dev by the same name.

    • 28/03/2011 at 18:20 bob_d says:

      Plus the credits are incredibly incomplete – I couldn’t even find a friend with a dozen+ titles under his belt, but instead it showed me an actor of the same name.

  73. 28/03/2011 at 17:19 Joof says:

    This whole thing reminds me of the movie Dead Poets Society, where they are reading about the chart for ranking poetry. Then they rip it out of the book because it’s really really dumb.

  74. 28/03/2011 at 18:20 Barman1942 says:

    Adam Sessler’s take on Metacritic:

    http://www.youtube.com/watch?v=0QsXrswJ-yM

  75. 28/03/2011 at 18:26 bluebottle says:

    So, anyone want to help me with my site that reviews and gives a numerical score to review-aggregation sites? I’ll going to be called metacriticcritic.

    If we get enough sites doing the same we can then set up another site that aggregates all their scores, and call it something like metametacriticcritic. The circle will then be complete*.

    *Which hopefully will cause the whole inbred scene to collapse into a pointless hole of retardation, where it belongs.

  76. 28/03/2011 at 18:42 Hodge says:

    This is just the beginning of the beginning.

    Within a few years, game development companies will have completely restructured their dev teams so they consist entirely of Metacritic ninety-or-aboves. Employee salaries will rise commensurately with Metacritic scores. Soon after, an unofficial blacklist will be adopted industry-wide to ensure that no ‘eighties’ remain to dilute the pool (famously, Jonathon Blow will be let go by EA because, although he has a Metacritic score of 93, he ‘reads more like an eighty-five’). Later still, governments around the world will adopt the blacklist and make it the central plank of an internationally recognised industry code of conduct.

    As a result, development teams will dwindle in both size and number as the dead wood is cut. The money saved in wages will be redirected to the games themselves, leading to an astronomical rise in game budgets. This combination of big money and small teams means that game development will become a protracted but lavish affair. Characters – even minor roles with one or two lines – will be voiced by the biggest Hollywood stars and cut-scenes directed by the likes of Fincher, Tarantino and the Coens. Development time will stretch into decades, to the point where the console manufacturers begin to fold as game systems live out their entire seven-year lifespans before a single game is completed to run on them.

    This will allow the development companies to swoop in and acquire the rights to the consoles for a song, effectively allowing them to control the release dates of the consoles to coincide with their big game launches. They soon will take it one step further, merging the two that each game is sold on its own dedicated console – thereby guaranteeing hardware sales and eliminating piracy. Encouraged by the windfall in sales this will surely bring, the game companies inflate their budgets yet again, and invest millions in dedicated hardware for their upcoming releases.

    Around this time, the industry will attract criticism from some quarters. John Romero will point out that, since adopting the Metacritic policy, the industry has shrunk to a fraction of its original size and spent millions of dollars, yet has failed to deliver a single game. Bill Roper will simply respond, ‘that’s exactly what I’d expect a 75 to say.’ But Romero’s observation will ring true – with no new products to bring in revenue, the game companies will finally begin to exhaust their cash reserves. With games now fundamentally tied to hardware, the softcos will be reticent to release anything – as soon as anyone announces a launch date, their competitors can announce a higher-specced system/game for the same date and effectively corner the market. Basically, the first company to announce will be doomed to failure, and so they all do nothing. One by one, the software publishers will run out of money and be acquired by their competitors. This climate of mergers and acquisitions will continue until the entire industry has conglomerated into a single company – Übersoft.

    The perfect gaming company: staffed exclusively with the industry’s highest Metacritic-approved talent and with a total monopoly over both the software and hardware markets. With no competitors to threaten their dominant position, Übersoft will immediately release their meisterwerk: Halo Medal of Duty (Singing Pony Edition). Larger than any other game ever, and making full use of its built-in dedicated renderfarm, HMoD(SPE) will be met with universal critical acclaim. The one remaining gaming magazine, Edge-O-GamerTM (the magazine industry having shrunk and consolidated in parallel with the games industry) will be ecstatic after decades of having no product to review, and in a fit of unbridled enthusiasm will award the game a perfect score of 100.

    In spite of this, the game will be a complete commercial failure. Gamers in a recession-riddled economic climate will be reluctant to splash out US$750 just to play a single game and within days the game will be cracked to run on any jailbroken iPhone, resulting in the highest piracy rate of any game in history. It will, however, prove to be a huge financial success for Übersoft. By this time Metacritic scores will have long since become the only metric by which a games company is judged. As Übersoft is technically a new company which has only released a single title, it now has an unprecedented perfect Metacritic score of 100. Übersoft’s stock price will surge as a result, making it the single wealthiest entity in human history, dwarfing the economies of China and the US.

    It will be the story of the decade, the Facebook of its time. The Time Magazine issue with the cover featuring Übersoft’s logo and nothing else will be the highest-selling ever. The Spielberg-directed docu-drama will break records at the box office. Families, businesses and governments alike will invest their savings in the one true company, as the share price continues to surge to stratospheric levels.

    ——

    Several years later, a retired games journalist will be riling around the $2 bin at his local chemist (an odd place to be looking for games for sure, but he’s acting on a tip from an old friend who always had a great eye for bargains) and he’ll come across a boxed Halo Medal Of Duty (Singing Pony Edition). He’ll indulge in a quiet snigger – the story of game’s failure is legendary by this point, having famously been recalled after two weeks and quietly buried as news of Übersoft’s market fortunes was spreading. He will briefly think it ironic that he found it in a chemist owned by Übersoft, before realising that since Übersoft own everything it would be ironic to find it anywhere. Still… remarkable that this one somehow slipped through the net. He’ll eagerly pay his $2 and run home to to see if it lives up to that famous Edge-O-GamerTM review.

    Soon after, he will post a review of sorts on his blog. In this review, he will make note of the game’s incredible graphics (although nominally remaining a games company, Übersoft will not release anything else after HMoD(SPE)’s failure, effectively ending the games industry. As a result, HMoD(SPE) will remain technically cutting-edge, even years after its release). The review will also praise the game for its vast size and variety of quests, but will also voice disappointment at the barren landscape, poor writing and the game’s general lack of ambition. In the weeks after the review is posted, the retired journalist’s blog will be inundated with comments from fans of the game saying that the review was far too negative.

    Meanwhile, sole remaining editor of the dormant-for-years Metacritic site will come across the review and, overcome by a wave of nostalgia, feel an obligation to update the HMoD(SPE) page. He will see that the new review has no score attached, but he will also see the dozens of angry comments citing it as too negative. To incite that kind of reaction, the review must indeed be quite negative, so he will arbitrarily assign the review a score of 3/10. He will update the Metacritic site with this new information, giving the game a new aggregate score of 6.5/10.

    News of the updated Metacritic score will spread first through the gaming blogs, and then through the business and finance journals. As Metacritic is still the sole metric by which game companies are judged and Halo Medal Of Duty (Singing Pony Edition) is still Übersoft’s sole release, Übersoft’s standing will plummet overnight, triggering a share price free-fall. This will immediately cause a worldwide depression, as families, businesses and governments alike realise that they’ve lost millions on their investments.

    Übersoft’s Board of Directors will call an emergency meeting to discuss ideas on how to turn the fortunes of the company – and of the world – around. Unfortunately, the outlook will be grim. As Übersoft fortunes were a direct result of its Metacritic reputation (which had now been shot to pieces), the only way to improve things would be to write some well-received games to try to bring the rating back up again. But the new review will have dragged Übersoft’s entire programming roster down below the 90-or-more Metacritic threshold, and Übersoft will be legally obligated to let them all go, as per the industry code of conduct. Even worse, this will mean that every single developer on the planet has a score below 90, so it will be illegal to hire any of them. The only way to change this would be to lobby governments to abolish the code of conduct, but this will require money which Übersoft doesn’t have.

    Realising that they have painted themselves into a corner, the Übersoft directors will opt to simply dissolve the company unofficially and go into hiding. Meanwhile, riots and looting will become commonplace as the Übersoft crash resets the entire world economy to zero.

    This will be the beginning of the end.

    And that is why Metacritic is silly :).

  77. 28/03/2011 at 19:01 Carra says:

    I rate Alec 92%.

  78. 28/03/2011 at 19:30 Derek Smart says:

    @ Hodge

    I frigging read ALL that. Can I have my 30 secs back now please? ;)

  79. 28/03/2011 at 19:32 Grayvern says:

    I want to say that metacritic have lost all shreds of social responsibility by doing this but at the same time given the reliance on quantification I want to say that this means metacritic is now lawful evil, the dilemma.

  80. 28/03/2011 at 22:31 Pretzel says:

    So, I gots me an 89%, which puts me about 18 of the 22 people on that list. I always knew I was great. Suck it Molyneux!

    Strange that no one’s heard of me. ;)

  81. 28/03/2011 at 22:56 Moonracer says:

    If they wanted to delve deeper in reviews how about a system where review sites get reviewed and their score influences their rating of a game’s effect on the total. That way a site with a 50% approval rating would only have half the effect on the total score as one with 100% approval.

  82. 29/03/2011 at 00:17 rammjaeger says:

    I’m not quite sure what to think about this. If your number is high, then it is bragging rights for sure. But your number could be low, and it has nothing to do with you. You could be an amazing programmer, but work at a company that has crap leadership and designs bad games. And that wouldn’t be the programmers fault. I hired a brilliant programmer that worked on 1 game that had a really bad metacritic score.

    Also, the system is highly inaccurate. For instance, looking at my score for John Gibson, I’ve got an 80. But it lists a bunch games I didn’t work on (Motorstorm, Star Trek, Family Feud), while leaving off games I did work on (Killing Floor, America’s Army Special Forces). The only game it got right was Red Orchestra.

    So anyway, by Metacritic’s reckoning, I’m cooler than John Carmack because he has a 77. And that guy is infinitely cooler and more successful than me in reality. Not to belittle my own achievements, but that guy created Doom for petes sake!!! He invented the FPS genre for all intents and purposes.

  83. 29/03/2011 at 03:19 Jason Moyer says:

    Did they remove the developer career scores? I looked up Emil Pagliarulo (who I’d imagine would rate above 90) and there’s no career score listed.

  84. 29/03/2011 at 05:39 MultiVaC says:

    Oh god, according to this Cevat Yerli is the greatest video game designer that ever lived. I already cringed when I saw “A Cevat Yerli Game” in the intro of Crysis 2. Now get ready for “Cevat Yerli Presents Cevat Yerli’s Crysis 3: MAXIMUM CEVAT YERLI (A Cevat Yerli Production)”!

  85. 30/03/2011 at 01:35 lordfrikk says:

    Haha this brings up very bad memories of World of Warcraft and the infamous Gearscore addon, which basically boiled your character down to a single number based on the absolute values of your gear and didn’t really tell you anything about the skill of the player or even if the gear is correct for that specific class or build. Still, it managed to take over the recruitment to groups on a lot of servers so when you entered a city you mostly saw “LFG (Looking for Group) DUNGEON_ACRONYM GS_VALUE+” where GS_VALUE was usually a number so high for the selected dungeons that people got flamed to all hell. It didn’t cease one bit despite this, though… it probably looks like that even know, but I wouldn’t know since I stopped playing some time ago…

Comment on this story

XHTML: Allowed code: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Respond to our gibber

Read our finest words

Wot I Think: Call Of Juarez – Gunslinger

Search for clues

Play indie, mods, demos

Browse the archive