Score: Metacritic Drops Dev Ratings

By Alec Meer on March 29th, 2011 at 10:35 am.

You can have an S in 3D now?

And there was much rejoicing. Following a web-wide waggling of worried eyebrows (of which I was just one of a great many), Metacritic has done the decent thing and removed its ratings of individual developers.

Mr 8%? It’s okay. It’s over now. You’re free.

Says the site’s games boss Marc Doyle, in comparing the dev ratings to a similar system used for movie actors and directors:

“This career score is not an independent evaluation (or an aggregation of reviews of) the individual person in question – it’s a simple average of all the individual Metascores assigned to those movies the individual worked on.”

Alas, while that may have been the intent, when you publish a guy’s name with a number next to it it’s very hard not to treat it as a rating.

However, he acknowledges in response to the “enthusiastic discussion in the gaming press and social media” that “although our credits database (which is powered by our sister site GameFAQs) is growing, as our users’ feedback has indicated, it is a work in progress and is not nearly as comprehensive as it needs to be to accurately provide a career score for these individuals.

“As such, we have removed that career score from the pages dedicated to creative individuals behind games on Metacritic.”

So that’s that. Good on them, quite frankly – it’s a site subject to a lot of criticism (some fair, some not) so picking out and acting on particular complaints is no mean feat. They’re still wanting to build a more comprehensive credits database however, and seem aware that the ones on GameFaqs are currently far from that. It could be a useful tool for hacks like us if they do get it right.

__________________

« | »

.

47 Comments »

  1. Malibu Stacey says:

    They’re still wanting to build a more comprehensive credits database however, and seem aware that the ones on GameFaqs are currently far from that. It could be a useful tool for hacks like us if they do get it right.

    http://www.mobygames.com/ perchance?

    • Matt says:

      One would think… except MobyGames is owned by GameFly Media while GameFAQs & Metacritic are owned by CBS Interactive, so it’s rather unlikely they’d use the better (MobyGames) site for sourcing credits.

    • Hunam says:

      All I know is that I’m on Mobygames and that’s all that matters :)

    • Malibu Stacey says:

      Well I was sort of replying to Alec’s comment that “It could be a useful tool for hacks like us” more than it being better suited for Metacritic since it’s far more comprehensive than the GameFAQs site.

  2. Hoaxfish says:

    I thought I’d look up David Gaider since he was involved in the trainwreck of DA2′s “over 9000″ negative user reviews… the site doesn’t even recognise that he worked on it?

    • Javier-de-Ass says:

      Credits do not land on these sites by magic, someone has to type it in. So for new games, especially on mobygames, it takes a while for new submissions to be sent in and approved and that.

    • Hoaxfish says:

      Well, since he’s only creditted with BG2 and NWN… I’m guessing he’s fallen through the cracks when it comes to the site’s accuracy.

      Which of course, makes me wonder what/who else is simply unaccounted for.

    • BobsLawnService says:

      Being a developer who is overlooked by metacritic seems to be a good thing though so I’m sure that they are not complaining.

  3. deejayem says:

    I’m actually a bit surprised that Metacritic seems to hold so much sway over publishers’ decision-making. I sort of thought they were all faceless suits following the all-powerful consumer dollar regardless of critical opinion.

    • poop says:

      I dont really think they give a shit about what critics think as much as they realise that if a game scores poorly even when you bribe as many reviewers as money can buy it means the game is a bit of a turd.

    • DJ Phantoon says:

      No, they realize idiots will base their judgement off nothing more than that score, and therefore stick to safe and easy projects like MANSHOOT KILLER 2011 rather than something that may not succeed.

    • Dances to Podcasts says:

      Even most publishers understand that a franchise can only take so many critical hits (…) before it becomes lifeless and unprofitable.

    • Urthman says:

      Publishers caring about Metacritic scores seems bizarre to me as well. Even if there were a pretty close relationship between Metacritic scores and sales figures (which I doubt), the publishers have the actual sales figures, so why would they need to bother with the Metacritic scores as well? Surely the actual sales figures are all that really matter?

    • drewski says:

      Sales figures probably have more to do with marketing budget than game quality, though.

      I suspect, however, that game companies know that really there is only so much polishing that can be done of a turd, so they want their devs to make games which rate highly in Metacritic as well as have a big marketing budget behind them, because it’s one less risk in a game with a big investment not selling.

  4. pyjamarama says:

    Metacritic decision is more a reflection on the sad state of game credits, MobyGames is the best resource but it’s not complete and always accurate. Games company’s seem to have no interest on maintaining there own complete and accurate data, with some exceptions they prefer that gamers associate the games with the company and not the creators.
    So I hope metacritic can succeed on creating a accurate database for creators and they can put, or not, the average score with it.
    I still think that the fears of publishers using the database in meaningful way were completed overblown and I find odd that game press is so obsess with metacritic when other media, Movies, for witch there are databases for the creators no one seems to mind.

  5. Branthog says:

    I don’t understand why there’s so much emphasis placed on Metacritic by publishes and developers. I’m an avid gamer and I never *ever* visit Metacritic. It wouldn’t even occur to me. It seems like the place you go to review videogames and movies when you’re done commenting on news articles that you linked to from Drudgereport or something. Unwashed masses and all.

    • ReV_VAdAUL says:

      It is quite useful to have easy access to a range of reviews. Plus one can use one’s own judgement to weight the reviews and decide which ones to trust without buying into their aggregated score. So its easy to use and popular which means a lot of companies conflate that with it being important

      Plus a less charitable soul might say companies like metacritic because its a handy database of how effective their bribes were which games journalists are most worthy of their advertising budget and exclusive previews.

    • Robert says:

      The point is that you usually see it when you buy it. Can be on the packaging in the store, or if you buy digital: Steam has it for example.

      When you grab the box and take it to the desk to buy it / when your (mouse)finger travels it’s way to the ‘Add to cart’ button, your eye will linger on the 2 digits that is the metacritic number. Will you still buy that game if it says 60? Will it push you over the ledge of doubt if it says 93?

    • Leelad says:

      If it wasn’t for steam I’d know nothing about it. I think the only reason anyone goes on it is to see top rated and bottom of the pile in the different categories.

      Always fun to pick a few from the bottom of PC games to see how bad they really are.

    • Hoaxfish says:

      I visit it to see the hilarious low scores things get (recent visits for FF14, and DA2).

    • mcwill says:

      It’s vaguely convincing to suits, and suits control the games industry. That’s about all there is to it.

    • Ginger Yellow says:

      “I think the only reason anyone goes on it is to see top rated and bottom of the pile in the different categories.”

      I use it when I want to read a lot of reviews of one game, as it gathers links to them all in one place. Alternatively, if it’s an obscure game that may not have been reviewed by mainstream sites, I’ll check if any reviews are linked from Metacritic. But the score itself is irrelevant to me.

    • drewski says:

      Publishers use it as a handy shortcut to how good the game was generally accepted as being, without having to worry about individual bias.

      Developers use it because publishers consider it, and publishers have the money.

  6. Navagon says:

    So can we chalk this up to another RPS victory for great justice? Or do we have to take into account all those other shifty looking types with gaming websites?

  7. Leelad says:

    Bat signal suggestion: http://bit.ly/dKB8mb – That head – cell shaded -BOOM.

    • Chris D says:

      That would certainly strike fear into the hearts of evil-doers everywhere. Also: small children, old ladies, mediocre games-journalists (Not RPS of course), and pretty much everyone else.

  8. Leelad says:

    Hmm was meant to be a reply.

  9. SuperNashwanPower says:

    So who was the “Last, Best Hope of Humanity” according to the scores? Can we know that, or is it classified now?

  10. Deano2099 says:

    Worth noting:

    From that, Metacritic removed the ratings as they felt (correctly) that their database wasn’t comprehensive enough for the numbers to mean anything.

    They had no problem with the methodology, the lack of position-based weighting, or the fact that publishers might use those numbers for dodgy hiring practices.

    They only removed the data as their dataset is too incomplete. And so they should have. I’m sure they’ll be back though, and I hope so as they’re sort of interesting. Frankly, it’s not metacritic’s job to protect the games industry from it’s own dodgy hiring/bonus practices.

    • bob_d says:

      “This career score is not an independent evaluation (or an aggregation of reviews of) the individual person in question – it’s a simple average of all the individual Metascores assigned to those [projects] the individual worked on.” In other words, ‘it isn’t a score meant for the evaluation of individuals, but a (supposedly) evaluative score that’s simply associated with individuals – it’s completely different! How could anyone be confused!?’
      They just don’t get it. The gaping holes in the credits data is the least of the issues here. The basic problem is that the data just doesn’t have any meaning whatsoever when applied to individuals. It’s a data set being used in a meaningless way, but presented as if it did have meaning. Moreover, given that it can cast developers in a negative light, regardless of what the industry actually does with the data, it’s fundamentally their problem, not the industry’s.

  11. bascule42 says:

    I bet it was Sion Lenton with an army of two lawyers….

  12. adonf says:

    It’s based only on names, so if several developers have the same name they will receive a rating that is not only a non-weighted average, but a non-weighted average of a developer’s own games and some other, unrelated person’s games too. It happened to me with a Canadian developer who has the same name as me. Mobygames thinks we’re the same person too, it doesn’t bother them that I’m a programmer and the other guy’s an artist.
    So yes, dropping this is a good idea.

  13. Bloodloss says:

    I don’t really get all the complaints about this. Instead of looking up the person they’ll just look up the last game or two that they worked on. This seemed like it might have punished people for working on awful shovel ware and whatnot, which could’ve been good.

    • Berzee says:

      Except that working on awful shovel ware doesn’t indicate you must be a bad developer; possibly a hungry one. =)

    • Berzee says:

      But what you say makes sense if you’re thinking in terms of Lead Designers, perhaps.

    • bob_d says:

      As a game developer, let me just say: FUCK NO. (Pardon my language.)
      First: most game developers have very little choice about what projects they work on (even most leads, honestly), even if they take jobs with the understanding they’re going to work on particular projects. This almost never happens, though; frequently when interviewing for jobs, they won’t even tell you what you might be working on, since the titles are unannounced at that point. I once took a job at a company understanding that we’d be working on one project, but that was canceled before it started, replaced with a rather poor licensed IP deal that ended up being severely underfunded (we had perhaps half the team and less than half the time we needed to do the job). Quitting wasn’t exactly an option, as no one wants to hire a worker who’s likely to leave at the first sign of trouble.
      Second, new studios (and green developers) usually have to survive doing underfunded work with ridiculous time constraints (aka “shovelware”). With this scoring system, those games would weigh down an entire career.
      Third, a game can get a low score without being “shovelware.” No one wants to make a bad game. For many games, it isn’t clear if the game will work or not until the end of development. A game may be unsuccessful in only some ways. (Which also means that an artist’s metascore would drop if their games had design or programming issues.) Issues with marketing and publishers can greatly impact how a game is received. I’ve seen more than one game destroyed by publisher shenanigans. Bad ports of games would drop the original developers’ scores. AAA development teams have upwards of 60 people; even if metacritic scores actually represented the “quality” of a game, and even if we make the (frankly ridiculous) assumption that the developers had total creative control and were solely responsible for the quality, is it remotely fair to use that score to represent any individual’s efforts on that project? No, it isn’t.
      A high metacritic score simply represents a great deal of sheer good luck on the developer’s part, that they were able to associate their name only with games that met with wide approval, nothing more.

    • Cunzy1 1 says:

      I don’t get these arguments at all. It’s not okay to release bad games. I don’t care whose fault it is and I don’t care if a game would have been good with a little bit more time.

      What’s the point in pretending that developers haven’t been behind some really bad games? It’s not your fault a game you worked on stinks? Well, sorry but it is. If it isn’t good then don’t release it and if you are releasing a shitty game well take some sodding accountability for it. Yet another example of the games industry seriously needing to grow up.

    • Berzee says:

      Edit: I tried some analogies and they didn’t work.

    • bob_d says:

      @Cunzy1 1: You are under some strangely hilarious delusions about how the game industry works. “If it isn’t good then don’t release it…” As the kids say on the internet: LoL wut? That’s so naive it’s adorable.

  14. ScubaMonster says:

    I don’t really care one way or another, but why would you care what a developer’s score was? I’m not going to assume a game is crap because it’s made by a certain developer. Or that a high scoring dev has great games.

  15. Fumarole says:

    I still find it useful for film directors. Just see Uwe Boll’s rating on Rottentomatoes.

  16. Fyr says:

    Oh the irony.

    I was excited about Spore before it was released but didn’t want to buy it at that point because it wasn’t on Steam. After it was released it became apparent that it sucked, er I mean, was nothing to get excited about.

    Having not played Spore I have no desire to play a spinoff.

    And they put that up for beta… on Steam.

  17. Cunzy1 1 says:

    @Bob_d HA! Not so much naive as idealistic.
    I still maintain it’s pretty sad that the way the industry works is to not actually ‘work’ much of the time. Sadder still that people are calling for evidence that the industry is broken, risky, immature, unstable and outright childish to be buried.

    UPDATE: Reply fail.