Study suggests Steam reviewers are bothered more by bad game design than bugs

steam-review-bombs

A study of over 10 million Steam user reviews has found that when reviews include negative feedback, they call out poor game design more often than they complain about bugs. Which is entirely unsurprising, really: I know that infuriating design decisions tend to stick in my head far more perniciously than occasional bugs. That’s not the only insight that the researchers from Queen’s University in Canada have to offer though, and I’m finding the fine-grained details of their study much more interesting than that headline conclusion.

When looking at all reviews that included negative feedback, the researchers found that only 17% of them mentioned bugs, while 57% talked about problems with game design. It’s also interesting that 42% of reviews that mentioned bugs were positive, “suggesting that having bugs in a game does not necessarily lead to negative reviews”.

Again that’s fairly obvious, though I’m mildly surprised that so many Steam reviewers are willing to point to bugs that bother them, yet are still positive overall. It’s worth noting, though, that we don’t know how many of those mentions were throwaway comments along the lines of ‘I had one minor bug that didn’t bother me’, or even ‘I didn’t encounter any bugs’.

While I’m on the subject of caveats, let’s deal with another major one. Extrapolating from Steam reviewers to players in general would be a mistake, given how only a small percentage of people that play a given game will actually leave a review (I just went looking for a specific statistic, but couldn’t find one). To me, that means the study works best as a way of offering perspective on how Steam reviewers think rather than providing developers with valuable information.

This is a good example: the medium time before a review gets left is 13.5 hours, but the majority of negative ones are left in the first 7 hours. For free-to-play games, there’s a peak in reviews after the first hour. That leads the researchers to suggest that “developers should pay particular attention to the design of the first 7 hours of gameplay, as the majority of negative reviews are posted within that period”. That’s still a good tip, but devs worth their salt will already know about the importance of a good first impression.

Here are some more tasty snippets: the average Steam review is 30 words long, and is at an eighth grade reading level (according to the Coleman-Liau readability index used by the US Office of Education). 42% of reviews included what the researchers deemed to be valuable feedback, categorised as such based on whether they made specific comments rather than just expressing an emotion.

The negative reviews contained more valuable feedback than the positive ones, though the researchers were keen to emphasis that positive reviews shouldn’t be ignored.

These are all statistics that offer potential insights into the minds of people who leave Steam reviews, though Rick Lane recently took a more direct approach and asked some of them about it. He also looked at the other side of the coin, and asked devs what they thought about Steam reviews.

You can comb through the Queen’s study for yourself here.

Cheers for pointing this out, Gamesindustry.Biz.

25 Comments

  1. skyturnedred says:

    Makes sense, bad design affects pretty much the whole game whereas bugs are an occasional irritation.

    • Shadow says:

      Questionable design is also much less likely to be “fixed” over time, since devs may well not see a problem.

      Working in a software company, I know that all too well.

      • Xocrates says:

        Though it might be even more likely that they do see a problem but either don’t know how to fix it or don’t have the resources to do so.

        • Shadow says:

          That may well be the case if the design complaints are widespread and are hitting the game’s success hard. Sometimes budgets run out and changing the course of the design is costly enough it’s just not viable.

          • Hedgeclipper says:

            But its not uncommon for even successful developers to get irrationally attached to idiotic design. See the debate over save games for instance.

  2. HiroTheProtagonist says:

    On the one hand, bugs are usually sporadic or can be somewhat easily remedied with patches made officially or by fans, so an otherwise good game with bugs still shines through. On the other, a polished title with generally dull or bad gameplay would need overhauling modification to be made good, and most games lack the tools or dedication in the fanbase to make that so.

    Just look at STALKER. Shadow of Chernobyl was completely broken on launch and still requires fan-made patches to run properly, but the gameplay still keeps people playing over a decade later. By the same token, Loadout is shutting down due to lack of interest, and that game was colorful but otherwise boring and lacked any kind of unique gameplay to hold attention.

  3. Flavour Beans says:

    I wonder how much of a boost that reading level gets because of the subject matter. Coleman-Liau is entirely driven by the length (in characters) of words and the length (in words) of sentences. Perhaps computer gaming–with a vocabulary that includes ‘uninstalled’, ‘framerate’, ‘disconnected’, ‘pixelated’, ‘first-person’, and more–inherently uses longer words than average. I also wonder how the sentence length factor comes into play when you have run-on sentences or a lack of punctuation.

  4. Robert The Rebuilder says:

    how only a small percentage of people that play a given game will actually leave a review (I just went looking for a specific statistic, but couldn’t find one)

    GamaSutra just posted an article about this. 1 out of 82 people who bought the game have written a review. Source: link to gamasutra.com

  5. Don Reba says:

    Nier: Automata’s bad PC port feels like it is on the border line between being buggy and badly designed. I’m still holding out on it until it is patched.

    • Arglebargle says:

      I did not buy it for just this reason. Not much tolerance for continual irritation, especially if it could have been easily (relatively) avoided.

    • Premium User Badge

      Mikemcn says:

      There is a mod to help with that stuff! I never really had any issues with the pc version but i also never tried without that mod. (Whose name escapes me.)

      And it is super dumb how often these console ports need a DSfix or something to become functional.

  6. Someoldguy says:

    Reviewer uses the word perniciously in the article. 10/10 would read RPS again.

    “Reviews” like that really annoy me in steam. The player can be bothered to write something but can’t be bothered to make it informative or meaningful. I can rarely be bothered to write anything myself and usually only do so when I want to counterbalance overheated reviews or praise a game that needs added publicity. Adding review 5001 to an already popular and well praised game (or universally panned one) doesn’t interest me.

    • wengart says:

      I write reviews like that because I expect people to use the review system in the same way I do. Which is to look at the wisdom of the crowd.

      If 10,000 people like a game with a 80% positive I can infer that it is a good game in the genre, but not best in class. I don’t care to read any of their individual thoughts because they are individually reliable. My short review is just there to quickly add to the +/- crowd decision.

      • Arglebargle says:

        I read the reviews for technical information that will tell me whether I’ll like the game or not. There are certain game design or technical design issues that will spoil my gameplay, even if the greater body of players don’t care.

        Sometimes the many glowing reviews can cite specific points that tell me to stay far away.

      • mike69 says:

        That’s a terrible way to use the review systems because it’s not how it works. Even some ‘mixed’ review games are brilliant, but have bad reviews for the most obtuse reasons. If that’s all you’re doing you should honestly stop wasting the energy.

        The percentage is useful as a super high-level thing, but it’s the content of the latest reviews combined with the top reviews that gives the context.

  7. nimbulan says:

    And then you have the times when both bad design and lots of bugs are present, and you end up with Batman: Arkham Origins.

    In any case, I have to personally agree with the study. Bad game design is simply infuriating and bugs aren’t generally a major issue unless they regularly interfere with gameplay (hell many are often quite amusing.)

  8. mike69 says:

    That’s because bugs are a technical support issue, and reviews are people’s opinions on the design of a game.

  9. fuzzyfuzzyfungus says:

    I wonder how much, if at all, this is skewed by the really epic disasters(usually trash console ports these days; or really really early early access) often attract enough attention that it discouraged buyers at least until the vendor patches drop or 3rd party shims are known to mostly work, ish.

    There have certainly been a number of high profile “run screaming unless this is fixed” cases; but “FFS don’t pre order!” is also a perennial topic, so perhaps those warnings don’t actually stop many people; I’d be curious to know if the true bugfests often don’t get the bulk of their reviews until they’re stabilized a bit(maybe we could measure by looking at average period of time between release and review?)

    The other factor, at least if you are my age, is that in relative terms we have it pretty good these days.

    It was not so very long ago that a hard lock or crash to desktop, on occasion, simply wasn’t all that exceptional. That’s why we could just “save”, without using “save points” like savages, and save we did, fairly frequently. It wasn’t always the game’s fault, either, until NT-based Windows having the OS fall over and die was pretty routine; and having the OS just shrug and blink for a second when the GPU driver falls over and dies is substantially newer than that(and, at some point, “sound card drivers” went from being a cryptic matter of fear to basically being free with every motherboard not sold in a 1U server case).

    Game stability, as well, seems to have improved. Quite possibly because there is less rolling your own and more Unity or equivalent; but stability is stability. In-game bugs are still pretty common(ie. broken dialog tree makes quest unfinishable, rather than “casting fireball crashes the game when using AMD GPU driver versions lower than X.YZ”); but those are also the ones that are more likely to have an unofficial fix(“hey Bethesda, funny you should happen by…”)

    It’s hardly bug free; but it’s really been a while since I felt the old “the load screen paused for a second, has it frozen? can’t alt-tab to check or it will definitely freeze…” twinge whenever a game stuttered for a moment; and games where you know that you really have to save before certain events(eg. loading screen, transition to cutscenes, similar changes in operation) because the odds of having to retry at least once, sometimes repeatedly, to make it through without crashing are something I haven’t really run into in a while.

    Also, possibly thanks to GPU improvements, possibly game engines, possibly both; it’s been quite a while since I’ve had to memorize a menu so that ai could “dead reckon” my way past it when graphical corruption turned it into a mass of illegible colored squares and made the mouse pointer invisible. That was never an amusing activity.

  10. Vandelay says:

    When we say that these reviews are mostly concerned with bad design, on what level is this study actually judging whether what the reviewer is calling bad design actually is bad design? If the answer to that is “not at all”, then the study seems rather meaningless to me.

    Just because someone leaving the review really dislikes the way a game has been designed (for example, as you have Firewatch pictured above, features no combat and is driven mostly by its narrative), that doesn’t necessarily mean the design of the game is actually bad. At most, it just means the game is more niche.

  11. Greg says:

    I can still want to play a game with a bug, but bad game design is a game killer. Those negative comments that developers hate have saved me from making a bad purchase many times. In this day and age I’m still seeing negative comments because a developer couldn’t be bothered to add a save system into a game. This info shouldn’t be hidden from gamers.

  12. temujin33 says:

    I bet most eighth graders would have spelled “emphasize” correctly…

  13. GemFire81 says:

    Lol!!! Why would you need a study to realise that bugs and pretty geaphics dont matter if the game is total crap…

  14. Turkey says:

    I’m jealous of the people encountering fun bugs. Whenever I get one, it’s usually just a freeze or a CtD.

Comment on this story

HTML: Allowed code: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>