Study suggests Steam reviewers are bothered more by bad game design than bugs

A study of over 10 million Steam user reviews has found that when reviews include negative feedback, they call out poor game design more often than they complain about bugs. Which is entirely unsurprising, really: I know that infuriating design decisions tend to stick in my head far more perniciously than occasional bugs. That's not the only insight that the researchers from Queen's University in Canada have to offer though, and I'm finding the fine-grained details of their study much more interesting than that headline conclusion.

When looking at all reviews that included negative feedback, the researchers found that only 17% of them mentioned bugs, while 57% talked about problems with game design. It's also interesting that 42% of reviews that mentioned bugs were positive, "suggesting that having bugs in a game does not necessarily lead to negative reviews".

Again that's fairly obvious, though I'm mildly surprised that so many Steam reviewers are willing to point to bugs that bother them, yet are still positive overall. It's worth noting, though, that we don't know how many of those mentions were throwaway comments along the lines of 'I had one minor bug that didn't bother me', or even 'I didn't encounter any bugs'.

While I'm on the subject of caveats, let's deal with another major one. Extrapolating from Steam reviewers to players in general would be a mistake, given how only a small percentage of people that play a given game will actually leave a review (I just went looking for a specific statistic, but couldn't find one). To me, that means the study works best as a way of offering perspective on how Steam reviewers think rather than providing developers with valuable information.

This is a good example: the medium time before a review gets left is 13.5 hours, but the majority of negative ones are left in the first 7 hours. For free-to-play games, there's a peak in reviews after the first hour. That leads the researchers to suggest that "developers should pay particular attention to the design of the first 7 hours of gameplay, as the majority of negative reviews are posted within that period". That's still a good tip, but devs worth their salt will already know about the importance of a good first impression.

Here are some more tasty snippets: the average Steam review is 30 words long, and is at an eighth grade reading level (according to the Coleman-Liau readability index used by the US Office of Education). 42% of reviews included what the researchers deemed to be valuable feedback, categorised as such based on whether they made specific comments rather than just expressing an emotion.

The negative reviews contained more valuable feedback than the positive ones, though the researchers were keen to emphasis that positive reviews shouldn't be ignored.

These are all statistics that offer potential insights into the minds of people who leave Steam reviews, though Rick Lane recently took a more direct approach and asked some of them about it. He also looked at the other side of the coin, and asked devs what they thought about Steam reviews.

You can comb through the Queen's study for yourself here.

Cheers for pointing this out, Gamesindustry.Biz.