We Spoke To Developers About Steam User Reviews

Recent changes to Steam reviews, which filter out reviews from keys that weren’t purchased directly through Valve’s digital store, have caused all sorts of worry and concern. The intent is to remove false positives in the form of reviews exchanged for keys and the like, but legitimate reviews are also affected. Games that were Kickstarted no longer have their backers’ assessment contributing toward the rating Steam displays at the top of the page, and people buying through Humble Bundles or elsewhere are similarly excluded by default.

We contacted a variety of developers and publishers, including Larian, Stardock and Mode 7, to hear if they thought the move might stamp down on unfair practices, or whether it would end up hurting rather than helping.

Loyalty, abuse and manipulation

Developers who have run successful crowdfunding campaigns were among those who showed immediate concern about the changes, and Swen Vincke, founder of Larian (who ran successful Kickstarters for Divinity: Original Sin and its sequel) captured the feeling of many who got in touch with me:

Well, these are your most loyal fans and so you’re cutting them away from the sampling pool. That doesn’t really feel fair. We have 42K backers whose opinion will be hidden behind a filter and not be accounted for in the score. If we sell say 500K units (which would be a success for a top-down turn-based RPG), that’s almost 10% of our audience. I don’t know what the statistical significance will be – it doesn’t seem to have made a big difference for Divinity:Original Sin 1 – but I can imagine that in the early days, when the opinions of early adopters are important, it might hurt us that the adopters with the strongest interest in the game won’t be heard, at least not in a way that matters. And since initial opinions often seem to set the trend, that feels like a loss for us. But as I said, I do understand why they’re doing it. A score only matters if you can trust it and I can imagine there was a lot of abuse and manipulation going on.

Not everyone has to imagine the abuse and manipulation. Brad Wardell, founder and CEO of Stardock, reckons the changes are for the good as they’ll help to fix one of the main problems with Steam reviews.

I really like the changes. It’s been one of the worst kept secrets that some studios were gaming the review system by handing out copies to super fans so that they could launch with 99% review scores. What will be interesting is when people start to notice those who were doing this.

The Steam review system is still very punishing to certain types of games. Particularly games with high hardware requirements as most PC gamers think their 5 year old machine is still state of the art and won’t accept that a game plays slowly on it. That could be solved if reviews by players who were refunded weren’t included in the rating. But overall, the Steam review system is still the best system currently available in my opinion.

Sean Colombo of BlueLine Games says this gaming of the review system is a trend that he’s noticed recently, the studio having first released a game on Steam in early 2014. The way he describes it, studios are approached by people willing to provide reviews in exchange for keys, and he understands why some developers might be tempted:

There is a new trend that Eastern Euro/Russian players will contact you in hordes to offer Steam Reviews of your game if you give them a free copy. As long as they don’t promise a positive review (which is super sketchy), that actually might be worth it for a lot of devs since a small pool of reviews is dangerous… until you have a bunch of total reviews, one bad review puts you in “Mixed” territory which will basically stone-wall your game’s growth.

Throwing out the data with the bathwater

Founders of Mode 7, Paul Kilduff-Taylor and Ian Hardingham, who are currently working on Frozen Synapse 2, also recognise that Valve needed to deal with “a real problem”. Hardingham appreciates concerns that the solution causes its own problems though:

Valve is unusually flexible with allowing developers to register as many keys with their system as you like – allowing us to sell hundreds of thousands of Steam keys through a Humble Bundle for instance. It also allows us to send out many promotional copies and run closed betas in any way we see fit. This is incredibly developer-friendly in comparison to some other distributors: I would never want Valve to change it.

Exempting non-Steam user reviews from the overall score is a response to a real problem where developers are giving out free games in return for good reviews. I would much rather Valve did this than change their open approach to external keys. However I do think you could look to remove specific accounts which have questionable reviewing/activating behavour and possibly get 90% of the benefit without losing the large amount of valid data that’s being thrown out with the bathwater here.

Kilduff-Taylor thinks similarly:

It’s frustrating that the idiotic behaviour of a few unscrupulous people has, once again, forced changes to a system which affects everyone. I definitely appreciate that Valve need to find a way of shutting down this type of fraud, though.

There is certainly an issue here for developers who want to release alphas on their own site, then make the jump to Steam at a later stage. The players that get hold of a game early on often tend to be some of the most passionate and vocal: they now won’t get to contribute to the game’s overall score. It’s a small thing – and it’s great that they will still be allowed to leave comments – but it could potentially have an impact.

Ultimately, everyone should be striving to make games which are well received throughout their life-cycle by different types of player: as long as developers stick to that, they shouldn’t have anything to worry about here.

Navigating the mountain of shovelware

Developers are worried though and many of those worries seem to come from young studios and those that were already worried about issues on Steam, including visibility and the difficulty of not being buried beneath the “mountain of shovelware”. That’s a phrase used by Adam Blahuta of Jellyfish Games, who was originally excited by the prospects of Steam Early Access, but now feels it might not be the best place for his game:

…Steam’s Early Access program excited me when it launched. The two most obvious benefits were early feedback and early revenue. The former would challenge our assumptions about the experiences our game provides and the latter would allow us to improve the game’s quality and marketing material prior to launch.

At first, only the most eager of early adopters bought these unfinished games. They generally came with an understanding of what they were getting for their money and would frame their reviews accordingly. The more Early Access became popular, the more players started to flood in. There was an obvious shift in tone as the generally constructive pool of reviews started to get overshadowed by obnoxious negative reviews based on a misunderstanding of what one should expect of a game prior to release.

What made matters worse was the mountain of shovelware that made it through the now relaxed Greenlight approval process. As these titles started to garner almost as much attention as the quality games in development, this created a perception that most Early Access games were garbage.

This means that anybody who buys a game in alpha with incorrect assumptions or coming at it from a negative mindset is more likely to post a negative review. If the review’s criticism is undeserved, this will unfairly affect sales, which will turn away people who would enjoy the game. Once things start going in the wrong direction, it can be very difficult to come back from that, no matter how much work you put into your subsequent updates. This makes it incredibly important to come up with a sales strategy that mitigates the impact of these undeserved reviews.

Colombo, at the digital boardgame developer Blueline mentioned above, has experience with a launch that backfired, although not for entirely predictable reasons. His story highlights how another recent change to user reviews – the introduction of a recent reviews filter – can help when a game stumbles out of the gates. Particularly since players very rarely update their reviews, even when developers update their games:

Users will sometimes update reviews, but not often. This means you have to be really careful when you launch. We launched Simply Chess for free on Steam (mostly because I love Chess and our board-game engine is really powerful… didn’t think many people would play it) with a “Premium” version (which is practically a donation). …our two-person team tested it, but we didn’t do Early Access or anything…. then we had a couple hundred thousand users in the first couple of days. Not only do this many users find way more bugs than we found, but they also caused issues that we had never seen in our other games (eg: more than 1,500 concurrent games made the list of in-progress-games really laggy). We understandably got negative reviews, but even though we were able to get almost every reported bug solved through several updates within the week, very few of the negative reviewers updated their reviews.

When Steam released their “recent reviews” feature, this gave us a second-chance since it was a strong signal to new users, that the game was worth their time even though the Launch was rough.
It took many months to get the amount of eyeballs on the game to outweigh the initial bad reviews. Some users update their reviews which helped a lot, but many more didn’t want to spend the time looking at a free game again.

And then there are those developers who seem to see this as yet another potentially confusing and hazardous aspect of an already baffling marketplace. Roger Valldeperas, developer of Flat Heroes, a game our Graham praised just yesterday, is still learning to navigate Steam as a dev and seeing the layout change feels like a setback:

…overnight we lost over 2/3 of our reviews (it’s still not much in real numbers because we don’t have many reviews yet, but way worse because that same reason, we didn’t have many and we have even less now), and that left the game in a worse position than it already was, because when you release a new game and are not a known dev you rely almost entirely on press, streamers and giveaways, since pretty much nobody will buy a game that has not appeared on the press and has no reviews on Steam.

From these 3, press and streamers receive way too many e-mails to cover half of the game (among many other sites we sent RPS a press release with a key) so many devs are left with no other choice than trying to give some keys to random strangers hoping they’ll like the game and will tell their friends, and maybe add a review (although I’m completely against asking for reviews in exchange of keys).

That said, even if in our specific case it was kind of unfair because of the timing (we had worse release conditions than the others before us), I do think it is a good measure overall, something had to be done with all the keys-for-reviews market, which was getting worryingly huge (like many devs, we received lots of e-mails asking for keys in exchange of good reviews).

Don’t panic?

To those worried developers, Failbetter founder-gone-freelance-writer Alexis Kennedy has a brief message. Valve, he reckons, have been here before, and they’ll steady the ship if it springs a leak.

My thoughts: DON’T PANIC. Valve’s modus operandi is to make a swingeing change with no warning, watch the explosions, and then calmly recalibrate. You can call this heartless, you can call it far-sighted, but it’s what they do. It’s horrible that honest developers with strong communities have suffered from this. But making good games and having a strong community is the best kind of future-proofing there is, if you can ride out this one.

The important thing, as Danny Day has pointed out, is to give them feedback so they can know what to change. Valve needs us too; even they have competitors; and they think about the long term.

Eugene Hopkinson of VoxelStorm, creators of AdvertCity, doesn’t call the changes “heartless”, as in Kennedy’s quote; he sees them as “an aggressive market control move”:

…any reviews that came from bundle sales have of course been removed from the count as well – and if a backer pays $2 for a bundle of a dozen games late in their life cycle, they’re far likely to judge the games less harshly than if they paid the full Steam $20 price early in the same game’s release history – an effect that’s compounded by the fact that early purchasers were more likely to encounter bugs that were ironed out in later iterations.

The fundamental problem, from our perspective, is that the system is rewarding the (proportionally more negative) opinions of early purchasers and disregarding the (predominantly more positive) perspectives of those who have obtained their key through another kind of interaction than a full price Steam purchase; for a game like AdvertCity with a very steep learning curve, the Kickstarter backers (for instance) are far more likely to play the game for longer, and our data has shown that with this game, play time is directly correlated to whether we receive a positive review from them or not. The net effect has been that this game’s overall rating on Steam has dropped from an unreserved “positive” down to a “mixed” overnight. It seems as though this is really an indirect way for Steam to punish bundles, third party stores and so forth, as well as dissuading the developers who use them – an aggressive market control move.

Bundles and benefits

Some developers have seen very different results regarding Bundle sales though, saying that eliminating reviews from external sources has actually boosted their Steam rating. Here’s Michael Molinari, developer of Choice Chamber:

I’m happy about Steam’s review change, as it’s brought my reviews back to Very Positive, from Mixed. I had less than 10k units purchased through Steam, but being in the summer Humble Bundle sent total units to over 110k, bringing with it lots of negative reviewers who likely didn’t even want the game to begin with.

Peter Willington of Auroch Digital tells me the same is true of some of Auroch’s titles, though also says that “this change basically nerfs the voices of the strongest advocates of projects borne out of crowdfunding”:

…the change has actually been beneficial for us so far: we’ve seen the scores of some of our titles slightly rise.

We think this is something to do with those games being somewhat niche. The people who buy our games directly from Steam tend to be passionate and vocal and really appreciate that we’re making the kinds of games they like. But we’ve also had these games appear in bundles, which puts them in front of players who might not be the target audience, and they subsequently mark the game down in a review for not being the kind of mainstream game they expected. Those bundle reviews now don’t count towards the rating, pushing our score higher.

Simon Roth, developer of Maia, has created a Patreon, partly as a result of these changes, which he reacted too with a degree of despair. He told me how he thinks Steam reviews could be improved as a whole:

…the issue with Steam reviews is it’s built in a way that doesn’t really benefit anyone. The system needs to focus on giving detailed information about a game to players and demand that from reviewers. Reviews should address aspects of the game and guide the reviewer with some structure to maximize its worth. A system like this would remove all the “funny” reviews, the low effort ones and also reduce the issues of scammers as it would no longer be a viable business model for them. A level of Valve managed light community moderation on top of that would filter out the rest of the chaff. I know Valve have very little interest in putting any manpower into such things, but not everything can be automated away.

Users should be given the choice to directly message the developers before writing a review. So many reviews are just tech support problems in disguise. Using the user’s survey data to present what machine they played on would also allow purchasers to know if a performance or tech issue might affect them.

I’ll leave the final word to Blahuta, whose closing paragraph to me captured what a lot of indies – particularly those without established names or communities – were saying to me:

I’m an indie developer who’s been working on a passion project for a little over 2 years and I’d really like to start selling an alpha build soon. If I go with Steam Early Access, I’m taking a risk that I’ll drag any early negative reviews with me all the way to full release which will hurt my sales in the most critical period. If I do my alpha release outside of Steam (Itch.io, the game’s website, etc.), I’ll still get the benefits of alpha while only having Steam reviews from the final version, when they are most likely to be positive. However, with Steam’s latest changes to user reviews, any champions of my game won’t be able post visible user reviews on Steam launch day because they’ll be using activated keys they were given as part of the sale outside of Steam. So, what do I do?

From this site

119 Comments

  1. Cinek says:

    I for one rarely pay attention to steam reviews. Got a phase when I was making all my steam purchase decision based on them, but then I started to read what people actually write, and… holly hell, it’s like a mine of stupidity.

    Currently steam reviews are basically an indication of how fanboyish or retarded community around the game is. That is a precious information, especially for multiplayer games, but they got very little to deal with the quality and value of the game itself.

    • Blackcompany says:

      Completely agree with this. In an age where we can literally watch a game being played for free…I dont understand how anyone is making purchasing decisions based solely – or even mostly – on Steam Reviews…

      • Press X to Gary Busey says:

        Same problem with GOG, but with a taller forehead.
        It’s at best a way to judge the technical state or if something is missing from the version they’re selling.

        The top reviews of older games are always Oh memories. I give my childhood 10/10

        Or (the most overused titles there) the “Diamond in the rough” reviews not even having played the version sold on GOG and reviewing it from when they played it on release 15 years ago in some time vacuum state rather than useful advice of what to expect for someone in the year 20XX.

    • dangermouse76 says:

      Second or third most helpful review for skyrim……. Is a chocolate chip cookie recipe.

      • cautet says:

        Why are you withholding this recipe? Is it really good?

        • dangermouse76 says:

          My bad sorry! I won’t withhold cookies recipes again.

          Ingredients
          1/2 cup (1 stick) unsalted butter
          3/4 cup packed dark brown sugar
          3/4 cup sugar
          2 large eggs
          1 teaspoon pure vanilla extract
          1 (12-ounce) bag semisweet chocolate chips, or chunks
          2 1/4 cups all-purpose flour
          3/4 teaspoon baking soda
          1 teaspoon fine salt

          Directions
          1. Evenly position 2 racks in the middle of the oven and preheat to 375 degrees F. (on convection setting if you have it.) Line 2 baking sheets with parchment paper or silicone sheets. (If you only have 1 baking sheet, let it cool completely between batches.)

          2. Put the butter in a microwave safe bowl, cover and microwave on medium power until melted. (Alternatively melt in a small saucepan.) Cool slightly. Whisk the sugars, eggs, butter and vanilla in a large bowl until smooth.

          3. Whisk the flour, baking soda and salt in another bowl. Stir the dry ingredients into the wet ingredients with a wooden spoon; take care not to over mix. Stir in the chocolate chips or chunks.

          4. Scoop heaping tablespoons of the dough onto the prepared pans. Wet hands slightly and roll the dough into balls. Space the cookies about 2-inches apart on the pans. Bake, until golden, but still soft in the center, 12 to 16 minutes, depending on how chewy or crunchy you like your cookies. Transfer hot cookies with a spatula to a rack to cool. Serve.

          5. Store cookies in a tightly sealed container for up to 5 days.

    • syndrome says:

      I don’t know about this… As much as I agree with you, I’m completely sure noone likes to see their game’s net review summary written in fucking RED.

      • syndrome says:

        In fact, if there’s a company that experiments with using slight color variations and similar subconscious minutiae to condition people into suggestive behavior and to cue them psychologically, that’s Valve.

        I don’t know about the rest of you, but to me it seems completely unfair to have this arbitrary review system color-coded. Red and even yellow summaries of ‘Mixed’, ‘Negative’, and ‘Mostly negative’ speak a lot, even when they don’t.

        I mean, THIS IS WHY THIS WORKS IN TRAFFIC.
        I don’t like the direction Valve is heading to, for quite some time.

    • Yglorba says:

      I find that as long as it has a decent number of reviews, whether it’s blue or red is useful to know. Anything beyond that is pretty silly, though.

      Steam reviews also tend to take price into account much more than professional reviews, which is a mixed blessing. On one hand, yeah, it does matter to some extent; on the other hand, this results in some free or $1 RPGmaker games having higher ratings than Skyrim and silliness like that.

    • Pizzzahut says:

      I feel sorry for developers, most of the time. When you read Steam reviews, the negative ones, a lot of them reference a game’s buggyness or unreliability. But then, when you read the same or similar comments in the game’s forum, it’s amazing how many of these problems are fixed by the individual simply updating their video drivers or other such similar software.

      It’s sad that there’s so many idiots out there, who run DX versions from three years ago, who never even question that it might be their fault. That running multiple AV solutions at once, that run registry cleaners and third party defrag solutions might actually be causing their game(s) to crash.

  2. Collieuk says:

    One thing with early adopters reviews being ignored with kick-starter keys etc.. more often than not the reviews are for early versions. Many of these games remain in early access for months if not years. The finished product can vary greatly to the first early versions which are played by and large by fans of the developer or genre. But to the average punter discovering the game later in its development (or after full release) these reviews are largely irrelevant. I’ve noticed a lot of games that have excellent ratings now have mixed ratings for the most recent reviews. Shows what the non fans think. A bit of filtering is probably for the best although Valve’s method seems a bit heavy handed.

  3. bandertroll says:

    I think there should be more separators reviews. For example, in many English-speaking games reviews the problem may be more in demand to add this or that language.

    • Yglorba says:

      I think it might be useful to have a drop-down where reviewers pick one word to describe the game, or maybe a list of checkboxes they can pick from. Something like:

      Fun
      Innovative
      Addictive
      Interesting Characters
      Interesting Story
      Buggy
      Boring
      Incomplete
      Overpriced
      Too Short

      …sort of like tags, but for subjective or negative things. That could then be broken down into eg. “here’s what people say is the best thing about the game; here’s what they say is the worst thing.” It’d give a more useful at-a-glance assessment than we have now.

      • Premium User Badge

        Angstsmurf says:

        I mostly agree, but those “interesting” really should be replaced with “good”. “Interesting” is one of those words that are more often used ironically than not.

  4. Hedgeclipper says:

    Its been an odd debate to watch, lots of shouting and hand-waving about how developer’s scores are going up or down and what that means for them shifting units but not much really addressing things from the customer’s side. The VoxelStorm quote for example commenting that someone buying a $2 bundle might judge a game less harshly than someone paying full price on steam seeming more or less oblivious to the customer’s interests.

    Other thought – people comment they don’t pay attention to the reviews, my suspicion is that its like advertising, everyone thinks it doesn’t work on them but it often does. Personally I usually glance through some of the more verbose ones as they’ll often give pointers to specific niggles or issues (and sometime even hot to solve them).

    • Someoldguy says:

      I do pay a little attention, but I always check Metacritic and pick up from there any professional review sites that I trust, other sites if my favourites haven’t reviewed, then compare to the aggregate user review to see if there’s a big disparity and mine it to find out why, if so. Only then do I glance at Steam user reviews to see if there are any steam specific issues, and try to filter out the people making a fuss over minor stuff or just making stupid quips like “Killed my brother with a rusty hacksaw. 10/10 would kill my brother again.”

  5. hammr25 says:

    People should be able to write reviews for Early Access games but they shouldn’t have a posted review score because they’re not finished games. When people see a game while browsing or searching they should see a logo showing the game is an Early access title instead of a thumbs up, thumbs down or tilde.

    • Microrocksima says:

      I agree with this.

      In fact, I’d go further to differentiate them from fully released games.
      It still seems bonkers to me that pre-release/beta game builds can be sold for cash on a store alongside fully released games, and yet maintain their independence as pre-release builds with all the caveats :S

  6. mtomto says:

    If developers make a great game, then what’s to worry about? You will get good reviews! If you make a crowdfunded game that your backers love, that doesnt mean that the average Steam user will love it. Early positive ratings from backers renders the rating system useless for the average user. I am totally fine with the change.

    • Someoldguy says:

      The problem is often that you don’t get the reviews you deserve. Offend some people by adding, cutting or modifying one element of your game and you can get a sea of reviews that focus only on their love/hate of that one change and has little bearing on how good or bad the overall product is now.

    • Jeroen D Stout says:

      Steam reviews don’t reflect whether something is ‘good’ they reflect whether it is popular. It is perfectly possible for a really good game to get ‘mixed’ reviews because the wrong people played and rated it. That in itself is the problem with aggregate scores; they mean very little unless you are a person who is perfectly average.

      • mtomto says:

        I think the review score that settles in after about 3 months is usually correct in my opinion, but by then it’s a bit late isn’t it? I have bought a lot of poor games early that had the “Positive” stamp – the games are usually early access, kickstarter, greenlight etc… So my layman/average opinion; the early complementary reviews are skewing the rating system at release. Now I follow some youtubers that does let’s play – find someone with similar interests and you are golden. Magazine-reviews, website-reviews and shop-reviews (Steam) are dead for new games. They are either heavily biased or only fed specific controlled information.

        • Jeroen D Stout says:

          But the score cannot be ‘correct’, methinks, you are just one person and there are many tastes and ideas on what makes a good game. At best you can ‘agree’ with the score.

          I mean, you bought games you did not like because of the score, but perhaps you are an outlier at that stage of the game? Perhaps you are not really the person for it. The score may be correct for the sort of person who likes the game. Then more people flood in, who perhaps simply should not play the game and get on with their lives, who leave negative scores; so the average person puts the score down, making it more useful to you, but less useful to the original people playing it.

          There’s to me no real feeling this score means much, it’s always wrong to someone at some point; and even then there is no indication to whom the score is correct at what point. If Steam told me “the average person, of which there are many and their desire to play this manner of game is minimal, did not like this much” at least that would be a fair indication.

        • Yglorba says:

          One problem is that more obscure games often never get enough reviews. I buy a lot of bundles, and a decent part of my Steam list has just a tiny handful of reviews (often all positive, but it’s just two or three people, so it doesn’t tell me anything.)

          I think there’s a difference between game-makers giving away keys in a quid-pro-quo trade for good reviews, and game-makers giving away keys in the desperate hope that people will say something about their game.

          (Specifically: You need 500 reviews to hit Overwhelmingly Positive, which is a big deal. If I am confident that my game is that good, giving away keys in hopes of getting reviews makes sense for me, and provided I’m not actually requesting a review in exchange I think it’s fine – it’s no different than the practice of sending keys to reviewers, which has always been how it worked.)

      • BlueTemplar says:

        How do you differentiate that “popular” from the “popular” in how many owners the game has (these kinds of statistics is provided by SteamSpy an the like)?
        Then a game can have a lot of reviews or not – I guess that would be not only depending on sales, but also how much love/hate the game can generate, which also depends on how close the marketing and pricing was to the game’s final experience : see No Man’s Sky for a recent example.
        Professional critics aren’t that great either : one cannot trust Metacritics’ critic score because most of the professional reviews will only be written in the first days/weeks after release – and the game can have changed a lot since then (not even getting into the other issues like paid critics…)

        • Jeroen D Stout says:

          I meant more popular in the sense of ‘the large average’ and its judgement. Its ‘popular vote’ to me is not a useful metric because I am not frequently aligned with the average person’s opinions on Steam.

          That goes for any aggregate, including one of professional reviewers. Most reviewers are different people to me. The best thing to do is find people you can trust in their opinion and seek them out on various platforms.

          Aggregates are only meaningful if you happen to be the sort of person who is essentially an aggregate of the population.

          • FreshHands says:

            I don’t know. As kind of a die-hard individualist myself I see your point, of course.

            However, the “average” person wouldn’t even consider buying, say “Avernum: Escape From the Pit” – hence only 500 people or so rated it, final verdict “very positive”. No Man’s Hypo Sky was bought by everyone, got 67500 reviews, “mostly negative”.

            Somehow that seems ok to me. I mean, statistics can be f***ing garbage, but there is a reason everyone relies on them.

            Doesn’t really address the “first week” problem though – hence Valve’s attempt to fix it (maybe).

          • Jeroen D Stout says:

            Statistics have merit, but this is an average of a giant group, which is just not that great… is it fair, in any case, that No Man’s Sky got that low a score? That just reflects a lot of extra events, like the hype. Is No Man’s Sky’s score of real use to me?

            I am not really saying statistics are wrong, I am just thinking it is a bit archaic to have one big figure to reflect a massively diverse audience. A more tailored recommendation system would make it _far_ easier for people to have an impression of whether they will like a game, which would in turn benefit niche games.

  7. genoforprez says:

    I am actually sort of annoyed that so many developers are crying about this. The developers are not actually concerned about the reviews, which valve has actually NOT taken away. They are concerned about the SCORE, which is something completely different and, I think, pretty telling. What that essentially boils down to is that they are complaining that they are upset about how these changes affect their ability to use that score as marketing. But scores are NOT MARKETING. They are not FOR the developer. They are for me, the consumer. The fact that this is being so widely discussed as a developer marketing problem is frankly kind of insulting to me.

    I also disagree with some their suggestions that this could all be improved if, for example, reviews were weighted based on Steam Level or # of games owned. I don’t understand these suggestions, because that doesn’t actually solve the fundamental problem of people giving good reviews to games for which they were handed a free key. Users can do that regardless of their steam level or # of games owned.

    The only suggestion I have heard that sounds like a real solution is for Valve to work out a way to verify where these keys have come from so they can verify whether or not that review should count. For example, if there is a certain kind of identifier on keys that come from Humble. The only problem with that is there would be no way to go back and apply that to all keys handed out in the past. It could only be implemented going forward.

    Also, it should be noted, that by doing this Steam is trying to improve the quality of its service to its customers, which it is allowed to do. I know that Steam is the Wal-Mart and not the locally-owned business in this equation, which automatically makes them the devil or whatever, but Steam does get to care about its own service. It should do what it can to help developers selling on Steam, for sure, but that doesn’t mean it has to bend over for them either, and that anything it does to protect itself and the quality of its service is some kind of act of hostility against the little guy.

    Anyway, I love indie games, and I wish the best for indies and want them to succeed, but as a consumer I am completely fine with this. It can probably be fine-tuned and improved, but generally speaking, this is fine.

    Consumer protection > marketing

    Period.

    • The First Door says:

      Actually, I think the issue is more subtle than that. I don’t believe that developers are worrying about the score because of marketing, I think it is more likely because it alters the algorithms behind what Steam chooses to show people, and so alters how many people sees their game and how many people therefore are likely to buy it.
      Beyond that, I’m dubious this is really entirely about consumer protection at all. If it was about consumer protection there are other ways you could go about it, you could crack down on the frankly comparatively tiny number of games who they thought were breaking the system. I’d personally agree that making anyone who buys the game away from Steam a second class citizen in terms of reviewing is much more about encouraging users to buy in Steam sales rather than from GoG or Humble.

      • genoforprez says:

        Actually, I think the issue is more subtle than that. I don’t believe that developers are worrying about the score because of marketing, I think it is more likely because it alters the algorithms behind what Steam chooses to show people, and so alters how many people sees their game and how many people therefore are likely to buy it.
        Beyond that, I’m dubious this is really entirely about consumer protection at all. If it was about consumer protection there are other ways you could go about it, you could crack down on the frankly comparatively tiny number of games who they thought were breaking the system. I’d personally agree that making anyone who buys the game away from Steam a second class citizen in terms of reviewing is much more about encouraging users to buy in Steam sales rather than from GoG or Humble.

        I’m frankly sick of hearing the “it’s a Valve scheme to squash competition” conspiracy theory as well. That suggests that Steam is trying to entice players to purchase only on Steam instead of elsewhere with the oh-so-tantalizing bait of being able to contribute 0.000001% to the metascore. That’s not super attractive bait. (And once again, let’s note that it’s only the scoring of key activations that Valve has removed. The REVIEWS are still there.) Most players won’t give a damn about that. They just want to buy good games for the best price possible. It also doesn’t make sense as a strategy to get developers to only sell on Steam because it doesn’t create any actual benefit for them doing that. This is just big-company-mistrusting conspiracy theory nonsense, if you ask me.

        I’m not really sure I understand the concern that this affects the algorithm for how Valve shows people games. There are a lot of different ways Valve shows people games. Different lists based on genres defined by the developer, based on tags commonly applied by users, based on recentness of release, based on recentness of updates, based on # of concurrent players, based on # of copies sold, etc. If anything, Valve is guilty of giving too much weight to games that are already popular (e.g. in # of concurrent players) and selling tons of copies, making already-successfully games exponentially more successful instead of shifting focus to other games that could use some spotlight, but I think that is a completely separate issue (although a valid one).

        I agree there are other things they could do in consumer protection terms. They have ALREADY SAID said they have banned those accounts they know for a fact were gaming the system (and will presumably continue to police it). So consider that done. But the problem is that the entire reason the system can be gamed is because the method by which users are receiving these keys is ambiguous and it is not easy to tell whether that key was purchased from Humble or whether a developer handed it to a person on twitter in exchange for a review. There are some cases where it’s pretty blatantly obvious (e.g. the same review copy/pasted multiple times). But in other cases it is not so cut and dry.

        It can and probably will be improved, but generally speaking it is fine.

        • davec1 says:

          Who knows why Valve do what they do. But you’re completely missing the point of the “it’s to squash competition” theory: Of course it’s not to incentivize buyers, because writing reviews is so much fun. It’s to incentivize developers to sell more on Steam and less elsewhere, lest they lose reviews, no matter whether good or bad!

          An example:
          I want to put my game on Early Access, the feedback will make it a better game. Now if I put it there too soon, I risk people not being satisfied with the early versions and giving it bad reviews that will stick forever. Now, if I do a pre-Early Access first, instead, on itch.io’s Refinery, I can build a community and ensure the game is up to par when it hits Early Access. Only now the longer I wait, the more I risk not getting enough reviews, because my audience already bought it on itch.io so their opinions don’t count.

          This is a new conundrum that benefits nobody, a consequence of Valve’s decision to simply discount all non-Steam purchases.

          As far as for how this affects the algorithm that shows people games on Steam, I’m afraid you probably don’t see this as such an issue because as a consumer it’s very easy to tune out the “back shelves” and tell yourself it’s probably all garbage anyway. And it’s in fact truly not a big issue for consumers. Miss out on a buried great game, and there’ll be plenty of other great games.

          But for people who have their livelihood riding on a fair discovery algorithm on what is the de-facto monopoly online store, it matters a big deal if heavy-handed measures cause a lot of collateral damage. There must be more surgical solutions to make the system less game-able. Every honest dev is all for that, it’s in our interest, too.

        • pepperfez says:

          The idea that this is anti-competitive isn’t that consumers are being encouraged to buy only from Steam, but that developers are being encouraged to sell only through Steam. I think such an effect definitely is an outcome of the policy shift, even if not the main or even an intended one.

      • davec1 says:

        People will complain about what affects them. Devs have their livelihoods riding on discovery and the Steam user rating plays a huge role in that (not the reviews so much, those are fine as is). So I’d say it’s legitimate for them to complain. You may find that you don’t care, as a consumer, fair enough…except, it does have the potential to make it harder for interesting games with small/no marketing budgets to be discovered. Consumers miss out. If you care about a diversity of games, you also have an interest as a consumer that small devs still have a chance to pay for their ramen.

        Yes, consumers have a lot less riding on this, but that doesn’t make the changes good. Especially if the statistical evidence presented by Valve makes fake reviews see a nuisance rather than a rampant issue. Developer and consumer interests are aligned here, not opposed, apart from the black sheep.

        I want that issue solved as much as any decent person (as a dev even more, actually). I’d just prefer it if there wasn’t so much collateral damage.

    • darkhog says:

      Generally agree, except for this:
      “The only suggestion I have heard that sounds like a real solution is for Valve to work out a way to verify where these keys have come from so they can verify whether or not that review should count. For example, if there is a certain kind of identifier on keys that come from Humble. The only problem with that is there would be no way to go back and apply that to all keys handed out in the past. It could only be implemented going forward.”

      Steam allows some websites to login via steam account. All it would need to do is to add option so users can connect they humble/kickstarter/indiegogo account to Steam and if a game is both in Steam library and in humble library/list of backed products on KS/IGG, the review counts toward the score. Same for other legitimate stores.

      • BlueTemplar says:

        Yeah, for instance Steam’s API was recently used with GOG asking people to allow them to connect to their Steam accounts to check whether they had some specific game, so that they could claim it for free on GOG :
        link to gog.com

    • ScubaMonster says:

      Honestly, I don’t pay any attention to the score. If something says it’s overall rated Negative, I go to read why. For instance, all the cry baby fits over Killing Floor 2’s cosmetic transactions introduced during Early Access. Was it misguided and time wasted that could have been spent on finishing actual features? Sure. Is it going to affect my game play experience because people can now spend money to unlock weapon skins and outfits? Not at all. The game was exactly the same game before it got review bombed.

      Even if it says Overwhelmingly Positive, I still go and look specifically at the negative reviews to see if it’s just a bunch of gushing fanboys covering up legitimate problems. Looking at the score alone and basing your decision is bad. Honestly, they can remove the scoring entirely and it wouldn’t affect a single thing about how I read Steam reviews. I think they should dump the scoring altogether. You can still see that sea of red thumbs down (or blue thumbs up) and get the idea. Expecting people to actually read reviews to see if the complaints are legitimate is a good thing.

      • moridin84 says:

        You are quite clearly paying attention to the Steam score.

  8. Capt. Bumchum McMerryweather says:

    so basically it’s fucked if you do and fucked if you don’t. I’m glad the only game devloping I do is on Software Inc (which is superb by the way, if you missed the article on RPS about it a couple of days ago, the TLDR is BUY IT NAO).

  9. Captain Joyless says:

    ITA: a misogynist ‘gater whines about the failure of his supposed “next gen” RTS.

    • Distec says:

      Who the fuck are you even talking about, you embarrassing human being.

    • pepperfez says:

      “Brad Wardell, noxious wanker,” really ought to be standard in every publication’s style guide.

  10. Captain Joyless says:

    Interesting that Wardell thinks it’s good. Ashes of the Singularity gets much, much better reviews from key activation than it does from Steam purchases.

  11. GrumpyCatFace says:

    I’m very much on the fence with this. I do take the ratings semi-seriously, and they’re the very first impression that I have of any game. I’ll always look at the negative reviews, to get a good idea of problems with the product as well. On one hand, I don’t want ‘compromised’ reviews from free users, but I certainly do want to see a wide variety of perspectives on a game.

    Removing the free users could be the best thing for the integrity of the reviews. It’s also a direct incentive for developers to stop giving away free games, which will probably lead to more honest journalism/reviews in the gaming press (also good).

    Unfortunately, as the last paragraph pointed out, this may well hurt the entire Early Access program, as I doubt that most people will ever revisit a negative review.

    This will be interesting to watch…

    My personal solution would be to wipe all reviews at the end of Early Access, and save that total rating as a 3rd “Early Access” rating, above the Recent and Overall scores.

    • Hedgeclipper says:

      But as genoforprez pointed out above they’re not even removing the reviews and you can still get them they just don’t help or hurt the score any more.

      That aside I think early access is a half thought through mess in general but I’m not sure what they should do about it.

  12. MeatMan says:

    The developer that you chose for the end of the article makes a statement that is not true.

    “However, with Steam’s latest changes to user reviews, any champions of my game won’t be able post visible user reviews on Steam launch day because they’ll be using activated keys they were given as part of the sale outside of Steam.”

    (emphasis is mine)

    This is false; those users’ reviews will be visible on Steam and readable by everyone. They just won’t count toward the overall score.

    • Premium User Badge

      basilisk says:

      They are only visible if you manually make them visible. They are hidden by default.

      • PancakeWizard says:

        There are really clear radio toggles. It’s not like you have to go delving into account settings.

        • Premium User Badge

          basilisk says:

          I’m not disputing that, just clarifying that what MeatMan said wasn’t *quite* true, either.

          But still the question remains how many people will click on the toggle to see those reviews, and I’m guessing it will be a minority. Because that’s how default settings usually work. But I obviously don’t have the data, so it’s really just a personal guess.

          • Jalan says:

            Undoubtedly what Valve and (likely) some developers/publishers are counting on (that is – people just not bothering with adjusting the defaults).

            Speaking for myself, I’d much rather have the option to just set something globally as far as the Steam only/key redemption factor is concerned and then still have the option to filter between positives and negatives at will.

  13. Mezmorki says:

    Here’s some suggestions I think would be good:

    #1 – No ratings for early access games.

    You can leave a comment, but as mentioned above, the game simply gets an “EA” for an overall rating. I think this would also help temper people’s perception of EA games maybe more complete than they are – and also encourage developers to use EA more for its intended purpose of furthering development, rather than pushing out a “product” early.

    #2 – Keys sourced externally PRIOR to the full release of the game (e.g. leaving Early Access) can still count towards the review score, but would need to be updated by users (since Early Access reviews don’t have a rating, per #1).

    This would allow developers using kickstarter / EA / etc. to have a pathway for their fans and early adopters to update their scores based on the final state of the game.

    Unscrupulous dev’s could still give out keys in advance of release, and then have their minions make a review… to which I say there needs to be more some better enforcement.

    #3 – Change the sequence of posted reviews to show the “most helpful” Positive AND most helpful negative reviews proportional to the overall positive vs. negative ratings.

    So for example, if a game is 66% positive, the first two reviews displayed would be the two most helpful positive reviews. The third review would be the most helpful negative review. And so on.

    This would give a more balanced picture of what people are saying, proportional to the overall scores, and limit fan-voting or hate-voting up the most helpful status of reviews (and hence flooding the written reviews will all positive or all negative as is often the case).

    #4 – Refunded reviews are flagged as “refunded” and the text remains in place but the positive/negative rating is removed.

    • davec1 says:

      Some very good proposals there, sir! It’s good to hear constructive ideas.

    • GrumpyCatFace says:

      +1, this is excellent.

      I don’t agree that #4 should remove the Negative rating from the overall score for refunded games. The person obviously had a very negative experience, and that should be reflected.

      Otherwise, I love this post 10/10. Would read again.

      • Mezmorki says:

        What you could do, is list the % of games that are refunded.

        The thing is – people might refund a game because they played it less than 2 hours and thought it sucked for whatever reason. But you also have people refunding games because their hardware can’t provide a satisfactory experience. Is that the fault of the game, or the fault of the buyer?

        • BlueTemplar says:

          Well, hopefully, it’s already more likely that someone will leave a negative review before refunding if :
          – he had technical issues while his PC is better than recommended specifications
          – and/or game’s tech support was really unhelpful

      • Mezmorki says:

        Another suggestion I forgot to mention:

        #5: Add options to sort reviews by play-time or min/max ranges of playtime.

        I wouldn’t the rating or default views to be tied to this, because it adds all sorts of other complications, but being able to see what a user with an “average/median” amount of playtime says versus someone with 1000+ hours can be useful.

    • freedomispopular says:

      A thought that I had: It would be great if developers could place a limit how many Early Access copies to sell. Say they just want 10 people available for feedback, give them that option, and when they’re ready, they can open it up to more people.

      • davec1 says:

        Itch.io Refinery allows devs to do this and scale up as development progresses (limit to 10 people for instance, then 100, then 1000).

    • darkhog says:

      I think “score” should be left entirely, as mentioned above, from all games. So people who to that point looked at it can see for themselves what a cesspool Steam reviews are (seriously, in most cases they’re actually worse than YT comments!).

  14. davec1 says:

    Indie dev here, looking at going Early Access. The last quote pretty much sums it up for me.

    I think many people don’t understand the issue is not the written reviews on the store page. Once people are on the store page it’s all good, they can read, filter, compare recent to all-time etc.

    The issue is the rating, which you can even sort by when you’re browsing steam. This determines whether a lot of people will even check out your game’s steam page. You have your name, your Logo, your price and all-time ratings for users to decided to check it out. That’s it. And it’s also the impact of having few reviews, something that does not matter for big indies like Larian or Mode7.

    As a small indie it’s in my interest that people can’t cheat with bought reviews, but it’s also in my interest not end up as collateral damage of something that seems more like a push from Steam to incentivize not selling through other channels than a precise solution to a rampant problem (considering their statistical analysis).

    • BlueTemplar says:

      You actually can’t directly sort by Steam Score, but by it’s “rank”.
      I’ve compiled a table looking at the games :
      link to docs.google.com
      The important information is that a game won’t be shown in the first 2% of games
      (aka “Overwhelmingly Positive”, normally 95-100% Steam Score)
      unless it has at least 500 (now) Steam reviews.

      The same for games that don’t have at least 50 (now) Steam reviews : they won’t be shown in the first 27%
      (“Very Positive”, normally 94%-80%) of games.

      I’ve seen a game (Rogue System) drop from page 14 to page 143 (out of 400)
      (while only dropping from 98% Steam Score to 94%)
      because it had more than 50 reviews (Steam + key), but less than 50 Steam reviews.
      (Weirdly, it only happened in the last 36 hours, after the announcement, looks like it takes some time for the database to correct itself?)

    • BlueTemplar says:

      I’m willing to be that it matters a LOT to small developers to be shown after the top 2% rather than after the top 27%, and therefore they will be heavily pushing for their games to be bought on Steam rather than trough a key reseller (like Humble Bundle or Kickstarter).

    • BlueTemplar says:

      And I forgot, SteamSpy compiled a table of the most affected games :
      link to docs.google.com
      (not including “rank”)

  15. 2late2die says:

    I want to respond to this because in my opinion it highlights the main dilemma for developers, but also points out to a straightforward solution.

    However, with Steam’s latest changes to user reviews, any champions of my game won’t be able post visible user reviews on Steam launch day because they’ll be using activated keys they were given as part of the sale outside of Steam. So, what do I do?

    Here’s the thing though, the “champions”, or the backers, or whatever other early adopters you get through your own (outside of Steam) sales, they’ll still be able to post visible reviews, it’s just that those reviews won’t contribute to the overall score. But, supposedly, by that point, using theirs and others input you have a polished, quality product, that will get mostly positive reviews from the folks buying it off Steam, so the score should be fine.
    Not to mention that the reviews from your biggest supporter will most likely be more detailed and so more people will find them useful and it will bring it to the top of the list when actually going through reviews.

    • anHorse says:

      “they’ll still be able to post visible reviews”

      No the reviews will be hidden with a disclaimer that they’re probably dodgy
      Then users can choose to unhide them.

  16. Danarchist says:

    I think another change they should add is limiting the amount of time you can spend in “Alpha” state. With a recent perma-alpha game releasing a paid expansion, before the core game has technically released, shows that some developers are hiding behind the Alpha label to cover up their shortcomings. This would also keep devs from releasing their games to the public before they are mostly functional out of desperation.

  17. H-street says:

    i doubt it will make any difference.. that just means those that want to buy reviews will still just buy and send the steam via gift .. if someone wants to inflate their reviews, paying 5000$ to give away for 100 guaranteed positive reviews at the beginning of the games release isn’t much for those that want to manipulate their review score at the start

    • moridin84 says:

      I doubt many companies would be as willing spend thousands for a better review score.

  18. Premium User Badge

    Ericusson says:

    More transparency = good.
    There, simple.

    • Nerdy Suit says:

      It literally is that simple. I can’t believe people are actually finding a way to complain about this.

    • pepperfez says:

      Please to be posting your full name, address, national ID number, bank account numbers and balances, and mother’s maiden name.

      • Premium User Badge

        Ericusson says:

        Oh yeah, because that has anything to do with what we are talking about : video game reviews.

        These informations would sure be pertinent. You should go into American politics you would be right at home in the grassroots crap.

  19. Nerdy Suit says:

    Valve is so clearly in the right that I can’t believe this is even a discussion.
    1) The system was abused in the former way.
    2) The consumer can very easily change the settings on the game’s page. This is not hidden, it’s very much out in the open, and super easy to change.
    3) This greatly helps the consumer make a more informed purchase decision by filtering the reviews according to the desire of the consumer.
    4) Maybe the devs complaining about this awesome feature should worry more about making good games that will get good reviews from everyone instead of worrying about which reviews will and won’t be filtered BY THE CHOICE OF THE CONSUMER.

    Sorry, but I can’t help but feel like the people complaining about this awesome feature are the kind of people that will complain about anything because it’s the internet and that’s what the internet does.

    • skeletortoise says:

      1) Indeed, but how often, by whom, and to what degree are fairly uncertain. No doubt something should have been done, but just because it was bad before doesn’t mean adjusting for that couldn’t possibly make it worse.
      2) But as someone said above, a significantly lower score will likely dissuade tons of people from ever even looking at the reviews. Speaking from my own experience, times I’ve been pushed over the edge into a purchase by a steam review: Maybe once. Times I’ve been put off from really considering a game from the review score: Probably 15-20 times.
      3) Letting people people know about/filter the circumstances of a review is good, agreed. Lumping every key review into one implicitly negative category, regardless of whether it’s a sketchy key or not, is less so.
      4) Expect in this case EVERYONE doesn’t actually matter as much, just the steam purchasers. No game is going to be beloved by everyone and inevitably certain people won’t enjoy games that are nearly unanimously considered excellent. That’s fine, let them review it that way and explain their problem with it. But the opposite end of the spectrum that feels this game was tailor made for them and changed their life should have an equal platform to do the same. This new system has a pretty good chance of giving a disproportionate voice to the naysayers over the true believers in many cases. That won’t give players a good understanding of the game.

      I almost feel like you have some kind of villainous Scrooge McDuck perception of developers like they’re always trying to pull one over on consumers. There are people out there trying to just make a living doing this, and I think it’d be a bummer if just one truly talented or visionary person never got off the ground because their steam reviews got wrecked by some change to the system.

      And yes, arguing on the internet may be silly and pointless, but being silly and pointless doesn’t make an argument wrong.

    • Chaoslord AJ says:

      Plus: the last thing Valve wants is hurt sales, I think that’s obvious.

    • davec1 says:

      I can’t help but feel that you can’t help but feel that way because you don’t fully understand the issue. The new review filtering options on the store page are fine and a welcome improvement.

      It’s about the ranking, the aggregated score. It’s what shows up next to the discount and prize in the list view. You can even sort games by it (which I frequently do when I browse through sales). Now, on top of that, please understand that it’s not just the quality of a game that count, but also the quantity. If you don’t get enough reviews, you can’t have a high rank. Now take into account that with their shotgun approach, Valve have also eliminated plenty of perfectly legitimate reviews from affecting that score ( kickstarter, itch.io refinery, humble, dev website sales ). You can have a 100% rating, if you don’t have enough reviews, you won’t be on the first couple of pages. Nobody will ever see your game’s page and read the reviews that are there.

      Does that sound like a good solution for you? Why should you care, as a consumer? Because dev and consumer interests are not opposed, except for the few scammers. I mean, it’s hard to argue that there’s not enough games, but if good stuff gets buried because they did a kickstarter or dared to sell their game somewhere other than Steam, a lot of people will miss out on good games.

    • moridin84 says:

      Well it’s basically just developers that are complaining. And for good reasons since it could reduce their review score and potentially reduce sales.

  20. SuicideKing says:

    What troubles me is that there’s still no way to see all time helpful reviews anymore, it’s only the last month of reviews that turn up at the top.

    • Titler says:

      Interesting, I just went to check this and there is indeed a new filter if you scroll down the Store page. You can get at the Most Helpful (all time) by setting the filters as follows;

      Review Type: All
      Purchase Type: All
      Display Type: Most Helpful

      “Summary” now seems to be “Past 30 Days”, and “Most Helpful” filters the entire review set.

      Do the above on Shroud of the Avatar for instance, on this page:

      link to store.steampowered.com

      You’ll see my review there, which was a Kickstarter account given a free key for Steam. Which is weird as it should disqualify me according to the new rules; If you search for just Steam Purchases, I’m under there instead, and not Key Activations which technically it was. I didn’t buy the game on Steam, I bought it on Kickstarter.

      The older choices and UI are still visible, by the way, if you go via the Community Hub for a game either on the Store Page or your Library, and click Reviews there.

  21. Urthman says:

    I’m really curious what evidence there is, if any, that Steam reviews significantly affect sales. Unless Valve is doing some surreptitious A:B testing where they don’t show the review score to some shoppers (and I haven’t heard any evidence that they’ve done that) I don’t know how they could possibly know how much of the sales of a game with Overwhelmingly Positive reviews are because of that label and how much are just because of every other way people could hear that it’s a good game.

    • BlueTemplar says:

      Generally, there’s a correlation indeed:
      link to arstechnica.com

      • Urthman says:

        Of course there’s a correlation. It would be astonishing if there weren’t. But it’s a pretty huge stretch to say that higher review scores cause higher sales, rather than the quality of the games causing both the scores and the sales.

        • frightlever says:

          Discovery is always going to be the biggest problem a new indie developer faces.

          Blade Ballet had a massive background ad on Eurogamer for a few weeks. They had a giveaway right on RPS last week.

          A developer who advertises heavily and sock puppets’ reviews is the real problem.

        • BlueTemplar says:

          Ah, but that seems to be an unfalsifiable proposition…

          • Urthman says:

            Yeah, but it seems to me the burden of proof is on those who want to claim that review score has a significant impact on sales. The fact that it’s very hard to prove/disprove doesn’t make it more likely to be true.

          • Urthman says:

            I suppose you could say that in the absence of evidence it’s prudent for developers to act as if they know the score makes a difference, in case it does. But there’s a lot of things you could be putting that time/energy toward, some of which have a more plausible/testable causal connection with success.

  22. TwwIX says:

    Buy Steamworks title from a certified third party vendor = Get labeled as a second class citizen.

    Is this supposed to encourage me to buy directly from their overpriced store? Angering me. That’s all they’ve accomplished with this incredibly idiotic change.

    As if reprehensible developers and astroturfers can’t manipulate your broken ratings system just by purchasing it directly from your store or by receiving a gift copy and by abusing your refund system. Naaah! That’s inconceivable!

    How about instead of punishing your user base, stop going into business with asshole studios and disallow repeat offenders from generating any further keys for any of their games.

    Let me guess… that would be too much work and hurt your bottom line in the end? Eh, Valve?

    I thought so.

  23. Chaoslord AJ says:

    Don’t think this changes much: 90% of reviews positive will still be 90% if you remove half the votes if everything’s legit.
    Buying based on reviews is very complex. For example a “very positive” with about 30 reviews doesn’t mean anything.
    Great games always emerge on top however even in times before the internet and their reviews like when I bought System Shock or Deus Ex or Daggerfall from the bin out of the blue.
    Whether Indie or AAA, the great games are played by thousands and already take up several of your lifetimes to play.
    The great game of your dream that vanished because nobody noticed it does not exist. You will not be the one to discover one first either.
    Larian must not fear loses as their next game is as good as bought by thousands, keys or not.
    Maia has 545/570 steam-bought reviews which add up to “meh”, don’t see what’s the difference in the new system to the dev.

    • BlueTemplar says:

      There are cutoffs at 500 and 50 reviews for Steam’s “rank”, see my comment above.

    • frightlever says:

      You’re missing the point of what Simon Roth said. He was hoping that his Kickstarters would sweep in like the cavalry and bolster his Steam review score, which, as you note, is decidedly mixed.

      From his Twitter: “That’s me probably going out of business then. My crowdfunders and direct sales were how I hoped to improve my score”

      I think he should make a better game and it’ll improve his score. Also, people are questioning the pace of development, but that is ALWAYS a gripe.

      Reviews and Early Access present a particular problem of their own.

  24. Premium User Badge

    Nauallis says:

    I’m amused that nobody else has directly acknowledged that the developers that are most worried about this change hurting them, are essentially insulting their potential customers’ intelligence by implying that potential customers aren’t smart enough to actually read the reviews themselves. Apologies for the run-on sentence. Add to the fact that you can toggle the code-activated reviews very easily and very visibly…

    And as Black Company stated above, it’s extremely easy and free to watch somebody either playing or reviewing a game.

    • davec1 says:

      I’m saddened that many commenters don’t quite understand what the issue really is. The developers complaining actually grasp that it’s not about the written reviews, but about the aggregate score and how the number of reviews affects that score. That score in turn affects the very first impression people will have on your game and even if your game is buried on page 43 and nobody will ever see it’s store page and read the actual reviews that somebody did write. Once a potential buyer is on the store page, it’s all good, he can read and filter reviews to their heart’s content.

      Most complaining devs don’t think customers are stupid. Most devs are just customers themselves and know they’ll probably only ever go through the first 5-10 pages of a Steam sale.

      • Premium User Badge

        Nauallis says:

        I don’t agree, but I see where you’re going. It’s still ultimately on the developer/publisher to sell me a game. Yeah, if the consumer reviews are shit, I’m a lot less likely to give the title a second chance. That’s not my problem!

        Granted, reading about new games and gaming news is about as much a hobby for me as actually playing games, and I have the disposable income to frivolously buy new games. I am the target market for these indie developers. Which is to say that if there’s an intriguing article about a title, I’m much more keen on trying the game than whatever the steam aggregate “score” is. Do I feel like this is somehow going to reduce the number of titles that I see on a regular basis? No! So that’s my bias.

        • davec1 says:

          But that’s just another thing that many people don’t know: You can have the best game ever with a 100% positive rating with raving reviews, if you only have 499 of them, that puts you in a category with low visibility. If you have only 49, because your 51 reviewing kickstarter backers are discounted.

          You are indeed part of the target market, but the amount of people who actually follow game news is really quite small. Plus, it’s actually really hard to get featured by the likes of Kotaku, RPS etc., if you don’t have something zeitgeisty or just kind of win the press lottery. I follow games reading multiple sites etc. daily, and yet I miss lots of games that are objectively fantastic.

          As a consumer, you can say: “Not my problem”. Fair enough. But that’s very different to “There is no problem.” The natural difficulty of getting visibility in an overcrowded market, that’s one thing, it comes with the territory. But that doesn’t mean it’s a good thing to further reduce that visibility and throw on the fog machine.

          • davec1 says:

            Didn’t finish a sentence there: if you only have 49 because 1 ks backer reviews don’t count, your game gets buried.

    • Yglorba says:

      I think you underestimate just how many games there are nowadays. Once you’re already considering a game, yes, you can evaluate it in many different ways. But the aggregate score helps attract people’s attention in the first place. If you’re one of the thousands of budget indie titles on Steam, you really need that to break out.

      Nobody is worried about what this will do to Skyrim. They’re worried about tiny indie budget titles that rely on a high placement in rankings and the like to draw those initial clicks. (And, as a player, I’m worried because it makes it harder for me to sift through that giant mountain of indie games – there’s no way I can consider them all in any depth, so I often do end up sorting by aggregate score and looking over the ones near the top.)

      • Premium User Badge

        Nauallis says:

        That underestimation is a fair point – I don’t delve much into the greenlight section of steam, let alone try many games that I haven’t read about first. Nor do I use unity or gog universe.

    • Press X to Gary Busey says:

      It was the same when Steam introduced the automated refunds. A few indie developers seem to have a mental image of customers as a mass of malicious, stupid hobgoblin cattle helplessly bumping around inside the marketplace.

      The only thing under you control when selling in a middleman market like Steam are your own actions. Collectively blaming the consumers on the other end of the system, outside your control, isn’t helping anything and they’ll still be around even if you burn your bridges.

      The cat herding is Steam’s job.

  25. Tinarg says:

    Adam, thank you for this great article. I really appreciate that you got in contact with so many indie developers. It helps to hear multiple perspectives on this interesting topic.

  26. Baines says:

    Devs selling games on Steam are going to be biased by how the change affects their games. If the change drops their rating, then it is bad. If the change raises their rating, then it is either less bad or even good.

    Blahuta’s quote is particularly interesting, as he pretty much says that he wants a way to avoid getting negative reviews while being able to keep positive reviews.

    • Yglorba says:

      That’s not what he’s saying. What he’s saying is that up until now, if you had a game with a weird concept, there’s been a problem where selling your game cheap in a bundle or giving it away for free could result in it going to a bunch of people who never had any interest in the concept to begin with and who will proceed to nuke your review score.

      I think it’s an interesting question, though. One problem game designers face is marketing their game to the right people – selling it to ones who will like it. I feel Spec Ops: The Line had a particular problem with that, since it looked (and was sometimes sold as) a generic military shooter. People who wanted that sort of game were likely to get bored with it; while the people who wanted its sort of story-driven game that looked critically at warfare and video-games were likely to overlook it.

      Of course, there’s nothing inherently wrong with someone who was looking for a generic FPS leaving a bad review on Spec Ops saying “hey, it’s actually all about this boring psychological story crap; if you’re looking for a generic FPS, avoid it!” That’s a useful review in that it warns away people who might’ve misunderstood what the game was about. But it becomes a problem when you’re putting all the reviews in a big pot and averaging them.

      If I’m considering buying a piece of choice-based interactive fiction, as someone who likes that kind of game, having a Mixed review because it’s full of people saying “choiced-based interactive fiction sux!” isn’t very useful to me.

      Basically, your Steam review average isn’t just going to reflect how good your game is, they’re going to reflect “how good were you at getting your game to the right audience?”

      • Baines says:

        But it is what he says, even if it isn’t what he was considering.

        He sees two options for early access, and finds both options flawed. While the real problem is that Steam Early Access attracts buyers that really shouldn’t be buying into Early Access titles, what he ultimately wants is to be able to avoid negative early access reviews while maintaining the positive ones.

    • davec1 says:

      I’m sure some devs have as simplistic a thought process as you assume. I’m pretty sure most of us, however, would rather have a system that weeds out illegitimate reviews without needlessly causing so much collateral damage to perfectly honest folks who were already struggling.

      • Baines says:

        Valve presumably doesn’t want to put that kind of work into the problem.

        And different people have different ideas of what is actually “damage,” which largely seems to come down to what they believe will affect them and their friends negatively.

        AdvertCity saw its Steam purchases giving it a lower score than its bundle purchases, so dropping the bundle purchases lowered its overall rating. The AdvertCity dev went into detail about how the change had a fundamental problem in that it favored (for him) harsher early Steam purchases while ignoring (for him) more forgiving late bundle purchases.

        With Choice Chamber, you see the opposite. Choice Chamber had higher store reviews and lower bundle reviews. The Choice Chamber dev was happy that the change shed those negative bundle reviews.

        Do you believe that the AdvertCity dev would have viewed the change as having a fundamental problem if his game had seen more favorable store reviews and less favorable bundle reviews?

        Of course not every dev’s final opinion is fully determined by the impact that the change had on them. Like the Flat Heroes’ dev that saw their review scores hit pretty hard, but sees the change as an overall positive because something needed to be done.

        • davec1 says:

          Again, sure, some devs don’t see beyond their specific case. And the big ones are probably oblivious to the fact that reaching the required number of reviews is actually a challenge for small developers with a huge impact on their chances to break even.

          Same as with bundles, you will find kickstarter backers who are more negative than the average and those that are more positive. What this really should tell us is that if we’re gonna filter games based on aggregated opinions, these shouldn’t be discounted. The larger the sample size (fradulent ones excluded), the more accurate the result, and that’s what everybody wants, or should want.

          The correct way to evaluate the changes is to judge them by their merits and flaws, not by whether somebody criticizing them might have a particular bias. Now, fully aware that I have a bias in the sense that this can potentially affect me ( some people seem to have a bias in the other direction, it doesn’t affect them directly, so it must not really be a problem ), I’d say that a system that can have a big negative impact on perfectly honest folks, is simply not a good system.

          In a way, this change is very typical of software development. You fix one problem and create another one instead. ;D

  27. racccoon says:

    Yippee steam is separating itself from true PC gamers! GREAT!
    Maybe now Steams own imprisoned reviews will be even more troller based than ever! And finally with this separation of class on the PC, Steam may finally collapse! as it slowly gets more and more depreciation of value & possibly end up totally leaving the PC for good!
    Finally in the end we will get our PC back from the monopoly known as STEAM! which today is now more & more less of one!! Yay! pee off! Silly STEAMIES!

    • Orwells Nightmare says:

      Yeah having almost every game currently on the market one click away and often insanely cheap (DAMN YOU GABEN!!!) is awful… And what are you on about? There are other platforms that provide digital downloads, but it turns out Steam does it much much better than the other guys and that is why it continues to grow.

      Steam isn’t letting people that get given a game for free review it, because companies were giving them away for free in exchange for good reviews. That seems pretty hard to argue with.

  28. BiscuitP1mp says:

    I straight up don’t trust most online reviews, for any product. As someone who is going to study games dev next and am starting to make games right now, this honestly doesn’t phase me too much.

    Completely appreciate that a lot of people do look at scores and take them on face value though. But if you make a decision like that, you are probably going to miss out on a lot of stuff you might like. Best example is the movie industry. Lots of “stinkers” over the years I’ve loved, have been critically panned.

    I do my research, watch videos, read articles, talk to friends, and if I can get my hands on said product to test it first.

    Then I pull the trigger.

    They should just make the reviews require a written critique. Certain amount of characters have to be entered to be taken as a “serious” review. Then have some moderation around it, and I’m sure only people who feel strongly for or against will take time to review a game. Rather then “this is shit”, down vote, and walk away.

  29. Premium User Badge

    keefybabe says:

    Not recommended
    Playtime 2534 hours
    This game sucks now they changed the textures on my favourite character.

  30. hpoonis says:

    Anyone who makes a purchase based on reviews from one mouthpiece deserves trouble. Safe practice in anything is to have a variety of different views. If one were building a house the best practice is to get a number of quotes from different tradesmen.

    I purchase software using similar methods…as well as waiting past release day in case the software is riddled with flaws (which appears to be normal practice by developers these days).

    • hpoonis says:

      and, for what it may be worth, I make a decision after actually reading any review content rather than just looking at the numbers. Statistics are for the needy…and middle management.

  31. Orwells Nightmare says:

    Not one mention of Gamer Gate in the article. Seriously? But to be fair, why would you mention a group that was formed because of corruption in games journalism, particularly with regard to indie developers? Nope, that doesn’t sound relevant at all. Good to see there is another Mr. Smith working here at the Ministry of Truth.

  32. Leprikhan says:

    Obvious solution is just to have another separation on the store page, so “Overall Reviews, Recent Reviews, Steam Purchaser Reviews”. Discrepancies between “recent” and “overall” already draw attention, this would do much the same thing. Maybe even separate out another category for “Early Access Reviews”. Boomdone?

  33. geldonyetich says:

    An idiot filter would be a nice addition to Steam, but unfortunately untenable due to the subjectivity of relative idiocy.

  34. moridin84 says:

    Steam will probably start adding “approved resellers” or something. So Steam keys from Humble Bundles will count. Though I don’t know exactly how Humble Bundles work so perhaps there isn’t even trust there (e.g. they could give away games for €1) for that to work.

  35. April March says:

    The many people who are pointing out that the reviews are still available, just don’t contribute to the user score; and all it takes to see them is clicking a radio button; these people are greatly overestimating the effort most people exert in buying a game, especially people with a lot of spending money. They look if the game has a positive rating, and maybe scroll down to read two or three reviews.

  36. Cik says:

    As long as one spends money on a product or service, they are entitled to share their thoughts. Just as with any opinion, consumer opinions should be balanced justaposed to others opinions and your interest in the title, genre, subjectmatter, etc.

    Steam reviews have their value, but their is no value in a dev team abusing the system.