Valve still trying to stop Steam user review manipulation

user reviews

Valve are making more changes to Steam reviews in response to a different kind of review manipulation. This time the changes are to how Steam decides which user reviews are most helpful, and to make the user reviews that are displayed on Steam’s homepage representative of its overall review score. This is on top of changes made back in September designed to reduce the impact of organised “review bombing” campaigns.

For some time, users have been able to rate Steam reviews as helpful or unhelpful, ostensibly making it easier for others to find the highest quality reviews. In a sane world, that’s exactly what would happen: the cream would rise to the top. But we’re talking about video games here, so obviously it’s not that simple.

In a perfect world, people would truthfully mark a few reviews that were helpful for deciding to purchase or not purchase the game and we could use that data to directly determine the ten most helpful reviews. Alas, it turns out that not everyone is as helpful as we would like. Instead, we are seeing more and more feedback from players that the helpful reviews shown on store pages aren’t representative of how well people are actually enjoying the game.

According to Valve, 11 million people have used the ‘helpful’ buttons, and most of them rate some reviews while trying to decide whether or not to purchase a game — the way it was designed. But then there are the weird outliers ruining the system for everyone else. Valve spotted some users rating more than 10,000 reviews for a single game, and they tend to rate up negative reviews, while pushing the positive ones down. Like review bombing, it’s often done out of spite.

To stop a small number of users from having an unreasonably large influence on a game’s reviews, Valve is changing the system in two ways.

1. Firstly, our system will use a new method of calculating the helpfulness of each review, taking into account the users that are trying to manipulate the system. One way we’re doing that is by counting the helpful ratings on reviews differently for users that are far outside the norm. Ratings from users that follow normal patterns of rating will continue to be counted the same way that they have, whereas accounts that rate an excessive number of reviews on an individual game will see the weight of each individual rating count for less and less.

2. Secondly, store pages will now show the default helpful positive and negative reviews in a similar proportion to that of the overall review score for the game. For example, if the game is reviewed positively by 80% of reviewers, then the ten reviews shown by default on the store page will be 80% positive, showing eight positive and two negative. This should keep the reviews shown on a game’s page from being so easily manipulated by a few determined players and should more accurately represent the overall sentiment of the people playing the game.

Obviously, with an open system like this, it’s not possible to stop people from abusing it without taking tools away from everyone. Even if skewing ratings in this way was a banning offense, that would just be damage control. Reducing the impact of multiple ratings, then, seems like a reasonable solution, though it remains to be seen how effective it really is. I’m less convinced by the second change. Sure, not every opinion or stance deserves equal attention, but a review is no less valid just because it goes against the grain.

These changes are in beta, but you can start testing them now by turning them on and off ad seeing how they impact user reviews.

52 Comments

  1. Ich Will says:

    The money they are spending on this, I think they could have funded a small curation team for decades, and had a vastly better storefront.

    • Urthman says:

      I see no evidence Valve is spending any money on this. Less than a rounding error.

    • Landiss says:

      Hahaha, what money? You are crazy, that is almost zero cost compared to having dedicated team of people who check and review every game they add to the store.

      • Ich Will says:

        I mean, I see why you think that a volunteer droped the code for this in half a day, but sadly, it’s not the case – Valve pays top rates, while a decade of curation would only cost them roughly 3 million dollars. Theve lost 3 million in lost sales alone, easily from their algorithm based disasters. Meanwhile, the algorithms have been in use for 5 years, so let’s pretend they have only been in development for 5 years. How much do you think the wages alone cost?

        • dog2 says:

          Incredible. When you see the weird insane numbers this guy made up, you’ll have no choice not to want curation.

          • Ich Will says:

            At least my figure wasn’t zero, as the reply I was replying to suggested…..

            As for made up, well no, it was ripped entirely from netflix’s curation team – 11 minimum wage employees, plus three supervisors on $9 an hour plus a manager who earns $50,000 + 20% HR costs.

          • Landiss says:

            That is interesting. Where are the numbers about Netflix coming from? Also, how does Netflix compare to Steam in this aspect? To be honest, I haven’t seen any reviews or curation in Netflix, what exactly is that team doing in there?

        • frosty2oo2 says:

          steam already has curators, subscribing to a few seems to work pretty well, pc gamer just suggested Tooth n Tail, thats a sale.

        • Cederic says:

          They’ve gained $billions in sales due to their algorithms too.

          Meanwhile even assuming your cost of curation is remotely accurate (which it’s not) $3m pays for a lot of design, development, test and decision making time, even at Valve’s internal rates.

          Adjusting this algorithm will have taken one person a couple of weeks analysis (I’m being generous here, more likely a couple of hours) followed by about a week’s worth of time spread across 25 people in 20 meetings, then another week to do the design/dev/test/release.

          So four weeks full time employee equivalent. I think that’s going to be less than $3m.

  2. Erinduck says:

    Yeah, I’ll believe it works when I see it. Valve’s obsession with algorithms over actual, honest to god, by-hand moderation is going to just further drive me away from using the platform as anything but a game library.

    • itsbenderingtime says:

      This is how I feel about the entire tech industry.

    • Rindan says:

      Valve is just being realistic. They literally can’t hand curate the games they have without hiring a massive exponentially growing staff. It doesn’t take a rocket scientist to see the number of games being produced, realize that it is going to quickly eclipse the capacity of reviewers, and further, they realize that they can’t possibly hand curate for people of different tastes. My tastes are not your tastes.

      If you want curated reviews, don’t use Steam. I don’t see what is so hard about this. You are on RPS for a reason. One assumes it is to learn about video games. That’s your hand curation. Don’t like RPS’s style of hand curation, use another website. In fact, Valve has made it so that you can see RPS picks on Steam itself. There is in fact a hand curated list of games to play from RPS sitting on Steam right now. Are you upset that list wasn’t made by a Valve employee? Would that improve the quality of the list?

      Valve’s job is far more messy than what RPS does with its hand curation. They need to figure out how to drink from a fire hydrant. They need to accommodate all tastes in all games, and accommodate, more or less, all games. I’m sorry, but code is going to have to be the answer. Humans just can’t keep up.

      Hell, I remember the day when I used to play basically every game in the genres that I like that was halfway good. I laugh at that idea now even though I more or less have the same amount of free time. You just can’t keep up.

      If you need hand curation, go, uh, use hand curation. Literally no one is stopping you. It’s silly to think that Valve is going to hand an exponentially growing pile of games.

      • LuNatic says:

        If anyone can afford to pay an exponentially growing curating team, it’s the money printing factory that is Valve.

        • Cederic says:

          Yes, but they can afford it because they don’t spend money on things like it.

          I can afford a Maserati, but only because I don’t buy a Maserati just because I can afford one.

  3. aircool says:

    Give the mob power and they will abuse it.

  4. aepervius says:

    They fought agaisnt review bombing, or negative review in general, but what are they doing about the slew of positive review on some game , which are not representative ? DEV have a problem with negative review… And they seem to do that for the dev…. But heck I have had a far more sour taste with positive review immediately after a game opens , often from early access, often 2 liners, which are not representative.

    How about simply filtering out *any* review which is a 2 liners ?

    • mgardner says:

      Line1
      Line2
      Someone will figure out a way around that.

    • King in Winter says:

      That’s because negative reviews will negatively affect Valve’s bottom line, while positive ones do not.

      • Herring says:

        I’m not sure it will. If someone is put off from buying Game X will they just not buy anything or instead buy Game Y? If it’s the latter Valve get their money either way :)

    • Chicago Ted says:

      While your proposed solution is far from useful, useless positive reviews are indeed a problem. Often, they’re quite funny, but I will admit reviews like “15 years of western touhou piracy have finally come to an end”, “Tan Cirno”, and “It’s touhou”, which make up all of the top Touhou 16 reviews even with the new beta enabled, are not actually useful to someone who’s not already a fan of the series.

      • wengart says:

        I’m not entirely sure why folks would use individual reviews to make a choice. Finding a reviewer(s) who you read consistently is easily the best way to make buying decisions if you want a text review. Otherwise a video review or Let’s Play is almost certainly better.

        What Steam reviews do is give you a general grasp of the entire audiences view on a game. Poor, mixed, positive, very positive. These give you a solid ballpark as to the quality of a game. If I know I enjoy TD games and I look at Defense Grid (Overwhelmingly Positive – 2539 reviews) and Legion TD 2 (mixed – 347) I can immediately tell which one is more likely to be better.

        • d3vilsadvocate says:

          This, a thousand times this!

          The average of “Poor, mixed, positive, very positive, overwhelmingly positive” has turned out to be THE most reliable “scoring” system there is. This beats metacritic and everything hands down.

          If a game rates mixed or poor, all I then need to do is fish for usable negative and positive reviews to see what all the fuss is about. Takes some work, but finding solid reviews always does.

          Seriously, that Steam rating into a couple of categories is the best rating thing that has ever happened to gaming!

  5. Dogshevik says:

    “…for users that are far outside the norm.”

    And this, boys and girls, is an excerpt from a press release of a market-leading, professional company working on a world-wide stage…oh boy.

    Also:
    If adjusting the algorithms time and again doesn´t solve your problem, maybe the problem doesn´t lie with the algorithms?

    Moderation and coupling empowerment with accountability are human processes. My guess is you need humans for that. But then, humans usually expect paychecks, don´t they, Gabe?

    • Landiss says:

      Give them few years, the AI is already in the making. It doesn’t require paychecks, it creates one for itself if it wants to.

      • Dogshevik says:

        Even if it creates, markets and sells the games to itself, plays all of them and writes the reviews, that hypothetical AI would still receive its input by the same out of touch idiots that approve such press releases.
        So, no matter the level of automation, the stupidity is here to stay, I am afraid.

  6. Vilos Cohaagen says:

    Valve’s motto these days seems to be “minimum effort.”

    • Megatron says:

      I picture Gabe Newell wearing The Money Suit, the reverse of Crysis’ Nano-armour.

      Gabe: “I need new curation in Steam”. *Flicks Switch on wrist*
      Suit Voice (GlaDos, but like she chain-smokes): “Minimum Effort“.

      • UnConsolable says:

        Aww damn dude, things are engineered to a cost for many reasons – not all of them to everyone’s detriment. But boy howdy am I sucker for the Crysis suit voice or a good ‘Maximum X‘ joke.
        Have a +1 from a ‘hater’ even :P

  7. Darth Gangrel says:

    I think this is a non-issue for people who are able to quickly understand when a review is highly biased and has the means to easily gather other valuable info. It just takes a small bit of effort and good judgment.

    However, that doesn’t mean that it’s okay or not a problem that people abuse user reviews to promote their own opinions/propaganda.

    • Megatron says:

      I think this is a non-issue for people who are able to quickly understand when a review is highly biased and has the means to easily gather other valuable info

      You said that with a straight face in the Era of President Trump and Brexit.

  8. Premium User Badge

    Aerothorn says:

    While I understand the skepticism, I think this is a great move. I had some friends release a great documentary on Steam (Game Loading) that immediately got review bombed by Gamergaters, and even after more people saw it and wrote positive reviews, the “helpful” Gamergate reviews dominated the page and pretty much killed any chance of success they would have had. It was incredibly depressing.

    • Dogshevik says:

      Yet the planned changes don´t change a thing about stampedes like that. It just changes how people must game the system for the result they want. In a twisted way it is probably even entertaining and engaging to find new ways to do so. A sick minimalist puzzle game on its own.

      EDIT: Seen this way, Valve might have just released a major patch…

    • mitrovarr says:

      The annoying thing, moderators would probably be really effective on steam. Many users have huge buy-in so bans/mutes would be really painful. Limit reviews to accounts that have spent over a certain value and every time there’s a review bomb, shadowmute reviews from all of the participants. You’d only have to do it a couple of times to drastically cut down their numbers.

  9. KillahMate says:

    I don’t think at this point Valve are even capable of imagining a way out of this mess that doesn’t involve endless tinkering with a bunch of algorithms. This Point&Clickbait article really summed up my feelings on the matter perfectly:

    link to pointandclickbait.com

  10. SaintAn says:

    This is just them manipulating reviews to be in favor of corporations. It’s disgusting.

  11. Frosty Grin says:

    One issue I have with this is that it suppresses ALL recent changes. A game update pushed a new, predatory microtransaction scheme? Too bad, 90% of the reviews are still positive – so that’s what you will see. A bad game improved a lot? You won’t see it either, ensuring its failure.

    • gnalvl says:

      Quake Champions is a good example of this conundrum. The limited, freemium version is available only through the Bethesda Launcher, and Steam only offers the $30 “full” version, albeit via Early Access, because the game is far from complete.

      By the very nature of this setup, in the immediate post-launch period, the Steam reviews automatically omitted opinions from people who tried the game outside of Steam and felt it was too unpolished to warrant spending money on. Thus, Reddit and Bethesda’s forums were full of complaints about game-breaking bugs in the netcode, hitreg, performance, and so on, giving a mostly negative impression – yet Steam reviews were “mostly positive”.

      In spite of this, even many of the positive reviews on Steam explicitly stated that the game had many technical problems and recommended readers against buying it until the bugs were fixed. If you’d played the game and knew this to be true, the logical conclusion would be that reviews which pointed this out should be rated the most helpful – yet in voting on this you’d arguably be guilty of abusing the system.

  12. SuperTim says:

    Lemme see… I’ve checked a few broken games (i.e. save file breaking, crash prone, loot boxes, can’t start without admin priviliges, and other stuff)

    Well, under this new design, all those good “negative” reviews disappeared.

    That basically means that this new design will cause even more refunds than before, because you’ll have to try them yourself instead of trusting on the reviews.

    Oh well, if Valve wants to deal with more refunds, that’s their problem then. ^_^

  13. Premium User Badge

    Alikari says:

    I may be being simple minded here, but why don’t they just restrict it such that only accounts that actually own the game on Steam can review it? If they’re worried about brigading reviews, just make it so that to leave a review, you have to have the game in your steam library? Who’s going to buy a game to just leave a negative review?

    • MattM says:

      It is restricted that way already. Steam review bombs occur when a company does something that angers a large number of current owners enough for them to take the time to write a review.

      • KidWithKnife says:

        What he said. Also, bear in mind that we live in a world where some gamers will donate thousands of real dollars to crowdfunded games that may never come out, or spend thousands on microtransactions that have little effect on their game experience. If anyone seriously believes that there aren’t people who’ll buy a couple hundred copies of a game in order to review bomb it and feel like a big internet hero, send them my way; I’m sure they’ll be interested in buying a lovely bridge in Brooklyn.

  14. waltC says:

    Dedicated and third-party game forums will inform far more on a game than any sort of user-review system. Positive reviews which say nothing are no more helpful than negative reviews which say nothing are harmful to a game’s sales…;) Generally, it takes more than “This game suxOrs” to discourage anyone…;) I don’t ever take the one and two liners into account–mostly they’re simply annoying whether pro or con.

  15. MultiVaC says:

    So at what point do we all finally just admit that “anonymous internet mob” is just not all that great of a method of providing people with the information to make intelligent decisions?

  16. LuNatic says:

    Steam reviews – what fun.

    I posted a negative steam review of Ark about a month back, saying that the game is fun and has great potential, but is plagued by performance issues, bugs, and half-implemented features. I even wrote that if they fix it, it will be a must buy.

    Since then I have received an angry reply each week from steam users with hidden profiles accusing me of being a trolling lifeless loser living in my Mum’s basement and using a potato PC.

    Ah internet, where would we be without you?

  17. Iseedeadgames says:

    Worthless attempt to show disgusting greedy games as good on the store page no helpful negative comments encouraged so that they can sell better.

  18. Premium User Badge

    MajorLag says:

    Personal opinion: Valve should just ditch public reviews entirely and go back to the “recommend to friends” method it came from. Suddenly, all the problems disappear except the internet complaining, which is a universal constant anyway.

  19. Someoldguy says:

    This does seem to be really unnecessary. I’d just cap anyone at around 20 review upvotes per game and have done with it. Ok maybe at a push also stop reviews shorter than a twitter post from being rated at all.

  20. KidWithKnife says:

    I think the genie’s out of the bottle on this one. The credibility of Steam user reviews is already destroyed. It’s to the point where if I see someone not employed by Valve encouraging others to take them seriously I just assume that person is a review bomber or somehow affiliated with one. Anything that can be a review bombed just cannot be trusted at this point, IMO.

Comment on this story

XHTML: Allowed code: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>