Richard Mitchell discusses why Joystiq isn’t scoring reviews anymore:
A score can’t answer these questions. A score can’t tell you what a critic liked or disliked about a game, or why. It can’t tell you what qualities are most valued by the review’s author. Without the full context of a review to explain it, about the only thing a score is good for is deciding whether you want to take the time to read the review in the first place. There’s nothing wrong with that impulse (goodness knows I’ve done it) but it serves to identify a problem. If a score is meaningless without context – context that can easily be ignored – then there’s no reason to have a score at all.
This is such a great point, and the decision to ditch scores is also a good one. Reviews in general are somewhat broken, mostly because what one person values might not be what someone else does, and even if two people agree on an aspect being important, there are varying degrees.
The conversation about Android vs. iPhone comes up frequently. Which is better? Which should a person buy? But the answer isn’t the same for every one. It will vary from person to person based on what is important. Prior to the iPhone 6/6+ people who wanted a much larger phone went to Android because that was the only option, and having a bigger screen was an important enough feature.
As Mitchell also points out, the numbers mean different things to different people. Reviews in the iOS App Store are perfect at depicting this issue. One-star reviews for silly, petty reasons pepper many of the app pages. These types of people might look at a 3-star review, buy a product and be disappointed because of something minor that was not important to the reviewer.
Rolling up review scores on something like Metacritic, or using a composite average of customers reviews like many sites (ex: Amazon) do helps some, but at the end of the day it still lacks context.
There has long been a gap with review websites that is dying to be filled. A website that is clear about what is important, and what isn’t to the particular reviewer. Individual blogs, like Brooks Review can gain a reputation over time that allows the reader to know what toe expect. But a site like Joystiq with a rotating and ever-changing review team makes it more difficult to figure out what reviewer shares interests with the reader. The Sweet Setup and The Wirecutter come closest to this by picking the best from a particular category and then explaining their rationale behind determining the best. That doesn’t work with something like video games where one is different than the next and having a review on day one is too important.
Joystiq’s new strategy of quickly summarizing thoughts on the game, and awarding a “seal of approval” on the very best games is a subtle step in the right direction, but it’s not revolutionary. And for games that do not receive the seal it still leaves a gap for people trying to determine if the game is any good.
Someone could try to solve this problem, but the answer is probably to complicated at the moment.