Product Reviews Are Unavoidably Biased

Jason Snell writing again at Macworld, talking about product reviews:

But the truth is that product reviews have always been personal, biased, and idiosyncratic. Scoring systems are made by people with their own opinions about what aspects of a product are more or less important. Reviewers bring their own usage history to the party. In those days, we cloaked all of that behind a veil of utter impartiality, but of course all our biases were baked in.

Nothing truer has ever been said. Thankfully the proliferation of online reviews, and people’s increased ability to seek them out on their own has help cut down on the number of people asking me “which phone/computer/gadget” should I buy, for the most part. When people would ask me why iOS was better than Android, I would always say the same thing, “it’s better for me, but it might not be for you.”

I can’t see myself switching to Android anytime soon, but not because I think it’s inferior. The truth is that I have a very limited amount of usage with it and couldn’t draw that conclusion. But as someone who exclusively uses Apple computers, and has invested a significant amount of money into both OS X and iOS apps, there is no good reason for me to switch.

Reviews try hard to cover all bases, but they will always focus on certain things and gloss over others. And something that is a deal breaker to one person might be irrelevant to someone else. Take some generic product like a vacuum cleaner. A reviewer points out that this model cleans carpet better than any vacuum he has ever used. But when he tried it on tile or hardwood it literally picked up nothing. At the end of the review he recommends it, because his site’s review format is either recommend or don’t recommend. And for someone who has a almost completely carpeted home, this might be a no brainer. But someone that has a mostly uncarpeted house would never want this vacuum.

Now of course the hope is that someone would read this and get that detail out of the review, but the point is that people’s opinions of this vacuum could greatly differ. And circumstances come into play (obviously in this case how much of a person’s home is carpeted). With gadgets, it’s the same thing. Everyone has different priorities. Some people need all-day battery life because they are constantly on the go. Others prefer more computing power because they are mostly at a desk all day with access to outlets.

The best reviewers do their best to point these kinds of things out, but at the end of the day everyone is still affected by their own personal preferences. The only real way to combat this is to get opinions from multiple sources. But even then it’s still an art, not a science.