Five Reasons You Can't Trust Reviews

People like you come to sites like these to read reviews for many reasons. Perhaps you want to decide which show to watch or which book to read. Perhaps you want to better understand a show that you have already watched, and bond with others who have had similar experiences.

But how much can you trust the reviews you read? The truth is that reviews are frequently biased, even if the reviewer wants to be honest. This piece looks at five ways that biases enter reviews – not to stop you from visiting this site, but to help you be aware of some of the things that influence reviewers.

Bias #1: Some reviewers want to make money.

This is the biggest reason for bias in reviews, and it generally means that reviews will be somewhat more positive than they would be if done neutrally. And it is understandable: any author/TV series/filmmaker wants good reviews, because those reviews can make or break sales.

Here’s an example: several years ago, some filters on the Amazon Canadian website broke down and could see the real names of reviewers instead of their pseudonyms. It turned out that lots of authors were reviewing their own books (and of course reviewing them positively). After this embarrassment, which damaged its reputation as a neutral vendor, Amazon put in some filters, making it impossible (or at least far more difficult) for this procedure to continue. Amazon also banned reviews from close relatives and associates.

But Amazon also wants to make money, and they make money by selling video content and books. They also own IMDb and GoodReads, the former site hosting many reviews from viewers of shows and linking to reviews from critics, and the latter devoted to reviews of books. And of course there are things that they can do to manipulate the reviewing process. I recall when a long-awaited sequel to a series was coming out, a sequel that was deemed more boring than watching paint dry, and many days passed before reader reviews appeared. I suspect this was because Amazon delayed the posting of these reviews, as the overwhelmingly negative reactions of actual readers could be expected to depress sales.

Professional reviewers are paid for their reviews. Their reputations are better when their reviews are considered both insightful and neutral. However, it is possible that they are nevertheless influenced in what they are allowed to say. Sometimes the pressure is explicit, when a boss wants a favor. You can tell this has happened when the professional critics praise something, and then, when ordinary folk get to react, they all agree that the movie or book really sucks. Sometimes the pressure is less explicit, when the boss chooses what can and cannot be reviewed.

Bias #2: Some reviewers only review stuff they like.

A site like Doux Reviews is staffed by volunteers, who only review shows they are watching. As time is precious, they tend only to watch shows that they like. So you’ll find a positive bias to these reviews, and this explains why the reviews may skew toward the higher ratings.

I think the upwards bias is fine, and it implies, too, that all the stuff that we’re not watching and hence, not reviewing, would receive lower ratings. Nevertheless, it can be considered a bias.

Bias #3: Some reviewers are nasty.

It is far easier to criticize than to do. Also, some people simply love creating a cutting phrase. Or perhaps they are suffering from an inferiority complex.

I have encountered several reviewers who fall into this category. You can tell by checking their other reviews and getting a sense of how they generally react. If they tend to give everything one star, then the bias is probably a part of their personalities. What unhappy malcontents they must be!

Bias #4. Some reviewers have a genre bias.

In 1991, when Beauty and the Beast came out – an animated version of the story – I praised it while at lunch with some of my colleagues. Immediately one woman proclaimed, “I don’t watch cartoons,” in an unpleasant, dismissive tone. She would probably belong to those with the third type of bias (there’s no reason a reviewer can’t have more than one bias), but the fourth type of bias belongs to those who have a bias either for or against a particular genre.

Some people love romantic comedies; those same films make others yawn. Some appreciate a good action film; others shy away from gore. Now, in general, most reviewers only watch shows in a genre they enjoy – but others, especially those who are assigned shows, may have to review shows in categories that they do not like. Hence, reviewers may bring a positive or a negative bias as they form their opinions.

On a related note, some people who review current shows may not have knowledge of the genre. I’ve been reviewing Supergirl, but I never read comic books, so I don’t bring the same background to my reviews as someone who does. I’m sure I miss many in-jokes.

In a similar vein, some reviewers have certain preferences. Most people are visually oriented, and will be swept away by some special effects. I am an audio, so my rating for a show with bad music will suffer, even when the storytelling is excellent.

Bias #5. Some reviewers have a political agenda.

In the United States one sees this often. As soon as a book is published by someone politically polarizing, bots or those with genuine opinions will swarm on to Amazon or wherever with ratings of one star or five stars – often before it is possible for the vast majority to have read the books (although some reviewers do receive advance copies or attend advance screenings).

I was startled, however, by some of the reviews of Supergirl at the IMDb database. I usually visit the site in order to look for photos and to check orthography. When I was preparing my review for one episode, I discovered a very early user review. The review was completely negative, slamming the episode for its anti-gun slant. I then watched the show, and discovered that it was not, in my opinion, that anti-gun, as it did present the other side. In fact, the episode was not that much about guns at all. So I believe the review was politically motivated, and very likely someone was paid to write and to post it.

So, what should you do?

You can still get a lot of insight from reviews, but here are some suggestions:

• First, take all reviews with a few grains, or even a whole box, of salt. If a review strikes you as biased, it probably is.
• Later reviews may be more reliable than earlier reviews.
• Remember reviewers (unless they’re bots!) are human and make honest mistakes (the bots' mistakes are dishonest).
• It’s good to give positive feedback to reviewers you respect.

Finally, remember to respect your own opinion, and allow yourself to love or to hate any show as much as you wish.

Victoria Grossack loves math, Greek mythology, Jane Austen and great storytelling in many forms.

6 comments:

Billie Doux said...

What a fun article, Victoria. A lot of excellent points, too.

I'm very proud of Doux Reviews even, or probably because, we do tend to lean toward the positive, and I'm okay with that kind of bias -- I can remember being really turned off by Television Without Pity, although I was addicted to it for a short while. I spend a lot of time reading Amazon reviews when I'm thinking of buying something expensive, especially the negative reviews.

And as a fan of science fiction, I've noticed genre bias a lot, all my life. Science fiction shows and movies often get a rating of at least a star lower than I believe they deserve. Although this particular bias is improving a bit as the genre has gotten more popular.

magritte said...

I think that the positive bias of Doux Reviews is fine, especially with episode reviews. I doubt that people reading about Buffy season 3 episode 6 are really reading to find out whether or not they should watch it. I suspect most people reading episode reviews have already seen the episode and are curious what somebody else thought of it.

Secondly, while nobody can be truly objective, I feel a good reviewer will try to be aware and honest about their own biases. If I am reading a review to determine whether a work is worth my time, I don't really care whether reviewer X liked it, I want to know whether I will like it. And a clear idea where the reviewer is coming from, and the content of the review, are a lot more useful to me in that regard than the number of stars.

Anonymous said...

An interesting article

I just wanted to say why I have returned to this site over and over again for... ugh I think it's been a decade now...

You guys always try to stay positive even when shows go through their rough patches. Other sites can get so negative and it can be really off putting if you still love that show or character. You're also honest when you have a bias and that's fine. We all have bias, but not everyone acknowledges that which is when it can become a problem. The reviews on this site are different because I can tell if I will like a show/book/movie based on your reviews.

Also this site has introduced me to so many shows I wouldn't have found out about otherwise which have since become enduring favorites such as Veronica Mars, Buffy, and Highlander.

Thank you

Billie Doux said...

anonymous, thank you so much for your lovely comment. :)

magritte, I agree that people don't come to Doux Reviews and read episode reviews this granular because they're deciding whether or not they want to watch a show. They're already fans, like we are. I hope we really do show awareness of our biases. I doubt I've ever mentioned this, but as an undergrad, I took a course on existentialism and instead of having us write papers on the content, our prof asked us to write in detail about how Sartre and Camus made us *feel.* That stayed with me, and when I started writing TV reviews in grad school, I included that emotional component.

And that might be a bit more than I meant to write, but I'm going to post the comment anyway. :) Thank you, Professor Nelson.

Omar Playa said...

Not bad. Not bad article at all.

Dragonfire said...

magritte, that's exactly why I read the episode reviews on Doux Reviews. To find out what other people thought about a particular episode, whether or not there was anything I missed.
And I really appreciate the positive slant of Doux Reveiws. My favorite show in the world is Supernatural and there are like two reviewers out there (Billie and Lynn Zubernis from MovieTvTechGeeks) who are both fans of the show all the way up into the later seasons (not just up to s5), and who do actual episode reviews. It's so nice to hear what other fans of the show think about it, not what someone paid to review it who thinks it should've ended eight years ago thinks.
So thanks for reviewing like this, Billie. It's made my entire experience of the show more enjoyable.