Deciding which supplements to buy has always been a tricky endeavour. The FDA has no real guidance for the quality of supplements on the market, so consumers have essentially had to rely on brand trust and word of mouth alone. With the rise in popularity of online reviews over the last dozen years though, consumers are feeling more informed and confident in their decisions. But can relying on online reviews lead us to the wrong choice?
Labdoor offers consumers an alternative to consumer reviews – actual laboratory tests. Their analysis verifies label claims, checks ingredient sources, and screens for controversial artificial sweeteners, preservatives, and coloring agents. They publish their results on their website and assign each product they test a rating, similar to customer review sites like Amazon and Bodybuilding.com.
I decided to stack up Labdoor’s protein grades against customer reviews from Amazon and Bodybuilding.com. My assumption was that there would be some correlation between the two. Obviously it wouldn’t be a straight line, but I figured that the products which did better in the lab tests would translate to better customer reviews. I started by copying the list of protein products from Labdoor’s Protein Rankings onto a spreadsheet. Then one by one, I found those products on Amazon and Bodybuilding.com and recorded their rating.
The first problem I encountered was that none of the ratings used the same scale, so I had to do some translating. I decided to translate everything to a 100 point scale, since this made the math easiest and Bodybuilding.com was already using it. (0.0/10 to 10.0/10).
Translating Amazon ratings into the 100 point scale was easy, simply multiply by two. Done.
Translating the Labdoor grades took a little more thought. Since they are rated A through F, I figured I’d use the academic grading scale that is popular in the US. This assumption didn’t matter too much since we are just looking for a correlation anyway. Here’s how I translated the scores:
Now that I had all the data on the same scale, I wanted to go through and check to make sure that the means and standard deviations were similar, which they were:
So as you can see, we’ve got a pretty neat looking dataset. Now here’s where it gets interesting. Using the Pearson product-moment correlation coefficient, we get a correlation for Labdoor ratings vs. Bodybuilding.com review ratings as -0.01, and for Labdoor ratings vs. Amazon review ratings as -0.172. (A perfect correlation would result in a Pearson’s value of 1 or -1, whereas values closer to 0 would mean less of a correlation exists.)
So as you can see in the chart above, there is practically no correlation between Labdoor ratings and consumer reviews – both on Amazon and Bodybuilding.com. I was quite surprised by this; I had assumed there would be some visible pattern, but I was completely wrong.
The data paints a clear picture: humans don’t analyze a product the same way a lab does.
Users are only able to base their reviews off of criteria that they can sense or perceive: flavor, mixability, smell, value, label claims, and digestion. The lab is testing for a lot more than is detectable by our human senses.
While humans are certainly better equipped to comment on flavor than a lab, only a lab test can identify the presence of controversial artificial sweeteners or if the sugar and carb content is in fact what the label states. Labdoor doesn’t claim to base their rankings on flavor – reading reviews is probably your best bet if flavor is important to you.
However, even reviewers can’t seem to get their stories straight when it comes to deciding on which flavors they like. Here are two reviews for the same product, in the same flavor on Amazon:
Obviously, different people have different palates, digestive issues, and standards for their supplements. But the shortcomings in reviews go deeper than that.
It’s a lot easier to convince a user to like your product than to convince a lab.
When I met with Neil to talk about Labdoor, I asked him how companies can rank higher. His answer: “Make better products.” When I asked him what he tells all the companies that complain about their ranking, he said “I tell them to make better products.” Go and ask an online marketing consultant how to get better reviews online, and they’ll have 101 answers: “Offer an incentive to post reviews, do a giveaway, contest, social media, instagram, discount codes, better branding, test team, send free samples out”.
Reviewers are more likely to buy into the hype than a laboratory is. Online reviews are easy to influence if you know what you’re doing – even without actually improving your product at all. We found some examples of reviewers that are clearly buying into the hype and don’t appear to be capable of an unbiased, factual analysis of the products at hand.
BSN Syntha-6 earned a D+ grade and ranked #54 of 56, yet they still managed to pull off a 4.6/5 star rating on Amazon. How did they do it? Marketing and branding.
Here’s an example of a user who gives the product 5 stars, and their only comment is “GREAT BRAND”:
Obviously BSN’s marketing team is working. They are able to convince the masses that their product is superior, which translates to higher reviews on Amazon.
Now let’s take a look at one of the highest rated proteins over on Bodybuilding.com: Pro Jym Protein. This is a product from a brand that has been formulated by the famous Dr. Jim Stoppani and available exclusively on Bodybuilding.com. Despite earning a B- grade on Labdoor and ranking #50 of 56 in the protein list, they still earned a staggering 9.5/10 on Bodybuilding.com. How did they do it? Again, building brand loyalty.
Here’s an example of a user who gives the product a perfect 10/10 rating and justifies it by confessing his unwavering trust for guy who started the brand:
The shortcomings of reviews are quite obvious. Consumer feedback isn’t a controlled, scientific process by any means. Big brands’ marketing dollars often translate into better reviews as consumers prove themselves to be quite impressionable.
Lastly, let’s compare consumer reviews on different sites.
To further add to the confusion, I went ahead and compared the ratings on Amazon.com with Bodybuilding.com. Using the Pearson product-moment correlation coefficient, we get a correlation for Amazon ratings vs. Bodybuilding.com ratings as 0.07.
Again, we’re looking at almost no correlation, but this time it’s between the same products on two different review sites. It appears that there is so much variation between the reviews on Bodybuilding.com and Amazon that the ratings are almost irrelevant at this point.
The fundamental problem with reviews on mainstream review sites like these is the complete and utter lack of control. It is an extremely common practice for brands to offer incentives for reviews (such as free products, giveaways, samples, or contest entries). While this may seem reasonable, the result is a “selection bias”, meaning that a product’s ranking can be bumped higher by consumers who have been primed to rate the product well because they were incentivized by those companies in the first place. Comparing products’ ratings is no longer an impartial evaluation of the products themselves, but instead a competition to see who has the biggest army of followers. This is now literally the definition of a popularity contest.
Incentivized reviews are permitted on both Amazon and Bodybuilding.com, massively skewing their average product ratings. The best way to incorporate customer reviews into your product research is to actually read the text of the reviews and not just rely on the rating number.
On the other hand, laboratory testing of products doesn’t have the same vulnerability for brands to exploit. While it doesn’t address every pre-purchase concern (flavor, mixability, etc), Labdoor offers an additional resource consumers can use in conjunction with other research to help them with that pesky chore of deciding which supplement to buy.
About the Author: Tommy Noonan is a guest author for Labdoor Magazine and the owner of SupplementReviews.com, an unbiased review site dedicated to bringing honesty into an industry known for scams and out of control marketing hype.