The Utter Uselessness of Camera Reviews

Some time ago I inveighed against Photography Blog for their review of the Sigmas SD-1. It was so horrible that the bias of the magazine didn't just reveal itself between the lines; the article screamed it out for the world to see. But this problem is not unique to Photography Blog. It is not even unique to the world of cameras. It is pandemic across all websites that review products.

I read an article written by Frank Bruni, the ex-food critic for the New York Times, where he was talking about his review philosophy. Basically, he never reviewed restaurants that couldn't take it. As he saw it, a reviewer has an immense amount of power, especially one for a major publication like The Times. A bad review can, no exaggeration, be the cause of a restaurant's failure. As we learned from Spider-Man, with that power, comes responsibility.

For my part, especially when dealing with massive companies, this concern does not exist. The obligation of the reviewer does not belong at all to the company and belongs entirely to the consumer. These companies are asking consumers to spend money — a great deal of money at times. A good reviewer rips those products to pieces. That's the fucking job of a critic. That's why critics are different from some schmo on YouTube. They have experience; they have taste; they have the resources to effectively understand a product vis-a-vis the totality of other products on the market. If anything, a critic should be actively trying to ruin a product's reputation.

That said, I don't think that most major online publications are actually concerned about this. Especially in the tech world, very rarely does a single publication wield enough power to do significant damage. Still, their reviews are frequently useless.

For example, Cnet is a completely pointless pile of shit at this point in its life. It's a scam for people who don't know about other, smaller websites. Many years ago, they overhauled their review system from a 0-5 star rating to a 0-10 rating. This new system would not follow the academic system where anything below a 6 is considered poor. Instead, 5 would be perfectly average, 0 would be far below average, and 10 would be far above average.

What happened exposed the bias of major publications in the clarity that only pure numbers can bring. Since 5 was average, you would expect their average review to be, I dunno', 5?

Of course, it wasn't five. It was 7.something.

What's amazing is not the result, but that no one thought that putting a quantifiable measure to their reviews may be a bad idea. It revealed beyond doubt that giving a review score some sort of objective value was a terrible idea. It allows people to actually compare products in a meaningful way. Of course, this is precisely what readers would want, but it is precisely what companies do not want. And this is, as I'm sure you've guessed, the reason these companies are useless cyphers that do little but produce quotes for marketing efforts. And just so you don't think I'm picking on any specific company: every major publication is suffering from the exact same problem.

The question is of course why the hell the publications would do this? The answer is simple: they want to keep getting invited to the party. If a company sends you a product to review and you then rip it to shreds, you won't get any more products sent to you. It injects necessary an unavoidable bias into the review process. That's why Consumer Reports buys all of their review cars on the open market.

A recent example of this was in the admittedly elite world of sports cars. The UK reviewer Chris Harris went onto Jalopnik to rant about the way Ferrari handles review cars. Ferrari's answer was, of course, to ban him from ever driving one of their cars again. He is still banned to this day. (side note: if you have the money, never buy a Ferrari. I don't care what Ferris Bueller said.)

It is for this reason that we must adjust the way that we read reviews. On many websites, if they have a 5-star review system, anything below a 4 is total crap. Of course, they won't say this in the reviews, but that's the hidden meaning. When a website like Photography Blog or DPReview reviews dozens, perhaps hundreds of products every year, and almost every product scores the equivalent of 4-stars or higher, you know that something is fishy.

Many will attack me for being a curmudgeon, and perhaps I am. I'm not hating on companies and their products because I'm chasing some dragon of perfection. There is no dragon and there is no perfection, but there absolutely is a best product at the moment.

An element of product criticism that seems to have been completely abdicated by critics is the quality of a product in relation to other products. It doesn't matter if a new camera or car is a solid camera or car, if it's not as good as other products for the same price, you give that fucking thing one star! It reminds me of the maxim, there are no bad products, only bad prices.

This company is saying "give us your money!" If that company is not giving me as much as another company for the same amount of money, I want to know about it!

But that would be a terrible thing for major companies. Canon has been producing cameras that are no way near competitive with other companies for many years, but they have used their market dominance to effectively shut down criticism of that. Do you think that DPReview would ever rip into a Canon camera? No. They never, ever, ever will. If Canon produced a camera that literally did not work, DPReview would still give it a 65/100.

(Side note: This is only a mild exaggeration. The Pentax Q, a joke in the photography world, scored a 70. While on the subject of DPReview, I've noticed that when cameras are just so awful that they could never give it a good review, DPR simply doesn't review it. They never reviewed the Canon EOS M, and they didn't review the Sigma SD1 until the Merrill version came out along with its much, much, much lower price.)

The major companies work very hard to keep the landscape as it is. They want to maintain control over the publications that write about their products. When a publication gets out of line, they snatch away access to junkets, review products, and freebies.

Obviously, not all websites are like this. Steve Huff is great, as are a few independent developers. But they are photographers first, critics second. We need someone who has the money and the drive to be both a photographer and a critic. Someone who recognizes that their obligation lies with the readers, not with being soft on some multi-billion-dollar company's shitty products.