What’s up with Consumer Reports?

I’ve got a few problems already with Consumer Reports claim that Apple’s new HomePod system sounds less good than the Sonos One and Google Home Plus. I thought I’d summarise them here. We’ll see how they compare to the publication’s final findings, once the headlines shrivel up a little.

Problem #1: Track record

“According to the expert audio testers at Consumer Reports, the Apple HomePod does deliver very good audio performance. It’s just not the best-sounding wireless speaker in our ratings—or even the best-sounding smart speaker,” they claim.

They’re entitled to their opinion, of course, but I can’t help filtering their thoughts with memories of other times when they seemed out of kilter with everybody else’s reality.

Such as:

• When the title conceded the Apple Watch “doesn’t drop calls” in such an incredibly curmudgeonly way one just could not help but think they wanted to think different.
• When it used its platform to slate the MacBook Pro just days before Christmas with the results of a test that was so obscure – and refused to delay publication to let Apple examine the results.

Problem #2: Subjectivity

I simply can’t agree with Consumer Reports claims the HomePod’s bass was “a bit boomy and overemphasized”, that midrange tones are “somewhat hazy”, and that musical nuances were lost or “underemphasized”. Certainly not in comparison to the speakers I heard HomePod play against.

I’ve been using a HomePod for a week. I have been consistently delighted by its sound and responsiveness. The vast sound stage, beautiful presence and astonishing balance means I already know that this speaker can replace my old home stereo. I listen to a huge range of music, it works with all of it.

Why should my opinion matter? Some time ago I wrote the majority of original content for several issues of a magazine then called iPod User (it was the UK version of US title, Playlist, and is now known as iPhone User).

This means I’ve reviewed plenty of digital audio kit. (I even looked at the iPod Hi-Fi). That’s when I learned that once you get past the specs and the technology, lots of the final value judgements about audio kit are subjective. It’s hard to define phrases like “better”, “worse”, or “muddy”.

To be fair, CR does say it assessed several tests by several people, but I look forward to seeing how come the methodology used in those tests generated results so much at odds with my own experience.

Problem #3: The comparison

I’ve been able to listen to the HomePod in action in a direct listening comparison test with a Sonos One, Amazon Echo and Harman/Kardon Invoke. Apple’s product clearly delivered a highly superior presence and a much more pleasing sound stage across every sound frequency than either the bass-heavy Harman/Kardon or the weedy sounding Amazon product.

The Sonos One certainly competes, but I felt (anecdotally) that it lacked some of the presence the HomePod provides. I loved that HomePod bounced sounds of the walls behind me to create a real 360-degree perception of sound, and having heard what it sounds like I know that when stereo pairing is introduced it will deliver even better sound.

I just cannot agree with Consumer Report’s claims that HomePod sound is less good than the other devices it tested to any noteworthy extent. I think those differences are both subjective and over-blown, and certainly don’t match my experience. It’s a peer player, at worst.

Problem #4: The importance of privacy

I’m highly concerned that when it comes to smart home devices, Consumer Reports is ignoring the utter importance of privacy and security protection in these connected devices.

Surely the built-in privacy and security protections of connected devices should be a critical consideration when choosing to deploy them in your homes? It’s weird to me that a title that says it cares about consumers seems utterly blind to the importance of privacy and security in people’s lives

Amazon and Google solutions are incredibly weak in this in comparison to those you’ll get from Apple. I believe that Consumer Reports’ failure to consider this matter in its group test makes it complicit in making its readers lives less secure.

Problem #5: When a group test isn’t a group test

It is interesting that despite claiming to be running full tests, Consumer Reports isn’t testing Apple’s most technologically advanced HomePod feature at all. Apple has developed really smart technology that empowers its device with the capacity to adjust audio output in real-time for better results in whatever space it happens to be in.

“Apple says that every time you move the speaker, it senses the motion, then automatically adjusts itself to its placement in the room using a series of test tones and complex algorithms to minimize reflections from nearby walls or other objects. That’s not a feature we evaluated,” they say.

“That’s not a feature we evaluated”? Why not? Surely this should count as some kind of advantage for Apple in term of a “full” group test? Why is this mentioned but not considered? At the very least it’s a feature that should provide higher scores in both music playback quality and ease-of-use?

[CP_POLLS id=”27″]

Problem #6: Objectivity

I also think it’s weird that Consumer Reports will both ignore the importance of privacy and of advanced technology while taking a time to castigate Apple for failing to introduce stereo pairing or multiroom capability in Homepod from launch. Apple has already said it will introduce these features later this year in a free software update.

I see no real objectivity in cherry-picking the features to criticise a company for not including, while avoiding those features that are unique to the company’s products. Is it fair to ask if Consumer Reports created a group test that’s flawed by design? Can anyone spell “MacBook Pro test”?

One more thing

Consumer Reports does appear to be consistent. Here’s a remaining trace of its thoughts on the iPod Hi-Fi, courtesy of my apple menu.

It sounded way better than that!!!