How we test headphones: The SoundGuys testing methodology

Whether you’re an audiophile or looking for your first set of headphones, finding information you can trust online is critical. With so many conflicting opinions, marketing, and—let’s be frank—lies surrounding personal audio products online, how is anyone supposed to know what’s right or wrong? We believe it takes a dram of experience and a gallon of testing.

SoundGuys is a collection of journalists dedicated to cutting out as much BS as possible while still giving you the information you need to maximize your happiness with your personal audio products. In our quest to demystify the world of music, we regularly post education features—but we also test the headphones that come our way with a standardized process. To make sure nobody misses out on what our reporting means, we want to take a minute to go over each of our tests.

Editor’s note: this article was updated on February 26, 2024, to clarify our description of Multi-Dimensional Audio Quality Scores.

Objective testing

To gather information about how well headphones work, we subject each product to a battery of tests we’ve created for our readers. Objective testing refers to the scientifically verifiable performance of the product using standardized tests in controlled conditions. While it is essential, we here at SoundGuys don’t subscribe to the false dichotomy of the objectivist vs. subjectivist debate in audio reporting.

For our headphone testing, we use a Bruel & Kjaer 5128 head simulator hooked up to a B&K 1704C-102 to power the ear simulators and then a high-quality audio interface to digitize the signal. Our test supercomputer controls the measurements, and we use a program called SoundCheck to record, analyze, and export all the data. This software is used in research and development labs and on some manufacturing lines so that we can test and quantify products repeatably to an advanced degree. It also means that, very occasionally, we might find defective products, so we can replace them before wasting time on a full review.

Frequency response

When anyone mentions “frequency response,” they’re talking about how each tone in your music is emphasized. Most headphones don’t reproduce all sounds with equal sensitivity, and your physiology also comes into play.

We measure each headphone on our Bruel & Kjaer 5128 test fixture, which represents the average listener, then take the data from that test and put it into a nice chart that’s a little easier to read than a spreadsheet with 57,000 data points. Here’s what that looks like:

There is no one true ideal that works for everyone. Every single person on Earth hears slightly differently. Not everyone wants a super analytical sounding headphone, and some people can’t abide by mass-market headphones. It’s just a fact of life. We listen to the headphones and point to songs to contextualize what you’d hear if you bought them.

Virtual Listening Tests

It’s difficult to translate lab results into scores that reflect how people would rate the sound quality of products we test. To produce results from our test setup that are better contextualized for our reviews, we collaborated with HEAD acoustics to add Multi-Dimensional Audio Quality Scores (MDAQS) to our testing suite. In short, these are virtual mean opinion scores as calculated by a very powerful machine learning model based on the input of real listeners. For more on how these scores are calculated, please read our explainer.

Using our anatomically correct acoustic test head, we capture the output of the product being tested as it plays back a specific stimulus file. The captured recording is run through the MDAQS algorithm, and we can see what the virtual panel of hundreds of listeners would rate each metric on average.

Isolation and active noise canceling

Because many people like to take their headphones out in the world, testing and showing how each set of headphones or in-ear blocks sound is essential. Not every noise is blocked out equally, and reading advertisements can’t tell you how well a set of headphones cancels out sound. Using the same scale as the frequency response chart, we can represent how different headphones respond to noise.

To test how well a set of headphones or in-ears keeps sound from reaching your eardrum, we first play a sample of shaped noise over loudspeakers and record with our artificial head. Then, we put the headphones on the head and record again. After subtracting one curve from the other, we can display the data in a way that makes sense.

In the example above, we can see that the Sony WF-1000XM5 do a great job of blocking out incidental noise, reducing noise by up to 15dB in the low end and up to 50dB for sounds above 1kHz. According to our testing, the ANC attenuates over 30dB of low-frequency outside noise (below 1kHz), combined with impressive passive isolation.

Each score (isolation or ANC) will tell you by what percentage the perceived loudness of audible noise around you is reduced. For example, a score of 5/10 equates to a reduction of 50%, or 10 dB. Similarly, a score of 8.75/10 will equate to a mean decrease of 87.5%, or 30dB.

Microphone

In 2021, we switched to a standardized test setup that plays back pre-recorded spoken phrases from a calibrated artificial mouth in our test chamber, either with or without simulated background noises, simulated reverberant spaces, or artificial wind. This means that samples from every product can be directly compared, making it far easier to make meaningful comparisons between products regarding the raw speech quality or the product’s ability to reject noise.

Since most wireless headsets incorporate digital signal processing, including speech processing algorithms, conventional test signals don’t give accurate results when quantifying the microphone behavior. For this reason, we use the ideal recordings to derive the microphone’s frequency response.

We also present our recorded speech samples on the product’s review page and encourage visitors to rate the quality of the samples. The microphone score follows how our readers rate the samples. We feel it’s important for the mic performance to be assessed in the appropriate context — in the case of headphones and headsets, the mics are generally used for hands-free calls and conference calls, not for live musical performances.

Battery life

Some people like Bluetooth headphones, which means having to charge a battery. Using test fixtures with human-like ears, we can tell you how long you can expect your cans to keep making music.

We set the product to play back music peaking at 75dB(SPL), a safe and realistic listening level. A recorded waveform tells us exactly how long the headphones or earbuds last from that point. Often, the result is different than what manufacturers quote, but because our test is standardized: you can compare figures across our reviews.

Amplifier requirements

Wired headphones are increasingly rare nowadays, but sometimes, they simply require too much power to be used on a smartphone. If it happens, we will let you know. By using a little bit of math and measurements, we can tell you if the power output of your smartphone is sufficient. If you don’t see a section discussing power, assume everything is okay, and you shouldn’t run into volume problems.

Subjective testing

Objective testing isn’t enough because so many factors can’t be controlled. Testing every last aspect is impossible given the insane range of head sizes, ear shapes, and noise-induced hearing loss of individuals across the globe. That’s why we take each audio product we review and give it an extended spin in a typical home with ordinary people like you.

Sometimes, we find some faults; other times, we don’t, but we always do our due diligence to determine what each product is like. Occasionally, our experiences will differ from yours, and that’s okay! Let us know in the comments if that happens, and we will periodically update our articles if we’ve missed something important.

Build quality

Not all headphones are made to last, and it’s usually quite apparent from their materials. We try to list as much as we can about the construction of the cans, whether they have removable cables and other concerns about durability.

Comfort

No two heads are identical, but we can figure out when headphones aren’t going to be right for you by actually trying them out. We check for heat buildup, how much force is put on your head, and other considerations. We never review a set of headphones without using them for hours on end.

Value

Sometimes headphones aren’t amazing, but they satisfy a need exceptionally well. Not only do we identify the primary target for every model of headphones we review, but we also weigh its value. That could mean how much you’re getting for how little money you spend, or it offers a unique feature not seen elsewhere.

Je ne sais quois

If there’s a feature or product so different from what we cover, we will do our best to develop and add testing for it. However, that’s not always possible. In the event that happens, we will usually put it into a real-world situation and kick the tires there.

We’re constantly re-evaluating how we do things here at SoundGuys and are always looking to upgrade our equipment, coverage, and education resources. We don’t chase numbers; we pursue the truth. You can read our ethics statement for more.