There’s a common misconception that in order to fully test a design, you need to talk to a lot of users. When I explain that we’re only researching with seven to 10 users, our business partners usually gasp, “How can you possibly be confident that you’re addressing our 5,000 users if you only talk to seven?!”
I like to fish from time to time, but by no means am I a fisherman. Nevertheless, let’s use fishing as an analogy to understand more around user research, specifically, how many users do we need to test to get an accurate sampling of our audience.
Imagine you’re fishing in a small pond, and you hook your first fish, it’s a sunny. You bait back up, send your line in, this time you catch a largemouth bass. Cool!
Largemouth Bass 1
Bait up again and catch another sunny.
Largemouth Bass 1
You bait up five more times, and you end up with three more sunnys, one largemouth bass and a pickerel.
Largemouth Bass 2
As a fisherman, you may be okay catching 15 more sunnys or a combination of other fish that day – you’re there to relax and enjoy nature. But as a user researcher or product owner, with the return on investment continuing to diminish the longer you fish, you probably want to pack it up for the day soon. That’s because evidence shows that five users is the number of participants to test before you start hearing more repeated findings than new findings.
Back to user research. Jakob Nielsen, aka The Godfather of Usability and former VP of Research at Apple, has done multiple studies to back the assertion that when performing usability testing, once you have hit five users per segment, you begin to hear so much repeat information that the time and cost of performing more research outweighs the unlikely chance that a new finding is discovered. Five!
In order to know how many total users to test, it’s important to understand segmentation. Segments are groups of users of your product who perform different activities while using your product. As an example, consider youth soccer. Most youth soccer organizations these days use an online registration and scheduling software. There are club administrators who set up the soccer programs and registration. There are coaches who schedule practice, games and to communicate with the players and parents. Then there are the parents and players who sign up for the leagues, as well as track schedules. Each segment – administrator, coach, parent – has its own unique way of using the youth soccer software, and each would have their own specific tasks they perform in the software.
Using this example, if I were hired by this software company to make their software better, I would target 15-20 total users to test: five to seven administrators, five to seven coaches, and five to seven parents. Any more than that, and according to Jakob Nielsen, we would be wasting our valuable time and money.
Our job is still not finished! After testing our segments and then iterating on the designs, we go back and test our newly improved designs with another 15-20 total users.
As long as the product is alive, UX Research is never done.