The myth of five

There’s a catchy notion in product development circles: test with just five participants to uncover all your insights. Simple and cost-effective, right?

Well, context is king. For iterative usability tests on a prototype, where you’re fine-tuning as you go, five can work wonders (with the right cadence). But when your methods span diary studies, surveys, or in-depth interviews, that magic number suddenly loses its sparkle. The risk? Skimming the surface of your customer base and missing out on critical, actionable insights.

In business, numbers talk. Stakeholders are quick to raise eyebrows at findings based on such a narrow view. The result? Nil pois (and certainly no commitment)!

So, what the underlying issue?

Sample size

Think of your sample size as the lens through which you view your customer population. The clearer and more comprehensive the view, the better your insights. So, understanding sample sizes and how to adjust them is not just smart—it’s strategic.

(Plus it ensures your insights have the depth and breadth to drive decisions, whilst building trust with stakeholders.)

Key considerations for nailing your sample size:

1. Research objectives: The specific goals of the study guide the depth and breadth of data needed.
2. Population variability: More diverse populations require larger samples to be accurately represented.
3. Volume of customers: The larger the customer base, the more varied the experiences and perspectives (typically).
4. Confidence level: Higher confidence levels increase the sample size needed to ensure the results reflect the population.
5. Margin of error: A smaller margin of error necessitates a larger sample size for precision.
6. Expected effect size: Studies looking to detect smaller effects or differences need larger samples.
7. Sampling method: This determines selection, affecting the representativeness and reliability of findings.
8. Resource constraints: Budget, time, and accessibility of participants limit the feasible sample size.

There are also other non-numerical considerations, which are classed as desirable characteristics:

1. Demographics:
   • Inclusions: Age range, country, household income, spoken language, etc.
   • Exclusions: Not in certain job roles, not in certain industries, not participated in the last N months, etc.
2. Business-specific:
   • Inclusions: Has downloaded the app, coming up to renewal, used live chat in the N days, etc.
   • Exclusions: Only communicates via the contact centre, doesn’t use self-service capabilities, just renewed, etc.

The pitfalls of getting It wrong:

1. Too small: You risk the outliers painting an unrepresentative picture, losing face, and stakeholder commitment.
2. Too large: Complexity skyrockets, and so do costs, without proportional gains in insight.

While we can’t give you a one-size-fits-all number (beware anyone who tries), we can share a tool that we love for making these calculations easy and accurate:

Sample Size Calculator