A researcher compares one programme with another on one predeclared outcome and obtains a small p-value. Another researcher examines twenty outcomes, five subgroups, four time points and several model variants, then reports the most exciting result as though it had been the only question asked.
The two p-values may look identical. Their evidential contexts are not.
Multiple testing is the problem created when many statistical claims are examined within a family of opportunities for false positives. Sequential analysis is the related problem of looking at accumulating evidence over time and allowing those looks to influence when data collection stops or which action is taken. Both require the analyst to account for the search process, not only the final number that survived it.
This article explains why multiplicity appears, how familywise error and false discovery rate differ, when Bonferroni, Holm and Benjamini–Hochberg answer different questions, why repeated interim looks require planning, and why error control never substitutes for effect size, design quality or scientific judgement.
Reading route: begin with the twenty-test example, distinguish families of claims, compare FWER with FDR, then move to repeated looks, optional stopping and the reporting standard.
Twenty tests create more chances to be surprised
Suppose twenty independent null hypotheses are each tested at a 5% Type I error rate. If every null is true, the probability of avoiding a false rejection on all twenty is 0.95²⁰, about 0.358. The probability of at least one false rejection is therefore about 0.642.
This 64% figure depends on the simplifying assumption of independent tests. Real outcomes are often correlated, so the exact probability can be lower or otherwise structured. The example is valuable because it isolates the mechanism: every additional opportunity to declare success can raise the chance that at least one success is produced by noise.
The U.S. Food and Drug Administration’s 2022 guidance on multiple endpoints describes this concern in a regulated clinical-trial context. The general statistical lesson travels beyond medicine: if a conclusion is selected from many candidate claims, the selection process belongs in the interpretation.
One small p-value is not the probability that the hypothesis is true
A p-value is calculated under a specified statistical model. It does not report the probability that the null hypothesis is true, the probability that the result happened “by chance”, or the practical importance of the estimated effect.
The American Statistical Association’s statement on statistical significance and p-values emphasises that decisions should not be based only on whether a p-value crosses a threshold and that full reporting and transparency matter.
Multiplicity is therefore not a reason to worship an adjusted p-value instead of an unadjusted one. It is a reason to make the entire testing plan visible and to connect statistical evidence with effect size, uncertainty, design and substantive consequences.
The hardest question is often: which tests belong to one family?
Correction procedures need a set of hypotheses to operate on. That set is not determined by software alone. It depends on the scientific or decision structure.
Imagine an education study with one primary reading outcome, two secondary language outcomes and ten exploratory process measures. Treating all thirteen identically may be unnecessarily conservative for the prespecified primary claim. Treating every measure as a separate family can make multiplicity disappear by definition.
A defensible family usually groups claims that jointly support the same inferential promise. If any one of ten outcomes would be used to say “the programme works”, those ten opportunities are part of a shared success rule and should not be analysed as though each lived in a separate universe.
Write the decision rule before seeing the results where possible. Preregistration is relevant here; see Preregistration and Registered Reports.
Familywise error rate protects against at least one false rejection
The familywise error rate, or FWER, is the probability of making one or more false rejections within a defined family. Strong FWER control aims to keep this probability below the chosen level under any configuration of true and false null hypotheses, subject to the method’s conditions.
This is an exacting goal. It is useful when even one false positive in the family is costly—for example, when multiple claims could independently authorise a consequential action.
But stronger protection usually reduces power. Error control is a design trade-off, not a free upgrade.
Bonferroni is simple because it spends the error budget conservatively
For m tests and a desired familywise level α, the Bonferroni procedure tests each hypothesis at α/m. With twenty tests and α = 0.05, each threshold becomes 0.0025.
Bonferroni does not require independence to provide its standard union-bound control, which makes it robust and easy to audit. It can also be conservative, especially when tests are numerous or strongly dependent.
The important idea is broader than the formula: if a project creates many ways to claim success, one can allocate a finite error budget across those opportunities rather than pretending every one receives the full 5% independently.
Holm improves on fixed Bonferroni thresholds for many families
Holm’s step-down procedure orders p-values from smallest to largest and compares them with progressively less stringent thresholds. It controls FWER while often rejecting more hypotheses than simple Bonferroni.
The operational lesson is useful even without memorising the algorithm: multiplicity methods differ in what structure they exploit and what error criterion they guarantee. “Adjusted for multiple comparisons” is incomplete unless the method and family are named.
False discovery rate answers a different question
When hundreds or thousands of hypotheses are screened, preventing even one false rejection can be too stringent for the scientific job. The false discovery rate, or FDR, controls the expected proportion of false discoveries among the rejected hypotheses under the method’s assumptions.
Benjamini and Hochberg’s 1995 paper introduced a practical FDR-controlling procedure that became foundational for high-dimensional research.
FWER and FDR should not be treated as two brands of the same guarantee. FWER asks about the chance of any false rejection. FDR asks about the expected false fraction among discoveries. The acceptable error concept depends on the research and decision context.
A worked Benjamini–Hochberg example
Suppose five ordered p-values are 0.001, 0.008, 0.021, 0.040 and 0.200. For an FDR target of 0.05, the Benjamini–Hochberg thresholds are i × 0.05/5: 0.01, 0.02, 0.03, 0.04 and 0.05.
The largest p-value that does not exceed its threshold is 0.040 at rank four. The classical step-up procedure therefore rejects the first four hypotheses.
This worked example assumes the familiar conditions under which the procedure has its stated control properties. Dependence structure matters, and there are alternative FDR procedures for other settings. The arithmetic demonstrates the decision rule; it does not show that four discoveries are substantively important.
Adjusted p-values do not repair an exploratory story written as confirmation
Imagine a study that explores fifty transformations, ten subgroups and several exclusion rules until one attractive effect appears. Applying a correction only to the final visible set of five p-values does not necessarily account for the full search path.
The effective multiplicity can include model selection, outcome switching, data-dependent subgroup creation and repeated inspection. Some of these choices can be represented statistically; others are better handled by transparent separation of confirmatory and exploratory analysis.
Exploration is valuable. The error occurs when discovery is narrated as though it were the original prediction.
Correlated outcomes do not make multiplicity disappear
Reading comprehension, vocabulary and writing quality can be correlated. Testing all three is therefore not equivalent to testing three independent coin flips. But correlation does not mean there is no multiplicity problem.
Some procedures can exploit known dependence or joint distributions more efficiently than Bonferroni. The analysis should use a method justified for the actual structure, not assume that “the outcomes are related” grants three unadjusted chances to make the same broad claim.
Hierarchical testing can encode the logic of the scientific claim
Sometimes hypotheses have a natural order. A project may first test an overall effect and examine subgroups only if the overall claim passes a gate. Or a primary outcome may be tested before secondary outcomes are formally considered.
Gatekeeping and hierarchical strategies can preserve error control while reflecting the intended evidence sequence. The order must be justified and preferably specified in advance. Rearranging the hierarchy after seeing results converts an error-control design into an outcome-dependent selection process.
Composite endpoints and multiple endpoints are different design choices
Combining several outcomes into one composite can reduce the number of formal tests, but it changes the object being measured. A composite is defensible only when its components and interpretation support the intended claim.
A broad “success” outcome that counts any small improvement among many components can become easier to achieve while becoming harder to interpret. Multiplicity should not be solved by hiding unrelated outcomes inside one label.
The FDA’s 2022 multiple-endpoints guidance is useful here because it distinguishes different roles for primary, secondary and composite endpoint structures within a formal inferential plan.
Looking repeatedly at accumulating data creates another testing dimension
Suppose a project plans to collect 300 observations but checks significance after 100, 200 and 300, stopping early if p < 0.05. The final decision rule is not “one test at 5%”. It is a sequence of opportunities to stop for success.
If three looks were independent, testing each at 5% would create a probability of at least one false rejection of 1 − 0.95³, about 14.3%. Real interim statistics from accumulating data are correlated, so 14.3% is not the correct general sequential error rate. The example simply shows why repeated looks cannot be treated as separate one-shot tests.
Group-sequential and alpha-spending methods design boundaries so the overall false-positive behaviour remains controlled while allowing planned interim decisions.
Early stopping can be for benefit, harm, futility or efficiency
A sequential design may stop because evidence for benefit is strong, evidence of harm is unacceptable, the probability of reaching a useful conclusion is too low, or the decision has already become sufficiently clear.
These stopping reasons have different implications. A futility stop is not proof that two options are equivalent. A harm stop does not require waiting for the same evidence threshold as a benefit claim if the governance framework sets asymmetric consequences.
Stopping rules belong to the design and should be connected to the decision responsibility, not added as improvised reactions to a graph.
Adaptive designs pre-specify how learning changes the experiment
An adaptive design can change sample size, allocation, arms or other features according to pre-specified rules based on accumulating data. Adaptation is not the same as changing the analysis whenever results look inconvenient.
The FDA’s 2019 guidance on adaptive clinical trials emphasises prospective planning, simulation where appropriate and preservation of interpretable operating characteristics. Although the guidance is for drug and biologic trials, the general methodological distinction is widely useful: planned adaptation is part of a design; unrecorded improvisation is not.
Optional stopping changes the experiment you actually ran
“We planned to collect until the result looked clear” is a stopping rule even if it was never written down. If a researcher repeatedly checks a conventional p-value and stops at the first crossing, the nominal one-look interpretation no longer matches the data-collection process.
Optional stopping also has a reverse form: continuing beyond the planned sample because a nearly significant result feels promising. The final sample size then depends on the observed evidence.
The repair is not to pretend the intermediate looks never happened. Use a sequential method appropriate to the design, report the actual stopping process, or treat the result as exploratory and seek fresh confirmatory evidence.
Peeking can happen through dashboards and informal conversations
A project can create sequential dependence without a formal interim-analysis meeting. Live dashboards, weekly metrics and repeated team discussions can reveal outcome trends while collection continues.
If people with authority can change recruitment, exclusions, measurement or stopping in response to those trends, the information pathway belongs to the study design. Blinding or role separation can sometimes reduce this risk.
Operational monitoring for safety or data quality may still be necessary. The key is to separate those legitimate monitoring functions from unplanned inferential fishing.
Bayesian sequential analysis has different mathematics, not no discipline
Bayesian methods can update probabilities as data accumulate and can support sequential decision rules. That does not mean any stopping rule is automatically harmless or that prior choices cease to matter.
Decision thresholds, priors, likelihood models and operating characteristics should match the use. The FDA’s January 2026 draft guidance on Bayesian methodology explicitly discusses uses including monitoring and sequential designs in clinical trials. It is a draft regulatory document, not a universal prescription, but it illustrates the continuing need to pre-specify and evaluate Bayesian decision rules.
Repeated confidence intervals can also be misread
A conventional 95% confidence interval calculated after every new observation does not automatically have 95% simultaneous coverage across the entire sequence of looks. Repeatedly asking whether the interval excludes zero is another way of repeatedly testing.
Sequential confidence sequences and adjusted intervals are designed for repeated monitoring under specified conditions. The correct tool depends on whether the project needs one final interval, a small number of planned interim looks or continuous monitoring.
Many subgroups create both multiplicity and low power
Subgroup analysis is tempting because heterogeneous effects matter. But splitting a study into many groups simultaneously creates more opportunities for chance patterns and fewer observations inside each comparison.
A difference between “significant in Group A” and “not significant in Group B” is not itself evidence that the effect differs between groups. The relevant question is an interaction or other direct comparison of effects.
Prespecified subgroup hypotheses with credible mechanisms are easier to interpret than a large retrospective search. Exploratory subgroup findings can still be valuable as hypotheses for future work.
Model specification can create a hidden multiverse of tests
Researchers choose covariates, transformations, outlier rules, missing-data methods and time windows. If many defensible specifications are tried and only the best is reported, the visible p-value understates the search.
Multiverse and specification-curve approaches can expose how conclusions vary across a defined set of defensible analyses. See Sensitivity Analysis and Robustness Checks.
The goal is not to assign a separate multiplicity penalty to every keystroke. It is to make analyst degrees of freedom visible enough that a reader can distinguish a stable conclusion from one selected from a large garden of alternatives.
Exploratory analysis deserves its own language
An exploratory screen may deliberately tolerate more false discoveries to find promising signals. That can be rational when follow-up is cheap and confirmation is planned.
Use discovery language: “These analyses identified candidates for follow-up.” Do not silently promote a screened signal into a definitive causal or policy claim.
The cost of false positives and false negatives should shape the error-control objective. A discovery system and a final certification system need not use the same threshold because they perform different jobs.
Power belongs inside the multiplicity design
More stringent thresholds reduce false positives but can increase false negatives. A study with twenty underpowered outcomes and severe correction may be technically controlled but scientifically uninformative.
Choose a small number of primary questions that matter enough to power properly. Use secondary and exploratory outcomes for distinct purposes rather than promising formal confirmation everywhere.
A good multiplicity plan begins at design, not after the p-value table is generated.
Multiplicity can occur without hypothesis tests
The same selection logic appears when choosing the most dramatic chart, the best-performing benchmark, the most favourable time window or the strongest quote from many analyses. A dashboard with fifty metrics offers fifty opportunities to tell a selective story.
Error control for formal tests does not automatically solve narrative selection. Preserve the full result set or a principled summary and explain how the highlighted result was chosen.
This is one reason Data Visualisation and Preregistration belong near statistical testing in the Library.
A practical design workflow
- State the primary decision or scientific claim.
- List every outcome, subgroup, time point and formal comparison that can support that claim.
- Define the testing family or hierarchy.
- Choose the error criterion: FWER, FDR or another justified target.
- Select a method whose assumptions fit the dependence and design.
- Power the important claims under the adjusted plan.
- Pre-specify interim looks and stopping rules.
- Separate confirmatory and exploratory analyses.
- Record deviations and additional analyses transparently.
- Report effect sizes, intervals and decision consequences, not only adjusted significance.
The reporting contract
Report the full number and identity of planned primary, secondary and exploratory analyses. Name the family or hierarchy and the error-control method. Explain whether correction covered endpoints, doses, time points, subgroups, interim looks or another source of multiplicity.
For sequential work, report when looks occurred, what information was available, who saw it, what stopping boundaries were used and whether the study stopped early. A final p-value without the monitoring history can be difficult to interpret.
Keep unadjusted estimates and uncertainty visible where informative. An adjusted inferential decision should not erase the magnitude and direction of effects.
Report deviations from the planned analysis. A transparent unplanned analysis is more useful than a supposedly prespecified analysis whose history has been rewritten.
A learner’s thought experiment
Give twenty fair coins to a class. For each coin, flip it five times and call any coin with five heads “special”. The chance that one specific fair coin produces five heads is 1/32, but across twenty coins the chance that at least one does so is much larger.
Then let learners choose which coin to keep studying after seeing the first five flips. The exercise reveals selection. The chosen coin’s unusual past does not automatically predict an unusual future.
Next, let them inspect results after every additional flip and decide when to stop. They can see how the stopping rule becomes part of the experiment.
Where this article sits in the eduKate Library
This article owns the general methodological problem of many inferential opportunities and repeated looks. It does not replace domain-specific regulatory, clinical, engineering or examination rules. Specialist owners remain responsible for the consequences and standards of their fields.
Use Statistical Inference and Uncertainty for single-analysis foundations, Experimental Design for allocation and controls, and Reproducibility and Replication for what happens when claims are recomputed or tested again.
Sources and further reading
Source pages were checked for this edition on 5 September 2026. The twenty-test, five-p-value and three-look calculations are original illustrative examples. Regulatory guidance linked here demonstrates rigorous applications in clinical trials; it does not turn this general methods article into medical advice.
- U.S. FDA, Multiple Endpoints in Clinical Trials, Guidance for Industry, final guidance, October 2022.
- U.S. FDA, Adaptive Design Clinical Trials for Drugs and Biologics, Guidance for Industry, December 2019.
- U.S. FDA, Use of Bayesian Methodology in Clinical Trials of Drug and Biological Products, draft guidance, January 2026.
- American Statistical Association, Statement on Statistical Significance and P-Values.
- Benjamini and Hochberg, Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing, 1995.
Continue through the Library: Statistical Inference and Uncertainty → Preregistration and Registered Reports → Sensitivity Analysis and Robustness Checks → Research Collections Directory.