Meta-Analysis, Heterogeneity and Publication Bias

Five studies report an effect. Four are positive. One is negative. The obvious temptation is to average the numbers and announce the answer.

But one study may be enormous and four tiny. They may measure different outcomes, use different populations, follow participants for different lengths of time, estimate different effect scales or carry different risks of bias. Their disagreement may be noise, genuine effect variation, methodological difference or missing evidence.

Meta-analysis is a statistical framework for combining estimates from multiple studies under a defined synthesis model. It can improve precision and reveal patterns across studies, but it does not convert incompatible evidence into one truth. Heterogeneity asks how and why underlying effects differ. Publication and non-reporting bias ask whether the available evidence is a distorted subset of the evidence that should have been available.

This article explains the full chain: review question → eligible studies → compatible effect measures → study estimates and uncertainty → weighting → synthesis model → heterogeneity → sensitivity → missing evidence → interpretation → transfer.

The numerical examples are constructed for explanation. They are not eduKate programme results or clinical recommendations.

Meta-analysis begins before the formula

If study inclusion depends on seeing the results, the synthesis can be biased before the first weight is calculated. A systematic review therefore defines the question, eligibility criteria, search process, outcomes and analysis plan before treating the retrieved studies as a dataset.

The existing eduKate owner How Systematic Reviews and Evidence Synthesis Work covers the full review process. This article owns the statistical and interpretive problem that begins once studies are considered for quantitative combination.

PRISMA 2020 is a reporting guideline, not a guarantee that a review is methodologically correct. Its checklist and flow diagram make the review process more inspectable by requiring clear reporting of identification, inclusion and analysis.

The unit being combined is an estimate, not a study title

Each study contributes an estimate of an effect and information about the uncertainty of that estimate. A meta-analysis combines those estimates according to a weighting rule and synthesis model.

A study with a larger sample is often more precise and receives greater weight, but sample size is not itself the universal weight. The standard error depends on outcome type, design, event frequency and model.

Two papers from the same underlying cohort are not automatically independent studies. Multiple outcomes or time points from one study also create dependence that standard inverse-variance formulas may not handle without adjustment.

Effect measures must answer the same enough question

Continuous outcomes can be synthesised as mean differences when studies use the same scale, or standardised mean differences when scales differ but are intended to represent a common construct. Binary outcomes may use risk ratios, odds ratios or risk differences. Time-to-event outcomes often use hazard ratios.

These measures are not interchangeable labels. A risk ratio of 0.5 and a risk difference of −0.10 can describe the same study under one baseline risk and different absolute consequences under another.

Choose the effect measure according to the decision and the assumptions about comparability. Standardising a score does not prove that different instruments measure the same construct. That prior question belongs to Construct Validity and Measurement Models and the measurement-invariance owner.

A weighted mean can be reconstructed by hand

Suppose three fictional studies estimate effects of 2, 4 and 8 points with standard errors 1, 2 and 4. Under an inverse-variance fixed-effect calculation, weights are proportional to 1 divided by the squared standard error: 1, 0.25 and 0.0625.

The weighted estimate is:

(1×2 + 0.25×4 + 0.0625×8) ÷ (1 + 0.25 + 0.0625) = 3.5 ÷ 1.3125 ≈ 2.67.

The smallest-standard-error study dominates the result. A simple unweighted mean would be 4.67, a very different number. The weighting model therefore matters even before heterogeneity is considered.

Fixed-effect synthesis makes a strong common-effect assumption

A fixed-effect model treats all included studies as estimating one common underlying effect, with observed differences caused by sampling error. Under that model, more precise studies receive greater weight.

This can be appropriate when the scientific question and study set genuinely support a common-effect interpretation. It becomes difficult when interventions, populations or implementations plausibly produce different true effects.

Cochrane’s current Handbook warns that fixed-effect synthesis ignores heterogeneity and that a common summary can be misleading when true effects vary.

Random-effects synthesis changes the target

A random-effects model assumes studies estimate different but related underlying effects drawn from a distribution. The pooled estimate represents the centre of that distribution under the model, not one universal effect shared by every study.

Between-study variance, usually denoted τ², is added to within-study variance when calculating weights. This reduces the dominance of very large studies and gives relatively more weight to smaller ones than a fixed-effect analysis would.

That property can be useful when genuine variation exists. It can also amplify small-study bias if smaller studies systematically report larger effects.

Random effects does not “solve” heterogeneity

Switching to a random-effects model does not make heterogeneous studies comparable. It incorporates an estimated distribution of effects into the synthesis. The reasons for variation still matter.

If effects differ because one study uses a fundamentally different intervention or a biased outcome measurement, the average can be scientifically unhelpful even when the model runs correctly.

Cochrane explicitly notes that random effects is not a substitute for investigating heterogeneity.

Heterogeneity begins with clinical, educational or contextual diversity

Studies can differ in participants, intervention intensity, comparator, outcome, follow-up, setting and implementation. These are substantive differences.

They can also differ methodologically: randomisation quality, blinding, missing data, measurement error or selective reporting. Statistical heterogeneity is the observed variation in estimates beyond what sampling error alone would predict, but its source can be substantive diversity, methodological diversity or both.

The forest plot is therefore not just a picture of numbers. It is a map of studies whose contexts need to be read.

The Q test asks whether variation exceeds sampling error

Cochran’s Q statistic compares observed study deviations with what would be expected under a common-effect model. A small p-value suggests more variation than chance alone would explain.

The test has low power when few studies exist and can detect trivial heterogeneity when many studies exist. A non-significant Q is therefore not proof of homogeneity.

Never choose fixed versus random effects solely from whether Q crosses a significance threshold. The synthesis model should follow the scientific estimand and plausible effect structure.

I² is a ratio, not the amount of effect variation

I² is commonly interpreted as the proportion of observed variability in effect estimates attributable to heterogeneity rather than sampling error under the model. It does not measure the magnitude of the between-study effect differences in the outcome’s natural units.

An I² of 80% can occur with small absolute effect differences when studies are extremely precise. A lower I² can occur with larger differences when studies are imprecise.

Cochrane warns against using simple I² thresholds mechanically, especially when few studies make the statistic uncertain.

τ² and τ put heterogeneity on the effect scale

τ² estimates the between-study variance in a random-effects model. Its square root, τ, is the standard deviation of the assumed distribution of underlying effects.

Unlike I², τ is expressed on the meta-analysis effect scale. That can make it more directly relevant to understanding how much true effects may vary, although interpretation can still be difficult for ratio measures and unfamiliar scales.

Prediction intervals answer a different question from confidence intervals

A confidence interval around the random-effects mean quantifies uncertainty about the mean of the assumed effect distribution. It does not show the full range of effects across settings.

A prediction interval aims to describe where an underlying effect from a new study similar to those included might lie, accounting for between-study variation and uncertainty.

Cochrane encourages prediction intervals when the number of studies is sufficient and interpretation is appropriate, while warning that they can be unstable with few studies and depend on the assumed effect distribution.

A pooled positive mean can coexist with harmful effects in some settings

Suppose a random-effects mean suggests a beneficial effect of +5, but the prediction interval extends from −3 to +13. The average is positive, yet the model allows materially negative effects in some comparable settings.

Reporting only the pooled mean can therefore hide the decision-relevant heterogeneity. The next question is what distinguishes settings at the negative and positive ends.

Subgroup analysis asks whether a prespecified characteristic explains variation

Researchers may divide studies by age group, implementation intensity, study design or setting and compare pooled effects. A plausible effect modifier can make the heterogeneity scientifically meaningful.

But testing many subgroup splits invites false discoveries. A difference between “significant” and “not significant” subgroups is not itself evidence that subgroup effects differ. The interaction or subgroup-difference test is the relevant comparison.

Subgroup definitions should ideally be planned from theory, not invented after looking at the forest plot.

Meta-regression extends subgroup analysis to study-level predictors

Meta-regression models study effect estimates as a function of study-level characteristics. It can examine whether effect size changes with mean age, dose, year or another recorded study feature.

Its unit is usually the study, not the individual participant. A study-level association can therefore suffer ecological bias. If studies with older participants show larger effects, that does not prove older individuals benefit more within studies.

With few studies, meta-regression is easily overfit. One coefficient per handful of studies can create a smooth explanatory story from noise.

Individual participant data can answer richer questions

Individual participant data meta-analysis obtains participant-level records from eligible studies and reanalyses them consistently. This can harmonise outcomes, improve subgroup analysis and model interactions at the participant level.

It also requires extensive data governance, harmonisation and missing-data work. Studies that cannot share data may be systematically different from those that can, creating another selection issue.

A forest plot can hide study dependence

One research team may publish several outcomes from the same sample. Treating each as an independent study artificially increases information. Cluster-randomised studies may also require adjusted standard errors before synthesis.

Multilevel meta-analysis, robust variance methods or pre-specified selection of one estimate per study can address dependence depending on the question.

Standardised mean differences can create hidden heterogeneity

The standardised mean difference divides a mean contrast by a within-study standard deviation. If variability differs across populations because of measurement or eligibility, the same absolute difference can produce different standardised effects.

Heterogeneity can therefore enter through both numerator and denominator. Standardisation solves a unit problem, not every comparability problem.

Baseline risk changes the meaning of relative effects

A constant relative effect can produce different absolute benefits when baseline risk varies. If a risk ratio is 0.5, reducing risk from 40% to 20% is a twenty-point absolute reduction; reducing 4% to 2% is two points.

Decision-makers often need absolute consequences in their target population. Transporting a pooled relative effect therefore requires target baseline risk and another layer of assumptions.

See External Validity and Evidence Transfer.

Risk of bias is not solved by weighting

Inverse-variance weighting gives precise studies more influence. It does not automatically give less biased studies more influence. A very large biased study can dominate a fixed-effect meta-analysis.

Risk-of-bias assessment remains a separate evidence judgement. Sensitivity analyses can exclude studies at high risk or model bias under explicit assumptions, but no generic weight converts study quality into truth.

Publication bias is one form of missing evidence

Studies with statistically significant or favourable results may be more likely to be published, published quickly, published in visible journals or fully reported. If missing evidence depends on the magnitude, direction or significance of results, the available literature can be systematically distorted.

Cochrane’s current Chapter 13 prefers the broader concept of bias due to missing evidence and distinguishes missing whole studies from selective non-reporting of particular results within identified studies.

The funnel plot is not a publication-bias detector

A funnel plot places study effect estimates against a measure of precision or size. In a simple world without bias or heterogeneity linked to study size, small studies scatter widely and large studies cluster near the underlying effect, creating a roughly symmetrical funnel.

Asymmetry can arise because unfavourable small studies are missing. It can also arise because small studies use different populations, interventions or methods, because poorer-quality studies exaggerate effects, because effect measures create mathematical relationships with standard errors or simply by chance.

Cochrane explicitly states that funnel-plot asymmetry should not be treated as diagnostic for publication bias.

Small-study effects are the broader phenomenon

A small-study effect means smaller studies tend to report different effect sizes from larger studies. Publication bias is one possible cause.

Other causes include different populations, more intensive interventions in small trials, methodological weakness or genuine effect modification by setting.

Label the observed pattern before assigning its cause.

Funnel-plot tests need enough studies

Tests such as Egger’s test examine associations between effect estimates and study precision. They have low power with few studies and can behave poorly under heterogeneity.

Cochrane gives a rule of thumb that such tests should generally be used only when at least ten studies are available, and even then interpretation requires caution.

Trial registries and protocols can be stronger evidence of missing results

If a study protocol prespecified three outcomes and the publication reports only the favourable one, the missing-result problem is visible directly. No funnel plot is required.

Comparing protocols, registries, statistical analysis plans and publications can therefore reveal selective outcome reporting that aggregate effect plots cannot identify reliably.

Grey literature expands retrieval but does not become high quality by being unpublished

Searching theses, conference proceedings, trial registries and reports can reduce publication-selection problems. It can also bring incomplete methods and difficult-to-verify results.

The right approach is to search broadly enough for the review question and then assess each source according to its evidence quality. “Peer reviewed” and “unpublished” are not perfect proxies for validity.

Trim-and-fill is not a machine that restores missing truth

Methods such as trim-and-fill attempt to estimate and adjust for funnel asymmetry under particular assumptions. They can be useful as sensitivity analyses.

They do not observe the missing studies, know why evidence is absent or prove that asymmetry comes from publication bias. An adjusted estimate should therefore be labelled as model-based sensitivity, not corrected truth.

Selection models make the missingness assumption explicit

Selection models specify how the probability that a result appears in the evidence base may depend on effect size, standard error, p-value or other features. They can estimate how pooled effects change under different selection mechanisms.

The missingness process is not fully identified from published studies alone. Sensitivity over plausible selection models is therefore often more defensible than presenting one adjusted estimate as certain.

p-curve and related methods answer narrower questions

Methods based on the distribution of statistically significant p-values have been proposed to detect selective reporting or estimate evidential value. Their performance depends on assumptions about power, heterogeneity, selection and the set of results included.

No single diagnostic can certify an evidence base free from selective reporting. Use multiple evidence sources and understand what each method can detect.

Living reviews add time to the synthesis problem

New studies can arrive after a meta-analysis is published. A living systematic review repeatedly updates searches and synthesis when the question warrants continual surveillance.

PRISMA-LSR, published in 2024, extends PRISMA 2020 for living reviews. Repeated updating adds version-control questions: which evidence base supported which conclusion at which date?

A living meta-analysis is therefore a versioned knowledge object, not one final number.

Cumulative meta-analysis can show when evidence changed

Studies can be added chronologically to show how the pooled estimate and uncertainty evolved over time. This can reveal whether later research confirmed, attenuated or reversed an early result.

It can also tempt retrospective stories about when researchers “should have known”. Earlier evidence must be judged using the information and methods available at that time, not the benefit of hindsight.

Network meta-analysis asks a different comparative question

When several interventions have been compared in overlapping trial networks, network meta-analysis can combine direct and indirect comparisons. If A was compared with B and B with C, information can contribute to A versus C under transitivity and consistency assumptions.

This requires stronger structural assumptions than a pairwise meta-analysis. Populations and effect modifiers across comparison sets must support the indirect bridge.

Meta-analysis of observational studies requires special caution

Observational studies can use different adjustment sets, exposure definitions and confounding controls. Pooling their adjusted estimates can combine different estimands.

A narrow confidence interval around a pooled observational association does not eliminate residual confounding shared across studies. Systematic bias can be repeated very precisely.

See How Observational Studies Work and How Causal Inference Works.

Meta-analysis cannot rescue poor measurement comparability

If studies label different constructs with the same word, the pooled estimate may be numerically valid and conceptually incoherent. Standardised effect sizes can hide this problem by putting every measure on a unitless scale.

Construct mapping and measurement invariance should therefore be inspected before synthesis when different instruments, languages or populations are involved.

Sensitivity analysis should target disputed decisions

Useful sensitivity analyses might compare fixed and random effects, alternative τ² estimators, exclusion of high-risk studies, different correlation assumptions for dependent outcomes, alternate effect scales or models for missing evidence.

The purpose is not to generate dozens of versions until one looks reassuring. It is to reveal which reasonable decisions materially change the conclusion.

See Sensitivity Analysis and Robustness Checks.

A study-count vote is not meta-analysis

“Seven studies were significant and three were not” discards effect sizes, precision and study quality. A small imprecise study and a large precise study become equal votes.

Even counting direction alone can mislead. Ten tiny positive estimates and one large negative estimate may not support the same conclusion as ten independent equally precise studies.

Statistical significance of the pooled effect is not the decision

A pooled effect can be statistically distinguishable from zero while being too small to matter. A non-significant pooled effect can hide a prediction interval containing both important benefit and harm.

Interpret magnitude, uncertainty, heterogeneity, risk of bias and target relevance together.

The synthesis audit

QUESTION + ELIGIBILITY
→ COMPLETE SEARCH + STUDY IDENTITY
→ EFFECT MEASURE COMPATIBILITY
→ WITHIN-STUDY BIAS ASSESSMENT
→ DEPENDENCE CHECK
→ FIXED / RANDOM / OTHER SYNTHESIS MODEL
→ HETEROGENEITY ESTIMATE
→ PREDICTION INTERVAL WHERE APPROPRIATE
→ SUBGROUP / META-REGRESSION WITH THEORY
→ MISSING-EVIDENCE ASSESSMENT
→ SENSITIVITY ANALYSIS
→ TARGET-POPULATION INTERPRETATION
→ VERSION + UPDATE PLAN

A forest plot reading protocol

What effect measure is on the x-axis?
Are all studies estimating sufficiently comparable quantities?
Which studies dominate the weight?
How much do estimates disagree in magnitude and direction?
Is the pooled model fixed effect or random effects?
What are τ², I² and their uncertainty?
Is a prediction interval shown?
Do high-risk studies cluster on one side?
Are small studies systematically different?
Would the conclusion change under a plausible alternative synthesis?

A funnel plot reading protocol

Are there enough studies for the plot to be informative?
Which precision measure is on the vertical axis?
Is the effect measure transformed appropriately?
Could heterogeneity explain the asymmetry?
Do small studies differ in population or intervention?
Are protocols or registries available to inspect missing outcomes directly?
What sensitivity analyses were run?
Does the report call asymmetry “publication bias” without considering alternatives?

What a strong conclusion sounds like

A weak conclusion is: “The meta-analysis proves the intervention works.”

A stronger conclusion is: “Across twelve eligible studies, the random-effects mean favoured the intervention, but between-study heterogeneity was substantial and the prediction interval included little or no benefit. Smaller studies reported larger effects, and protocol comparison identified selective non-reporting in two studies. The average estimate should therefore not be interpreted as a universal effect.”

The stronger version tells the reader what the synthesis knows and where the evidence becomes fragile.

The World Return of meta-analysis

Meta-analysis can turn scattered research into cumulative knowledge. It can reveal a stable signal too weak for individual studies, show that an average hides meaningful variation, identify where evidence is missing and expose how conclusions evolved through time.

Its greatest value is not the production of one final number. It is disciplined synthesis: preserving enough of each study’s uncertainty and difference that the combined answer remains accountable to the evidence it came from.

Sources and current guidance

Sources were checked for this edition on 5 September 2026. Cochrane Handbook chapters cited below were last updated in 2024. PRISMA sources are current at the time of checking.

Cochrane Handbook — Chapter 10: Analysing data and undertaking meta-analyses, last updated November 2024.
Cochrane Handbook — Chapter 13: Assessing risk of bias due to missing evidence in a meta-analysis, last updated August 2024.
PRISMA — PRISMA 2020 statement.
PRISMA — PRISMA 2020 checklist.
PRISMA — PRISMA for Living Systematic Reviews, published 2024.

Continue through eduKate

Editorial takeaway: A pooled estimate is not a blender. Good synthesis keeps the shape of the evidence visible: what studies agree on, where they diverge, what is missing and which target still needs its own answer.

How Meta-Analysis, Heterogeneity and Publication Bias Work | Combining Studies Without Flattening Their Differences