You collect one sample. Then a statistical question asks you to reason about what might have happened across many samples you never collected.

That apparent contradiction sits near the heart of statistical inference. A standard error describes how an estimate would vary under repeated sampling. A confidence interval is calibrated by repeated-sampling behaviour. Yet in an ordinary project, we do not get to rewind the world and recruit a fresh population a thousand times.

Bootstrap and resampling methods create controlled artificial repetitions from the information already observed. They do not create new evidence. They create new calculations that help us study the behaviour of an estimator, test statistic or prediction procedure under a stated resampling scheme.

The crucial word is stated. Resampling is only as defensible as the unit being resampled, the dependence structure being preserved, the target quantity being estimated and the assumptions that allow the observed data to stand in for repeated draws from the underlying process.

This article owns that computational reasoning job. Statistical Inference and Uncertainty owns the broader inferential framework; Clustered and Multilevel Data owns grouped dependence; and Reproducibility and Replication owns the difference between rerunning an analysis and obtaining genuinely new evidence.

The central problem: one realised sample, an unknown sampling distribution

Suppose five fictional observations are 48, 52, 55, 61 and 84. Their sample mean is 60. The number 60 is easy to compute. The harder question is how unstable that mean might be if the data-generating process were repeated.

If we knew the full population distribution, we could repeatedly sample five observations from it, calculate a mean each time and examine the resulting distribution of means. In real research, that population distribution is usually unknown. The nonparametric bootstrap replaces it with an empirical approximation: the observed values themselves, each given probability 1/5.

A bootstrap sample then draws five observations with replacement from 48, 52, 55, 61 and 84. One sample might be 52, 52, 61, 84, 55. Another might contain 48 three times. Some original observations may be absent from a particular bootstrap sample. This repetition and omission are not mistakes; they are what makes the resamples vary.

There are 5⁵ = 3,125 ordered bootstrap samples in this tiny example. If we enumerate all of them, the mean of the bootstrap means is exactly 60 and the standard deviation of those bootstrap means is about 5.69. That 5.69 is the bootstrap estimate of the standard error under this empirical resampling model.

The number does not mean the original mean is wrong by 5.69. It describes the spread of the estimator across the bootstrap world we have constructed from the observed sample. That distinction between an estimate and an estimate of its uncertainty should remain visible.

Why sampling with replacement matters

If we shuffled the five observed values without replacement and always kept all five, every resample would contain exactly the same values. The mean would always remain 60. We would learn nothing about the sampling variability of the mean.

Sampling with replacement treats the empirical distribution as a stand-in population. Each bootstrap dataset is then a new sample from that stand-in. The operation approximates repeated sampling from an unknown distribution by repeated sampling from an estimated distribution.

This is why bootstrap logic is not simply “copy rows many times”. It is a plug-in argument about a data-generating distribution. Bradley Efron’s 1979 paper Bootstrap Methods: Another Look at the Jackknife introduced the method in this form. Modern software implements many variants, but the conceptual question remains the same: what empirical or fitted distribution is being used to approximate the sampling process?

What exactly are we bootstrapping?

The word bootstrap is incomplete without a resampling unit. In a simple independent sample, the unit may be one person or one measurement. In a paired study, the pair must often travel together. In a clustered study, an entire school, household or site may need to be resampled as a unit. In a time series, individual time points usually cannot be shuffled freely because adjacent observations carry dependence.

Resampling at the wrong level can create artificial information. Imagine 300 learners nested in 10 classes. If class-level teaching conditions create correlation, treating all 300 learners as independent bootstrap units can make the dataset appear to contain more independent evidence than it does.

The repair is not always “use a cluster bootstrap”. The correct procedure depends on the estimand, design, number of clusters and model. The principle is broader: the bootstrap must preserve the dependence structure needed by the inference.

Nonparametric and parametric bootstrap are different promises

In a nonparametric bootstrap, we resample observed units from the empirical distribution. This avoids specifying a full parametric family for the data, but it does not make the method assumption-free.

In a parametric bootstrap, we first fit a probability model, then simulate new datasets from that fitted model. If a count is modelled as Poisson, for example, the resampled datasets can contain count values never observed in the original sample. The method can therefore explore beyond the finite empirical support, but only by trusting the fitted parametric structure.

A parametric bootstrap can be excellent when the model is well justified and the statistic has difficult analytic behaviour. It can be misleading when the fitted model omits the mechanism that generates unusual values, dependence or heterogeneity.

The two forms answer the same broad computational need through different approximations. Neither is automatically more advanced. The question is which approximation better represents the sampling process relevant to the claim.

The bootstrap can estimate standard errors

The basic recipe is simple:

1. Define the statistic of interest.
2. Generate a bootstrap dataset under the chosen resampling scheme.
3. Recalculate the statistic.
4. Repeat many times.
5. Measure the spread of the bootstrap statistics.

For the five-value example, exact enumeration gave a bootstrap standard error of about 5.69 for the mean. A Monte Carlo bootstrap using a large random subset of the 3,125 possible resamples would approximate the same quantity.

This illustrates an important separation: sampling uncertainty and Monte Carlo uncertainty are different. The spread of the bootstrap distribution estimates sampling variability. The extra numerical noise caused by using only a finite number of bootstrap replications is Monte Carlo error. Increasing the number of resamples reduces the second; it does not create a larger original sample or eliminate the first.

Confidence intervals require more than taking two percentiles

A natural first idea is to take the 2.5th and 97.5th percentiles of the bootstrap statistic as a 95% interval. This percentile interval can work reasonably in some problems, but its calibration is not universally good.

The five-value example illustrates the mechanics. Across all 3,125 ordered bootstrap samples, the 2.5th and 97.5th percentiles of the bootstrap mean are approximately 51.0 and 72.4. Those numbers describe the central region of the bootstrap distribution of the mean. They should not be presented as an automatically exact confidence interval merely because the percentiles are easy to calculate.

Different bootstrap intervals correct different problems. Basic intervals reflect the bootstrap distribution around the observed estimate. Bias-corrected and accelerated, or BCa, intervals adjust for median bias and changing standard error. Studentized approaches resample a standardized statistic. Each method has conditions and implementation details.

The current SciPy bootstrap documentation, checked for this edition, exposes percentile, basic and BCa procedures and explicitly describes resampling with replacement. Software availability does not remove the need to decide whether the underlying resampling model is suitable.

Bootstrap bias estimation asks a different question

Suppose an estimator tends to systematically overshoot or undershoot its target across repeated samples. The bootstrap can approximate that bias by comparing the average bootstrap estimate with the estimate from the original sample.

If the original estimate is θ̂ and the mean of the bootstrap estimates is θ̄*, a common bootstrap bias estimate is θ̄* − θ̂. A bias-corrected point estimate might then subtract this estimated bias.

Bias correction is not automatically improvement. Removing estimated bias can increase variance, and the bootstrap bias estimate itself can be noisy. The relevant criterion is performance for the target problem, not whether the corrected number looks more sophisticated.

The jackknife and bootstrap are related but not identical

The jackknife repeatedly recalculates an estimator while leaving out one observation, or more generally one resampling unit, at a time. With n observations, the ordinary leave-one-out jackknife produces n derived estimates rather than thousands of bootstrap estimates.

The jackknife is computationally economical and can work well for smooth statistics. It also helps estimate acceleration terms used by BCa intervals. But it behaves poorly for some nonsmooth estimators and should not be treated as a cheaper bootstrap with identical properties.

Historically, the bootstrap was developed as a broader resampling approach that could tackle problems where simple jackknife approximations were limited. The family resemblance matters; the methods are not synonyms.

Permutation tests answer a different counterfactual question

Bootstrap samples usually resample observations to approximate a sampling distribution. Permutation or randomisation tests usually rearrange labels or assignments under a null hypothesis that makes such rearrangements exchangeable.

Imagine two groups with observed outcomes. A permutation test may shuffle group labels to ask how unusual the observed group difference would be if, under the null structure, the labels were irrelevant. A bootstrap may instead resample within groups to estimate uncertainty in the group means or their difference.

Both involve repeated computation. Their logical reference worlds differ. Confusing them can produce a correct-looking distribution attached to the wrong inferential question.

Cross-validation is also resampling, but its job is prediction

Cross-validation repeatedly splits or rotates data between fitting and evaluation roles to estimate out-of-sample predictive performance. Bootstrap procedures can also estimate prediction error, but an ordinary bootstrap standard error and k-fold cross-validation are not interchangeable.

In prediction, the central question is often how a fitted procedure performs on new cases. In classical bootstrap uncertainty, the central question is often how an estimator varies across repeated samples. One algorithmic vocabulary can therefore conceal several different statistical jobs.

This is why resampling methods should be described by what is resampled, why it is resampled and what quantity the repetition estimates.

Small samples expose the edge of the empirical distribution

The nonparametric bootstrap can only resample values and patterns represented in the observed empirical distribution. With five observations, the empirical stand-in population contains only five support points. If an important tail behaviour or subgroup is absent, resampling cannot magically reconstruct it.

Consider a rare failure mode that occurs once in a thousand operations. A sample of twenty with zero failures does not prove the true failure probability is zero. Bootstrapping those twenty observations nonparametrically will produce zero failures in every resample, giving a dangerously reassuring picture if the analyst forgets what the empirical distribution omitted.

The repair might involve a parametric model, exact small-sample method, prior information, additional data or a bound. Which is appropriate depends on the scientific problem. The bootstrap’s inability to invent unseen tail structure is a limitation to respect, not a bug to patch with more resamples.

Extreme values can dominate the bootstrap world

Our fictional value 84 is much larger than the other four observations. Some bootstrap samples omit it; others contain it several times. That is why the bootstrap distribution of the mean is asymmetric enough to deserve inspection.

This behaviour can be informative. It tells us that one observed unit has substantial influence. But it also reminds us that the bootstrap treats the observed empirical mass as if it approximated the population. If the extreme value is a data error, resampling amplifies the error. If it is a legitimate rare observation, deleting it for convenience can hide real instability.

Data-quality review therefore precedes resampling. Data Quality owns the question of whether the record is accurate and valid; the bootstrap owns what the estimator does when the accepted record is repeatedly sampled.

Time series need their chronology preserved

If today’s observation is related to yesterday’s, independently resampling individual days destroys temporal dependence. The resulting pseudo-series can contain transitions the real process would rarely or never generate.

Block bootstrap methods resample contiguous stretches of observations so local dependence can survive. Variants include moving blocks, circular blocks and stationary bootstrap schemes. Their block length becomes another modelling choice: blocks that are too short lose dependence; blocks that are too long leave too few effectively independent pieces.

A model-based residual bootstrap can instead fit a time-series model and resample residual components under assumptions about the innovation process. Again, the method inherits the structure it assumes.

Regression needs a resampling scheme that matches the error story

For regression, one can resample observed rows, resample residuals under a fitted model, use wild bootstrap procedures for heteroskedastic errors or resample clusters when observations share a group-level disturbance.

These choices are not cosmetic. Resampling residuals assumes a relationship between the fitted model and the error structure. Resampling rows treats the observed joint distribution of predictors and outcomes differently. A wild bootstrap is designed to preserve certain forms of nonconstant variance.

The regression owner, How Regression Models Work, should remain the route for coefficient interpretation, adjustment and diagnostics. Resampling becomes a supporting computational method, not a second canonical explanation of regression.

Bootstrap does not fix selection bias

Suppose a survey systematically misses people who are least likely to use a service. Repeatedly bootstrapping the respondents can estimate the instability of statistics among the respondent distribution. It does not recover the people who were never represented.

The same applies to convenience samples, survivorship bias and selective measurement. Resampling can make a biased estimate look extremely precise if the observed dataset is large. Precision around the wrong target is still wrong.

Use Survey Weighting, Calibration and Nonresponse Adjustment for representation problems and Missing Data Analysis when unavailable values may alter the inference. Bootstrap methods can be embedded inside those analyses, but they do not replace the identification argument.

Bootstrap does not repair a wrong causal design

A thousand bootstrap replications of a confounded observational comparison remain a thousand recalculations of the same design. The resampling distribution may honestly describe the statistical variability of the chosen estimator while the estimator itself targets the wrong causal contrast.

For causal questions, the route returns to How Causal Inference Works and How Quasi-Experimental Designs Work. Computational uncertainty cannot substitute for causal identification.

Monte Carlo error should be measured, not hidden

When only a random subset of possible bootstrap samples is used, the reported bootstrap statistic changes slightly with the random stream. Ten thousand resamples will usually provide a more stable percentile than one hundred, but “more” is not an epistemic law.

If a decision depends on the third decimal place of a bootstrap estimate, rerun with more resamples or estimate Monte Carlo error. Preserve the random-number generator and seed where reproducibility matters. Then distinguish reproducibility of the computation from replication of the research result.

A fixed seed helps another analyst reconstruct the numerical path. It does not make the path more representative of reality. The seed is part of computational provenance, not evidence for the substantive claim.

A practical resampling decision sequence

1. Name the estimand. What population quantity or predictive property is the analysis trying to learn?
2. Name the estimator or procedure. Mean, median, coefficient, accuracy, difference, quantile, prediction error or something else?
3. Identify the independent resampling unit. Person, pair, household, class, site, block of time, residual or simulated dataset?
4. Preserve structure. Pairing, clustering, stratification, time dependence, censoring and design weights may need explicit treatment.
5. Choose the bootstrap world. Empirical, parametric, residual, wild, cluster, block or another justified scheme.
6. Choose the inferential output. Standard error, bias estimate, interval, test statistic, predictive error or stability diagnostic.
7. Check failure modes. Tiny samples, rare events, boundaries, extreme influence, absent subgroups and model misspecification.
8. Check numerical stability. Increase resamples when Monte Carlo error matters and preserve random-state provenance.
9. Report the scheme. “We bootstrapped” is not enough.

A reporting paragraph should let someone reconstruct the logic

A useful report states the resampling unit, number of resamples, statistic, confidence-interval method, treatment of dependence and random-state policy. If the procedure is nonstandard, state why it was selected and what failure it was designed to prevent.

For example: “We estimated uncertainty in the median using 20,000 nonparametric bootstrap resamples of participants, preserving each participant’s paired pre/post measurements. We report a BCa interval and verified that increasing to 50,000 resamples changed the interval endpoints by less than the reporting precision.”

That statement says what the artificial repetitions represent. A reader can disagree with the resampling assumption without having to reverse-engineer the software.

What a learner should remember

The bootstrap is powerful because it lets one observed sample teach us about the behaviour of a statistical procedure without requiring a neat analytic formula for every problem. Its weakness is the same move: it asks the observed data, or a fitted model based on them, to stand in for the repeated world we did not observe.

So do not ask only, “How many bootstrap samples did you run?” Ask, “What was resampled, what structure was preserved, what population did the resampling world represent, and which uncertainty did the resulting distribution actually measure?”

Resampling is not a machine for manufacturing evidence. It is a machine for interrogating the consequences of a stated sampling model.

Sources and further reading

Source records and current software documentation were checked for this edition on 5 September 2026. The five-value calculations are original demonstrations and can be exactly enumerated; they are not observations from an eduKate dataset.

• Bradley Efron, Bootstrap Methods: Another Look at the Jackknife, The Annals of Statistics, 1979.
• Bradley Efron and Robert Tibshirani, An Introduction to the Bootstrap, Chapman & Hall/CRC.
• SciPy, current scipy.stats.bootstrap documentation, accessed for this edition.

Continue through eduKate: Statistical Inference and Uncertainty → Regression Models → Clustered and Multilevel Data → Reproducibility and Replication → Research Collections Directory.