The model recommends Option A. Its advantage is five points. Before anyone acts, one person asks: “Which assumption would have to change for Option B to become better?”
That question can be more valuable than another page of polished charts. It asks whether the recommendation is supported across a defensible range of conditions or balanced on one convenient choice.
Sensitivity analysis examines how an output changes when specified inputs, assumptions or analytical choices change. Robustness checks ask whether a stated conclusion survives a declared set of defensible alternatives. Neither is a universal certificate of truth. Their value lies in making the dependence of a result visible enough to challenge, improve or use cautiously.
This guide develops that distinction through constructed examples involving planning, research and library decisions. All numerical examples are illustrative. They are not measured eduKate results, policy recommendations or forecasts.
Reading route: Begin with the four questions often confused, examine a decision threshold, explore the interaction that one-at-a-time checks miss, then move to researcher choices and the reporting contract.
Four different questions behind one confident result
Uncertainty analysis asks how uncertain an output is under a specified model of its inputs. Sensitivity analysis asks which inputs or assumptions contribute to changes in that output. Robustness asks whether a particular conclusion survives selected alternatives. Validation asks whether the model or analysis is adequate for its intended real-world purpose.
A model can produce a narrow uncertainty interval while being structurally wrong. A conclusion can be insensitive to every parameter tested while depending on an untested assumption. An output can vary considerably while the same action remains preferable throughout the range. Conversely, a tiny change in an estimate can reverse a decision located close to a threshold.
Saltelli and colleagues’ review of sensitivity-analysis practice distinguishes sensitivity from uncertainty analysis and emphasises the need to explore the relevant input space. That is the conceptual starting point here, not a reason to assume that every analysis requires the same numerical technique.
The broader model-building task is covered in How Models and Simulations Work. This article addresses the next question: once an answer exists, how do we discover which choices are holding it in place?
Name the conclusion before trying to break it
“The result is robust” is incomplete. Robust in sign, size, ranking, statistical significance, policy choice or performance under a specified disturbance? These are different properties.
Imagine an estimated improvement ranges from one point to seven points across reasonable analyses. The sign is stable. The magnitude is not. If a programme must produce at least four points of improvement to justify its opportunity cost, the implementation decision is also unstable. Calling the whole result robust because every estimate is positive would conceal the decision problem.
Alternatively, two estimates may fall on opposite sides of an arbitrary significance threshold while being nearly identical in size. Describing that as a complete reversal of the substantive finding can exaggerate a minor numerical change. The conclusion should be expressed in the quantities and thresholds that matter to the reader, not automatically in the categories emitted by software.
Write the proposition being challenged. For example: “Option A has greater expected net benefit than Option B under the stated operating conditions.” This provides a clear target for sensitivity analysis and a clear boundary for its eventual claim.
A threshold turns uncertainty into a concrete question
Consider a fictional planning comparison. Let the advantage of Option A over Option B be D = 40 − 3d − c. Here d is delay in days, each day carries a three-point penalty in the agreed scoring model, and c is an additional penalty already expressed in the same points. The score is constructed for explanation, not an objective measure of social value.
At d = 10 and c = 5, Option A leads by five points. Holding c at 5, the options tie when d = 35/3, approximately 11.67 days. The recommendation therefore depends on whether a delay beyond that point is plausible and consequential.
| Delay d | Other penalty c | Advantage D | Preference within this score |
|---|---|---|---|
| 8 | 4 | 12 | Option A |
| 10 | 5 | 5 | Option A |
| 11 | 7 | 0 | Tie |
| 12 | 8 | −4 | Option B |
Notice what has and has not been learned. The analysis has identified combinations that reverse the recommendation. It has not established their probabilities. A table containing four scenarios does not imply that each has a twenty-five per cent chance of occurring.
The next useful task might be to investigate the delay distribution, negotiate a delay-limiting arrangement or select an option less exposed to delay. Sensitivity analysis becomes useful when it changes the question asked of the world, not merely the number of charts produced.
A large slope is not necessarily the largest uncertainty contribution
In the constructed equation, one additional day changes the advantage by three points, while one additional point of c changes it by one point. It is tempting to conclude that delay is always three times as important. That conclusion ignores how uncertain the two inputs are.
If delay is known to within a tenth of a day but c may vary by ten points, uncertainty in c can matter more to the result. Local responsiveness and the plausible range of variation work together. Comparing derivatives measured in different units without interpretation can produce a misleading ranking.
There is also a difference between an influential input and an actionable input. A factor may strongly affect the result but be impossible to change. Another may matter less but be cheap to measure or control. A useful report keeps influence, uncertainty, measurability and controllability separate.
This distinction prevents a common management mistake: turning “the output is sensitive to X” directly into “spend the next budget on X”. A decision about additional research requires another calculation concerning whether better information could change action and improve its expected consequences.
Ranges are evidence-bearing choices
Every sensitivity exercise has a domain. An analyst may vary a parameter by ten per cent, use a confidence interval, consult an engineering tolerance, compare published estimates or construct a deliberately adverse scenario. These choices mean different things.
Ten per cent is not automatically plausible because it is easy to calculate. A fixed numerical range may be too narrow for one input and physically impossible for another. A historical range may omit a regime change. A stress test may deliberately include unlikely combinations because its purpose is to explore failure rather than estimate typical performance.
For each varied input, record where its range came from. Distinguish a measured interval from an expert judgement, a design tolerance and a hypothetical bound. When probabilities are assigned, explain their origin. When they cannot be justified, leave the analysis as scenarios or bounded comparisons rather than giving an arbitrary distribution an undeserved air of precision.
A sensitivity result is conditional on these choices. The research on robustness of Sobol’ indices to distributional uncertainty makes this dependence explicit: changing assumptions about input distributions can change sensitivity conclusions themselves.
The interaction that disappears when everything else is held fixed
Take a deliberately simple mathematical model: y = xz. Let both x and z range from −1 to 1, with a baseline of zero for each. Vary x while z remains zero and the output never changes. Vary z while x remains zero and the output still never changes.
A baseline-only one-at-a-time exercise could declare both factors irrelevant. Yet at x = 1 and z = 1, the output is 1. At x = 1 and z = −1, it is −1. The effect is entirely an interaction: each factor’s influence depends on the other.
This is an original mathematical counterexample, not a fitted model of education or organisational behaviour. Its purpose is to show why exploring one-dimensional paths through a multidimensional space can miss an important property.
Not every method that changes one factor at a time is defective. Some screening methods make local changes at many different locations in the input space. The weakness here is fixing every other input at one baseline and treating that narrow exploration as evidence about the whole domain. The methodological distinctions are discussed in Making sense of global sensitivity analyses.
In practice, the analogous question is whether two modest constraints become serious together. Training may matter only when staff have time to use it. Extra capacity may help only when the next process can receive its output. Sensitivity analysis should test such proposed interactions rather than assume that the whole system is the sum of isolated changes.
What local and global methods can legitimately say
Local analysis studies behaviour near a selected point. It can be appropriate when the operating region is narrow, the model is smooth and small perturbations are the question. A derivative or small finite change then has a clear interpretation: the response around this state, under the stated conditions.
Global analysis explores a broader input domain, often with inputs varying jointly. It can reveal nonlinearities, interactions and changing influence across the range. It usually requires more decisions about ranges, distributions, dependencies and computational design.
Neither label guarantees quality. A global analysis over an unjustifiably narrow domain can be less informative than a carefully targeted local test. An enormous simulation can still omit the disputed mechanism. A small analytical calculation can answer a well-bounded question exactly.
The method should match the question. If the issue is a local engineering tolerance, examine that tolerance. If it is a ranking across uncertain futures, explore relevant combinations. If the question concerns an unsupported causal assumption, varying a numerical input may not be enough; the structure or interpretation of the analysis may need to change.
Variance-based indices: useful, conditional and easy to overread
For independent input variables under a specified probability model, variance-based sensitivity analysis can apportion output variation. A first-order index concerns variation associated with one input’s main effect. A total-order index includes interactions involving that input. The JRC record on sensitivity measures and variance decomposition provides a research route into this family of methods.
Our model y = xz makes the distinction concrete. Suppose x and z are independent and uniformly distributed from −1 to 1. Both have mean zero and second moment 1/3. The output has mean zero and variance (1/3) × (1/3) = 1/9.
Conditioning on x alone leaves the expected value of y at zero, because the average of z remains zero. The first-order index for x is therefore zero, and the same is true for z. Nevertheless, each total-order index is one: every part of the output variation involves the interaction of these two inputs.
Total-order indices can therefore sum to more than one because interaction contributions appear in more than one total. That is not necessarily a computational error. It is a consequence of what the indices measure.
These statements depend on the specified model and independence assumptions. With dependent inputs, the interpretation of familiar decompositions becomes more complicated. Iooss and Prieur’s research on Shapley effects with correlated inputs examines one approach to that problem. An index is not a property of nature detached from its probability model.
Do not create impossible combinations
Joint variation is important, but careless joint variation can create states that the real system cannot occupy. If total expenditure is defined as quantity multiplied by price, varying expenditure, quantity and price independently breaks an accounting relationship. If two inputs are linked by design, treating them as unrelated can distort the analysis.
Imagine a library’s staff hours are divided between cataloguing and enquiry support. Under a fixed staffing budget, increasing both allocations independently may produce scenarios that exceed available hours. A model that then discovers excellent performance has not found a feasible operating policy.
Document physical constraints, accounting identities, logical dependencies and known statistical relationships before designing the variation. Distinguish impossible states from merely unusual states. The first should usually be excluded from the model’s feasible domain; the second may be exactly what a stress test needs to inspect.
Feasibility itself can be uncertain. Where that is true, make it another explicit question instead of silently choosing whichever constraints make the desired recommendation easiest to defend.
Structural assumptions need more than a parameter sweep
A parameter sweep explores values inside a model structure. It does not automatically test whether the structure is suitable. A queue model with no abandonment cannot reveal the consequences of frustrated users leaving, regardless of how many service-time values are simulated.
A hypothetical library planning model might assume that every enquiry requires the same effort. An alternative model could distinguish simple directional enquiries from complex research support. Varying the single average effort in the first model is not identical to representing the mixture in the second, especially if staffing constraints affect the categories differently.
Structural sensitivity therefore compares defensible representations, not merely different numbers. This can include alternative functional forms, aggregation levels, boundary conditions, timing rules or mechanisms. Each alternative needs a rationale. Adding an arbitrary model only to enlarge an ensemble contributes less than testing a specific disputed assumption.
Agreement among alternatives is informative only to the extent that the alternatives challenge different weaknesses. Five models built from the same biased inputs can agree for the same wrong reason.
Researcher choices can be inputs too
Before analysis begins, someone decides how to define variables, handle missing values, identify outliers, select time windows and code ambiguous records. More than one choice may be defensible. Reporting only the route that produces an attractive result can hide how much the conclusion depends on those decisions.
Steegen and colleagues proposed multiverse analysis to make the implications of reasonable data-processing choices visible. The idea is to examine a defined family of analyses rather than present one apparently inevitable dataset and one apparently inevitable result.
For example, a constructed study of reading activity could define engagement as visits, active reading time or completed activities. These may answer different questions. Within a chosen outcome, there might still be reasonable choices about an inactivity threshold or a reporting window. A useful analysis distinguishes changes in measurement procedure from changes in the underlying research question.
A multiverse is not improved by admitting every imaginable specification. Del Giudice and Gangestad’s critical framework for analytic decisions explains why poorly justified alternatives can mislead. Defensible variety is not the same as indiscriminate variety.
Predefine eligibility where possible. Record why each branch belongs. Keep invalid analyses out even if they produce a convenient balance of favourable and unfavourable estimates. The purpose is to expose uncertainty in legitimate choices, not manufacture uncertainty by mixing incompatible questions.
Why counting favourable analyses is not a probability of truth
Suppose ninety of one hundred defensible specifications produce a positive estimate. That does not by itself mean there is a ninety per cent probability that the claim is true. The specifications are usually dependent, their number reflects design choices, and some may differ only trivially.
An analyst could create dozens of near-identical versions of one favourable model and only a few alternatives that genuinely challenge it. A vote count would then reflect how the analysis family was enumerated rather than an independent body of evidence.
More informative reporting shows effect sizes, uncertainty, which choices drive differences and whether the decision changes. Cluster the alternatives by the substantive assumption being tested. A single well-motivated reversal may be more important than fifty nearly redundant confirmations.
Likewise, do not treat every robustness run as a fresh independent replication. Reanalysing the same observations can reveal analytical dependence, but it does not supply new participants, new settings or independent measurements.
Check the code and numerical behaviour separately
Some apparent sensitivity comes from numerical instability or implementation defects. A model may fail to converge at certain inputs. A randomised computation may vary because too few simulations were run. A spreadsheet formula may reference the wrong cell. These are not substantive discoveries about the world.
Begin with basic verification: units, identities, known special cases, simple examples and reproducibility under the same configuration. Where a calculation has a closed-form answer, use it as a reference. Our linear threshold and multiplicative interaction examples are useful precisely because their results can be checked without trusting a complex package.
For stochastic computations, distinguish changes caused by the input from ordinary simulation noise. Record seeds where appropriate, assess convergence and report numerical uncertainty when it can affect interpretation. Running more simulations improves numerical estimation of the selected model; it does not make an unsupported model more realistic.
Failed runs also need investigation. Silently dropping them can make a model appear more reliable by excluding the difficult part of the input space. A failure may reveal infeasibility, a numerical limitation or a programming defect. Those have different implications.
Negative controls, placebos and perturbations have different roles
A robustness review may include checks that are not parameter variations. A placebo time point can ask whether an apparent effect existed before the intervention. A negative-control outcome may help detect certain biases. A leave-one-group-out analysis can reveal whether one group dominates the finding.
These checks require substantive justification. A negative control is useful only if the assumptions that make it a control are credible. Removing an inconvenient group without a reason changes the question. A pre-intervention pattern does not automatically identify the source of bias, but it may challenge an overly simple explanation.
The lesson is not to add a ceremonial checklist of tests. Ask what specific failure each test is designed to detect and what a failure would mean. For causal applications, keep the identification problem anchored to How Causal Inference Works rather than treating many checks as a substitute for a valid design.
Robust action can differ from the action with the best baseline score
The baseline winner may have a narrow advantage and a large downside under plausible alternatives. Another option may perform slightly less well at the baseline but remain acceptable over a wider domain. Choosing between them requires an explicit decision rule.
Possible rules include maximising expected value under justified probabilities, satisfying a minimum performance constraint or limiting regret over a defined scenario set. These rules embody different attitudes to uncertainty and consequence. A model should not silently switch between them whenever the preferred option changes.
In our planning example, a decision-maker might negotiate a cap on delay rather than immediately abandon Option A. Alternatively, the analysis might show that no feasible protection prevents unacceptable outcomes. Sensitivity analysis then supports redesign, not merely a ranking.
There are also non-negotiable constraints. A favourable average score does not compensate for violating a safety or rights requirement. Such requirements should define which options are admissible before an optimisation or trade-off calculation begins.
Choose the next investigation from the vulnerable assumption
A useful sensitivity result often identifies where further evidence would matter. If the decision flips only when an uncertain delay exceeds a threshold, the next investigation may focus on delivery reliability. If every plausible delay yields the same choice, collecting a more precise delay estimate may have little immediate decision value.
Be careful, however: sensitivity does not automatically measure the value of information. A highly influential input may be impossible to learn more about before the deadline. A moderately influential input may be cheap to measure and sit close to the decision boundary. Research priorities depend on both the structure of the decision and the attainable improvement in evidence.
A good handoff from analysis to action therefore names the uncertainty, the decision it could change, the evidence that would reduce it and the cost or delay of obtaining that evidence. “More research is needed” becomes a specific request rather than a reflexive ending.
The reporting contract: say what survived, and what was never tested
Report the baseline, the conclusion being tested, the altered inputs or choices, their ranges and their justifications. Explain dependencies and constraints. Identify what remained fixed. Show the cases that changed the answer, not only a selected set of reassuring results.
Where sensitivity indices are used, name the probability model and the output whose variation was analysed. Where scenarios are used, do not imply probabilities that were never assigned. Where multiple specifications are used, explain their eligibility and dependence rather than treating them as independent votes.
Preserve the code or formulas, data edition, configuration and record of exclusions. These make later corrections possible. A result rerun under a different set of assumptions is a different analytical object, even if its final headline happens to be unchanged.
A strong summary might read: “Option A remains preferred for delays below 11.67 days when the other penalty is five points. The preference reverses in several tested joint scenarios. No probability of those scenarios has been estimated, and the model does not include user abandonment.” That sentence gives the reader both a usable result and the edge of its support.
A classroom exercise in disciplined doubt
Give learners the equation D = 40 − 3d − c and ask them to find three combinations that favour each option, one tie and one impossible combination under a stated constraint. Then ask them to explain why four scenarios do not automatically define four probabilities.
Next, present y = xz at a zero baseline. Ask them to test one variable at a time and then both together. The contrast makes interaction visible without specialist software. Finally, ask them to write a conclusion containing the words “within”, “assuming” and “not tested”.
The exercise teaches more than caution. It teaches how to make a claim stronger by defining the territory it can defend. A conclusion with visible dependencies is easier to improve than a confident statement whose assumptions have vanished.
For a growing knowledge library, this is a crucial habit. Every explanation simplifies. Sensitivity and robustness analysis help readers discover whether a simplification is harmless for their question, decisive for their decision or still too uncertain to carry the weight being placed on it.
Sources and further reading
This edition was prepared with source checks on 5 September 2026. It combines established methodological distinctions with original worked examples. It does not claim to validate any operational model. Technical implementations require method-specific documentation and relevant subject expertise.
- Saltelli and colleagues, Why so many published sensitivity analyses are false: a systematic review of sensitivity analysis practices, 2019.
- Making sense of global sensitivity analyses, Computers & Geosciences, 2014.
- Archer, Saltelli and Sobol’, Sensitivity Measures, ANOVA-like Techniques and the Use of Bootstrap, JRC research record.
- Hart and Gremaud, Robustness of the Sobol’ indices to distributional uncertainty, research preprint, 2018.
- Iooss and Prieur, Shapley effects for sensitivity analysis with correlated inputs, research preprint, 2017.
- Steegen and colleagues, Increasing Transparency Through a Multiverse Analysis, 2016.
- Del Giudice and Gangestad, A Traveler’s Guide to the Multiverse, 2021.
Continue through the Library: Read Models and Simulations for model construction, Statistical Inference and Uncertainty for estimates and intervals, and Strategic Foresight and Scenario Planning for the distinct task of exploring plausible futures. The Research Collections Directory connects these methods to the wider collection.