A line through a cloud of points looks simple. That simplicity can hide several different jobs: describing a pattern, estimating an association, predicting an outcome, adjusting a comparison, testing a model, or supporting part of a causal analysis.

The calculation may be identical while the meaning is not. A coefficient of 2 can mean “two more units on average for each one-unit difference in X, conditional on the included model” without meaning that changing X will cause Y to increase by two. Regression is powerful partly because the same mathematical machinery can serve many purposes. It is dangerous for exactly the same reason.

Regression models relate an outcome to one or more predictors through an explicit mathematical structure. They estimate parameters, generate fitted values, quantify uncertainty and expose residual variation. Their usefulness depends on the design, the variables, the functional form, the error structure and the interpretation placed on the coefficients.

This article is a general research-methods explainer. Its examples are constructed. It does not replace specialist models for survival data, clustered data, causal inference or medical prediction.

The regression reasoning loop

QUESTION
→ DEFINE OUTCOME
→ DEFINE PREDICTORS
→ DEFINE INTERPRETIVE JOB
→ CHOOSE FUNCTIONAL FORM
→ FIT PARAMETERS
→ INSPECT RESIDUALS / PREDICTIONS
→ CHECK ASSUMPTIONS
→ QUANTIFY UNCERTAINTY
→ TEST SENSITIVITY
→ INTERPRET WITHIN DESIGN BOUNDARIES
→ VALIDATE OR REPLICATE
→ RETURN TO THE WORLD

The first question is not “Which regression command should I run?” It is “What relationship am I trying to represent, and what claim will I make from it?”

1. A regression model is a conditional model

Suppose Y is an assessment score and X is hours of practice. A simple linear regression writes:

Y = β0 + β1X + ε.

β0 is the intercept, β1 is the slope and ε represents variation not captured by the systematic part of the model. The fitted equation replaces the unknown parameters with estimates.

The slope describes how the model’s expected outcome changes with X. It is conditional on the model specification. Add another predictor and the coefficient can change because the conditioning statement changed.

2. Least squares chooses coefficients by minimizing residual squares

In ordinary least squares, the fitted coefficients minimize the sum of squared differences between observed outcomes and fitted values. The NIST Engineering Statistics Handbook describes this criterion directly.

If the observed outcome for case i is yi and the fitted value is ŷi, the residual is ri = yi − ŷi. Least squares chooses coefficients that make the sum of ri² as small as possible.

Squaring does two things: positive and negative deviations no longer cancel, and large residuals receive disproportionate weight. That is mathematically convenient and statistically powerful under the usual model assumptions, but it also makes ordinary least squares sensitive to extreme observations.

3. The intercept may be useful even when it has no substantive meaning

The intercept is the model’s expected outcome when all predictors equal zero. Sometimes zero is meaningful. Sometimes it lies far outside the observed range.

Suppose age in a dataset ranges from 12 to 16 years. The intercept at age zero may have no useful educational interpretation. Centering age at 14 changes the intercept into the expected outcome at age 14 without changing the fitted line.

Parameterisation affects interpretation, not necessarily model fit.

4. A slope is a model contrast

If the fitted slope is 2, then two observations differing by one unit in X are predicted to differ by two units in Y, under the model and holding other included predictors fixed where relevant.

The phrase “holding other variables fixed” is mathematical conditioning. It does not necessarily describe a feasible intervention. Holding age, prior attainment and school fixed while changing a variable may be conceptually meaningful for estimation yet impossible as a literal real-world manipulation.

5. Regression can describe association without identifying cause

A regression coefficient can be estimated from observational data even when important confounders are unmeasured. The software will still return a number.

Suppose practice hours are positively associated with scores. Learners who practise more may also differ in motivation, prior knowledge, support or available time. A regression of score on practice does not by itself distinguish the effect of practice from those differences.

The canonical owner for causal interpretation is How Causal Inference Works. Regression is a tool inside some causal designs; it is not a causal design by itself.

6. Multiple regression changes the question by conditioning

A multiple regression might write:

Y = β0 + β1X1 + β2X2 + ε.

β1 now describes the association between X1 and Y at fixed X2 under the model. It is generally not the same quantity as the coefficient from a regression containing only X1.

This is not evidence that one coefficient is wrong. The models answer different conditional questions.

7. Adjustment can reduce confounding or create bias

Including a variable because it is available is not a valid adjustment strategy. A pre-treatment common cause of exposure and outcome may need adjustment. A mediator may block part of the effect one wants to estimate. Conditioning on a collider can create an association that was absent before conditioning.

Variable selection for causal analysis should follow the causal question and assumed structure, not a mechanical rule such as “include every significant predictor”.

For prediction, the goal differs: a variable can improve prediction even when it should not be interpreted causally.

8. Regression coefficients depend on units

A coefficient of 0.5 per minute becomes 30 per hour if the predictor is rescaled from minutes to hours. The underlying fitted relationship is unchanged.

Always read a coefficient with its units. “The effect is 0.5” is incomplete.

9. Standardising predictors changes coefficient meaning

Standardisation can place predictors on comparable numerical scales by subtracting a mean and dividing by a standard deviation. A standardised coefficient then describes the model change associated with a one-standard-deviation predictor difference.

This can aid comparison, but it does not make variables equally reliable, equally manipulable or equally important in the real world. One standard deviation of family income and one standard deviation of practice time are not interchangeable interventions.

10. A straight line is a substantive assumption

Linear regression assumes that the conditional mean can be represented by the specified linear combination of terms. That does not mean the world is literally linear. It means linearity is the chosen approximation over the region of interest.

If study time helps until fatigue sets in, a straight line may misrepresent the relationship. A residual pattern or domain knowledge can suggest curvature.

NIST repeatedly emphasises model checking after fitting rather than interpreting numerical output first. See its initial model fitting example.

11. Polynomial terms create curves within a regression framework

A model with X and X² can represent curvature:

Y = β0 + β1X + β2X² + ε.

Once X² is included, β1 is no longer the overall slope. The slope depends on X. Interpreting β1 alone as “the effect of X” is incomplete.

Higher-degree polynomials can fit complex shapes but may behave wildly near boundaries and outside the observed range. Flexibility is not free.

12. Splines allow flexible curves with local structure

Splines represent a smooth curve using connected basis functions. They can model nonlinear relationships without one high-degree global polynomial controlling the entire shape.

Choices about knots, smoothness and degrees of freedom still matter. A flexible curve can reveal structure or chase noise depending on data volume and regularisation.

13. Interactions mean one predictor’s association depends on another

An interaction model might include X, Z and X×Z. The coefficient on X then describes the X slope when Z equals zero. As Z changes, the X slope changes.

Suppose tutoring hours matter differently for learners with different baseline scores. An interaction can represent that heterogeneity. Reporting only the interaction p-value hides the shape. Plot predicted outcomes across meaningful values of both variables.

14. Main effects remain in interaction models for a reason

Dropping main effects while retaining an interaction imposes special constraints on the model. Sometimes such constraints are justified. Often they are accidental.

Use the hierarchy principle unless a substantive model says otherwise: if an interaction is present, retain the lower-order components needed to interpret it.

15. Residuals are what the model failed to predict

A residual is observed minus fitted outcome. Residuals are not the true errors in a philosophical sense; they are estimates of unexplained deviations under the fitted model.

The NIST model discussion describes fitted values and residuals as central to judging whether a model approximates observed responses usefully.

16. Residual plots can expose wrong functional form

If residuals form a curved pattern against fitted values, the mean structure may be misspecified. If their spread increases with the fitted value, constant variance may be implausible. If residuals move in runs over time, independence may be doubtful.

A random-looking residual cloud does not prove the model true. It means some obvious forms of misspecification are not visible in that diagnostic.

17. Heteroskedasticity changes uncertainty more than the fitted mean

Ordinary least squares can still estimate the best linear projection under broad conditions when variance is not constant, but conventional standard-error formulas may be wrong. Robust standard errors or a different variance model may be needed.

NIST’s discussion of non-constant variation shows why residual structure should be inspected and why variance patterns can be confused with a poor mean function.

18. Weighted least squares changes influence intentionally

When observations have known or well-modelled differences in precision, weighted least squares can assign greater influence to more precise observations. NIST provides a dedicated weighted least squares discussion.

Weights are not decoration. They redefine the objective function. Survey weights, inverse-variance weights and causal inverse-probability weights arise from different problems and should not be interchanged merely because software accepts a weight argument.

19. Independence is often a design assumption, not a residual-plot detail

Learners in the same class, repeated measurements from one person and observations along a time series are correlated by design. A residual plot may hint at dependence, but the sampling structure already tells us that ordinary independent-error standard errors can be inappropriate.

Use Clustered and Multilevel Data for hierarchical dependence and Forecasting and Prediction for time-dependent prediction.

20. Normality is often about errors or sampling distributions, not predictors

A common misconception says every variable in a linear regression must be normally distributed. That is not the usual assumption.

Classical small-sample exact inference often assumes normally distributed errors conditional on predictors. Predictors themselves can be skewed or binary. Large-sample robust inference can weaken some distributional requirements.

Always ask which quantity the assumption concerns.

21. Outliers are not a category of people to delete

An outlying observation may be a data error, a rare but real case, evidence of a different process, or a point the model fits poorly. Deleting it simply because it changes the coefficient is not a defensible rule.

Investigate provenance, measurement validity and influence. Report sensitivity to defensible inclusion rules. Preserve the original record unless there is a justified correction.

22. Leverage and influence are different

A high-leverage observation has unusual predictor values. An influential observation materially changes the fitted model when included or excluded. A point can have high leverage without large residual if it sits close to the fitted relationship.

Influence diagnostics identify cases worth investigation. They do not issue deletion orders.

23. Multicollinearity makes individual coefficients unstable

If two predictors carry nearly the same information, the model can struggle to separate their conditional coefficients. The overall predictions may remain stable while individual coefficients vary sharply across samples.

This is not necessarily a data-quality error. It can reflect the scientific structure: the predictors rarely vary independently in the observed population.

Do not interpret a large standard error as proof that the predictor is unimportant. The model may lack information to distinguish its unique conditional association from related predictors.

24. R-squared is not a universal measure of model worth

R-squared measures the fraction of observed variance explained by the fitted linear model in the sample under the usual definition. A high R-squared does not establish causality, calibration, fairness or future predictive accuracy.

A low R-squared can coexist with a precisely estimated average association. A high R-squared can arise from strong time trends in two unrelated processes. Judge the model against its job.

25. Adding predictors almost never decreases ordinary training R-squared

Because least squares is allowed to choose new coefficients, extra terms cannot make the training residual sum of squares worse. That means raw R-squared rewards complexity.

NIST notes a related point in its model-selection discussion: residual fit alone is insufficient because adding terms tends to improve fit. Parsimony and validation matter.

26. Adjusted R-squared is only one complexity correction

Adjusted R-squared penalises additional parameters in a particular way. Information criteria such as AIC and BIC use different penalty structures. Cross-validation evaluates out-of-sample performance more directly for predictive purposes.

No single metric should be treated as the universal model-selection oracle.

27. Regression for prediction and regression for explanation are different jobs

A prediction model is judged by performance on relevant future or held-out data: calibration, error, discrimination and decision utility. A scientific explanatory model may care more about interpretable parameters and a defensible identification structure.

A variable can improve prediction while being a poor target for intervention. Removing it for causal purity can make prediction worse. Including it for prediction can make a coefficient difficult to interpret causally.

28. Overfitting happens when the model learns sample-specific noise

A flexible model can fit training data extraordinarily well and perform poorly on new observations. Overfitting grows more likely as model flexibility rises relative to available information.

Hold-out evaluation, cross-validation, shrinkage and external validation are responses to this problem. The canonical broader route is How Forecasting and Prediction Work.

29. Regularisation trades fit for stability

Ridge regression shrinks coefficients through an L2 penalty. Lasso uses an L1 penalty and can set some coefficients to zero. Elastic net combines both.

These methods can improve prediction or coefficient stability in high-dimensional settings. Their penalised coefficients answer a different estimation problem from ordinary least squares and should not be interpreted as though no shrinkage occurred.

30. Logistic regression models a transformed probability

When the outcome is binary, ordinary linear regression can predict values below zero or above one and impose constant effects on the probability scale. Logistic regression models the log odds as a linear function of predictors:

log[p/(1−p)] = β0 + β1X + …

Exponentiating a coefficient gives an odds ratio for a one-unit predictor difference conditional on the model. Odds ratios are not the same as risk ratios, and a constant odds ratio implies different absolute probability changes at different baselines.

The official R documentation for generalized linear models provides a computational reference for the GLM framework.

31. Generalised linear models change the response distribution and link

The GLM framework extends linear modelling to outcomes such as counts and binary responses by combining a probability distribution from the exponential family with a link function relating the mean to the linear predictor.

Poisson regression often models counts through a log link. Binomial logistic regression models probabilities through log odds. The coefficient scale therefore changes with the link.

32. Offsets let a count model represent rates

Suppose one library branch records 100 enquiries over 1,000 visitor-hours and another records 80 over 400 visitor-hours. Comparing raw counts confounds volume with exposure time.

A count model can include log exposure as an offset so the model concerns a rate. The offset coefficient is fixed rather than estimated.

33. Zero inflation is not solved by naming a model after the symptom

Count data can contain more zeros than a basic Poisson model predicts. That might reflect a distinct structural-zero process, overdispersion, heterogeneity or a poor mean specification.

Use a zero-inflated model only when its data-generating story is credible. A complicated distribution does not substitute for understanding why zeros occur.

34. Time-to-event regression belongs to a different owner

Cox regression relates covariates to hazards and relies on risk sets and censoring assumptions. Its coefficients do not have the same interpretation as ordinary linear slopes.

Use How Time-to-Event and Survival Analysis Work for the full survival-analysis framework.

35. Multilevel regression belongs to a different dependence structure

Random-intercept and random-slope models can represent grouped observations and varying relationships across clusters. Their coefficients and variance components require hierarchical interpretation.

Use How Clustered and Multilevel Data Work for the canonical treatment.

36. Measurement error can attenuate or distort coefficients

Classical random error in a continuous predictor can bias a simple slope toward zero under specific conditions. Misclassification and differential measurement error can create more complex bias.

The regression does not know that a recorded predictor is noisy unless the model tells it. See Measurement Error and Misclassification.

37. Missing data can change the fitted population

Software often performs complete-case regression by silently dropping any row missing a required variable. The resulting coefficient can describe a selected subgroup rather than the target population.

Inspect the number and characteristics of excluded observations. For model-based approaches, route to How Missing Data Analysis Works.

38. Selection on the outcome can create distorted models

If the dataset contains only learners who passed an entry threshold, relationships among predictors can differ from those in the full applicant population. This is range restriction and selection, not merely small sample size.

Regression estimates are always conditional on which observations entered the analysis.

39. Extrapolation leaves the support of the data

A fitted line through ages 12 to 16 can generate a prediction at age 40. Mathematics permits it. Evidence may not.

Outside the observed predictor range, the functional form is doing more work than the data. Report extrapolation as such and test whether the domain mechanism supports it.

40. Simpson’s paradox is a reminder about conditioning

An aggregate association can reverse within groups. This can happen because groups differ in both predictor distribution and outcome levels.

Regression adjustment can reveal within-group relationships, but the choice of grouping variable must be substantively justified. A reversal is not automatically proof that the adjusted coefficient is causal.

41. Coefficients can change when you condition on post-treatment variables

Suppose a teaching intervention changes attendance, and attendance changes scores. A regression adjusting for attendance estimates a different contrast from one that does not adjust for it.

Neither coefficient should be called “the effect” without specifying which effect and which causal assumptions are intended.

42. Prediction intervals and confidence intervals answer different questions

A confidence interval around the expected mean at a predictor value concerns uncertainty about the mean relationship. A prediction interval for a new individual includes both uncertainty in the mean and individual residual variation, so it is wider.

Do not give a narrow mean interval to someone asking how uncertain one person’s outcome is.

43. Calibration matters even when discrimination is strong

A binary prediction model can rank high-risk cases well while systematically overestimating probabilities. Discrimination and calibration are different performance properties.

If decisions use absolute probabilities, calibration can be decisive.

44. Cross-validation estimates a pipeline, not an abstract model family

If feature selection, scaling or imputation is performed before the cross-validation split, information can leak from validation folds into training. Performance then looks better than the deployable pipeline.

All data-dependent preprocessing should occur within the resampling loop when evaluating a future-facing pipeline.

45. Regression tables compress more than they reveal

A table of coefficients can hide nonlinearity, interactions, data sparsity and distributional assumptions. Plot the data and fitted relationships where feasible.

For nonlinear link functions, marginal effects or predicted probabilities at meaningful covariate values can be more interpretable than raw coefficients alone.

46. A coefficient’s sign can be stable while its magnitude is fragile

Run sensitivity analyses over plausible functional forms, outlier rules, missing-data methods and adjustment sets. A positive coefficient in every model does not mean every substantive conclusion is robust.

The magnitude may range from trivial to important. The prediction may remain stable while interpretation changes. Route to Sensitivity Analysis and Robustness Checks.

47. Model selection after looking at the outcome changes inference

If dozens of models are tried and only the most favourable one is reported, conventional standard errors and p-values no longer describe the full selection process.

Pre-specification, penalisation, cross-validation, model averaging or post-selection methods can address parts of this problem. The right method depends on whether the goal is confirmatory inference or prediction.

48. A causal regression needs a design story

Before interpreting β1 as causal, explain why the comparison represented by β1 approximates the intervention contrast of interest. What confounding is blocked? What variables must not be conditioned on? Is there overlap? Is the treatment version well defined? Does interference matter?

The regression equation is the computational layer. The causal identification layer sits above it.

49. An instrumental-variable regression has a different estimand

Instrumental-variable methods use variation in an instrument under strong assumptions about relevance, exclusion and independence. The resulting estimand can be local to particular compliance behaviour rather than a universal average effect.

Do not treat two-stage least squares as ordinary regression with an extra predictor. Its interpretation comes from the identification assumptions.

50. Regression discontinuity uses local comparison near a threshold

When treatment changes sharply at a threshold in a running variable, regression can estimate a local discontinuity under continuity assumptions. Polynomial choice, bandwidth and manipulation near the cutoff are central.

The design is the threshold assignment mechanism, not the regression syntax.

51. Difference-in-differences is a contrast of changes

A regression with group, time and interaction terms can represent a difference-in-differences contrast. Its causal interpretation depends on a parallel-trends style assumption and treatment-timing structure.

Again, the same regression engine can implement a specialised design whose meaning comes from assumptions outside the coefficient table.

52. Good regression work preserves provenance

A reproducible model should record the dataset edition, variable definitions, transformations, exclusions, missing-data method, formula, software version, weights, variance estimator and validation procedure.

Without that record, a coefficient can become detached from the exact model that produced it.

53. A worked interpretation example

Suppose a fictional regression predicts score from practice hours and baseline score:

Predicted score = 20 + 1.5 × practice hours + 0.7 × baseline score.

Within this model, two learners with the same baseline score but a one-hour difference in practice are predicted to differ by 1.5 points. That is a conditional association. It does not prove that forcing one additional hour of practice would cause a 1.5-point gain.

If the relationship curves after five hours, the linear coefficient averages over that curvature. If practice is measured with error, the coefficient may be biased. If high-motivation learners both practise more and improve more, confounding remains.

The equation is useful because it makes a specific comparison. The surrounding design determines what that comparison can mean.

54. A regression audit should work backwards

CLAIM
→ COEFFICIENT / PREDICTION
→ MODEL FORMULA
→ ADJUSTMENT SET
→ OUTCOME + PREDICTOR DEFINITIONS
→ MISSINGNESS / WEIGHTS / EXCLUSIONS
→ SAMPLING DESIGN
→ DATA SOURCE
→ WORLD

If the reverse path cannot be reconstructed, the model may still produce numbers but is weak as accountable evidence.

55. A practical checklist before fitting

1. State whether the job is description, explanation, prediction or causal estimation.
2. Define outcome and predictor units.
3. Inspect distributions and impossible values.
4. Map the sampling and dependence structure.
5. Decide which nonlinearities and interactions are plausible.
6. For causal work, define the identification assumptions before selecting covariates.
7. Plan missing-data handling.
8. Plan how model performance and assumptions will be checked.
9. Reserve genuinely external data for validation where possible.
10. Document the model before examining the final outcome if the analysis is confirmatory.

56. A practical checklist after fitting

1. Inspect residuals or appropriate diagnostics.
2. Check influential observations and provenance.
3. Check whether predictions stay inside the support of the data.
4. Report estimates with uncertainty, not coefficients alone.
5. Translate nonlinear models to meaningful predicted values.
6. Test sensitivity to defensible model choices.
7. Separate prediction claims from causal claims.
8. Validate on relevant new data where prediction is the goal.
9. Preserve code, formula, data edition and exclusions.
10. State what the model cannot answer.

57. The Library ownership boundary

This article owns general regression logic: conditional models, coefficients, residuals, functional form, diagnostics and interpretation. It crosswalks rather than replaces adjacent owners.

• Statistical Inference and Uncertainty owns interval and hypothesis reasoning.
• Observational Studies owns cohort, cross-sectional and observational design logic.
• Causal Inference owns identification.
• Forecasting and Prediction owns predictive validation and deployment.
• Clustered and Multilevel Data owns hierarchical regression.
• Measurement Error and Misclassification owns imperfect variables.

58. What a learner should remember

A regression coefficient is not a self-interpreting fact. Ask what outcome was modelled, which predictors were included, what functional form was assumed, which observations entered, how uncertainty was calculated and what design makes the intended interpretation credible.

The fitted line is a representation. Its value lies in making a relationship inspectable without forgetting that the line was chosen, estimated and bounded by data.

Sources and further reading

• NIST Engineering Statistics Handbook — Least Squares.
• NIST — Model Fitting: Initial Model.
• NIST — Detecting Non-Constant Variation.
• NIST — Weighted Least Squares Regression.
• NIST — What Is a Model?.
• R Project documentation — Generalized Linear Models.

Continue through eduKate: Data Visualisation → Statistical Inference and Uncertainty → Causal Inference → Forecasting and Prediction → Research Collections Directory.