Model Selection, Regularisation and Cross-Validation

A model fits the training data beautifully. Its error is tiny. The coefficients look convincing. The feature engineering was clever. Then the model meets new data and performs badly.

The failure may not be in the algorithm. It may be in the evaluation design. The analyst used the same evidence to choose the model and to judge it, allowed information to leak from held-out records into preprocessing, tried many alternatives and reported only the winner, or split dependent data as though every row were independent.

Model selection is the process of choosing among candidate models or hyperparameter settings for a defined objective. Cross-validation estimates performance by repeatedly fitting a model on one subset and evaluating it on held-out data under a specified splitting scheme. Regularisation changes the fitting objective to discourage overly flexible solutions, often trading some fit to the observed sample for greater stability or generalisation.

This article owns the workflow that connects fitting to honest evaluation. How Regression Models Work owns regression interpretation and diagnostics. How Forecasting and Prediction Work owns the broader predictive job through time. How Reproducibility and Replication Work owns repeatable research and independent evidence.

Reading route: begin with overfitting, build the training-validation-test split, learn cross-validation, protect against leakage, choose the right splitting scheme, then study ridge, lasso and elastic net, nested selection, and what a performance estimate can legitimately claim.

Overfitting is successful memorisation of the wrong target

A flexible model can follow real signal and random peculiarities of the training sample. Training error then falls while performance on new observations stops improving or deteriorates. The model has learned details that do not transport.

Overfitting is not limited to enormous machine-learning systems. Choosing a polynomial degree after looking repeatedly at one small dataset, adding variables until a p-value crosses a threshold, trying many transformations and keeping the best, or manually adjusting a forecasting rule until it matches recent history are all forms of adaptive fitting.

Scikit-learn’s current cross-validation documentation states the core problem directly: testing a predictor on the same data used to learn it can produce excellent apparent performance and poor prediction on unseen data.

Generalisation is always relative to a data-generating target

“Works on new data” is incomplete. New data from the same people at a later time? New people from the same institution? New institutions? A changed policy environment? A future regime after a technological shift?

A validation scheme can estimate performance only for a target it reasonably mimics. Randomly splitting yesterday’s data may be appropriate for exchangeable future cases and misleading for forecasting next year after a trend. Leave-one-student-out validation asks a different question from randomly splitting rows when each student contributes many rows.

How External Validity and Evidence Transfer Work owns the broader transfer problem. The model-selection lesson is to define the future or target use before choosing the split.

Training, validation and test data have different jobs

The training set fits model parameters. A validation process compares candidate models or hyperparameters. The test set estimates final performance after the modelling decisions are frozen.

If the test set influences feature engineering, model family, threshold choice or hyperparameter selection, it has become part of the development process. It is no longer an untouched final test in the strict sense.

This is why a model can “overfit the test set” without ever using test labels in gradient descent. Repeated human inspection of test results feeds information back into the modelling choices. The whole team, not only the optimiser, can overfit.

A simple holdout is useful but can be unstable

Split a dataset once into training and validation subsets, fit on the first and evaluate on the second. This protects the validation cases from direct fitting and is conceptually clean.

But one split can be lucky or unlucky. In small datasets, the chosen validation subset may contain an unusual case mix. Using a large validation fraction also leaves fewer observations for fitting.

Cross-validation reuses observations across different training folds while keeping each evaluation fold held out from the model fitted for that fold. It reduces dependence on one arbitrary split, although it introduces its own design choices and dependence among fold estimates.

K-fold cross-validation rotates the holdout role

In k-fold cross-validation, the development data are partitioned into k folds. For each fold, fit the model on the other k−1 folds and evaluate it on the held-out fold. Combine the fold-level scores to summarise performance.

Five-fold and ten-fold cross-validation are common choices, not universal laws. Larger k trains on a larger fraction of the data in each iteration but increases computation and changes the bias-variance behaviour of the estimate. Leave-one-out cross-validation is the limiting case with one observation per validation fold.

Scikit-learn’s stable documentation describes k-fold, repeated k-fold, leave-one-out, grouped and time-aware strategies and warns that the splitting rule should reflect the data-generating structure.

Fold scores are not independent replications

The fitted models in k-fold cross-validation share most of their training data, so their scores are statistically dependent. Treating five fold scores as five independent studies exaggerates the amount of evidence.

The standard deviation across folds can be descriptively useful for seeing instability across splits. It is not automatically a standard error for the true generalisation performance. Formal uncertainty for cross-validation performance is more subtle than applying an ordinary t-test to k numbers.

This distinction matters especially when comparing models whose score differences are small relative to split-to-split variation.

Repeated cross-validation studies split sensitivity, not independent populations

Repeating k-fold cross-validation with several random partitions can reveal whether conclusions depend on the particular fold assignment. That is valuable robustness information.

It still uses the same underlying observations. One hundred repetitions do not create one hundred independent datasets. The repeated results estimate behaviour under repeated partitioning of the available sample, not replication across new populations.

Data leakage occurs when held-out information enters the learning process

Suppose every feature is standardised using the mean and standard deviation computed from the full dataset before cross-validation. The held-out fold influenced the transformation applied to the training fold. That influence may be small in one problem and consequential in another, but the evaluation is no longer clean.

The same problem occurs with feature selection, PCA, imputation, target encoding, outlier thresholds and any learned preprocessing. Fit these steps only on the training portion within each fold, then apply the fitted transformation to the held-out fold.

Scikit-learn’s cross-validation guide explicitly recommends learning preprocessing from training data and using pipelines to keep transformations inside the validation workflow.

Target leakage is more serious than a technical preprocessing mistake

A feature may contain information that would not exist at the time a prediction is made. A hospital outcome code recorded after discharge, a final exam component used to predict the earlier examination, or a manually corrected value created with knowledge of the outcome can make a model look extraordinarily accurate.

Random cross-validation does not detect this if the leaked feature appears in every fold. The information boundary must be defined from the real decision timeline.

A good feature table therefore includes provenance and availability time, not merely column names. Metadata and Data Lineage owns the infrastructure required to preserve that boundary.

Grouped data require grouped validation

Suppose each learner contributes twenty records. If rows are randomly divided across folds, the model may train on nineteen records from a learner and validate on the twentieth. The score then estimates performance on another record from a partly known learner, not performance on a genuinely new learner.

If deployment concerns new learners, all records from one learner should remain in the same fold. If the target is a new school, the grouping unit may need to be the school. The split should remove the entity whose novelty defines the future use.

How Clustered and Multilevel Data Work owns the statistical dependence; grouped validation is the model-evaluation consequence.

Time-series validation must respect the arrow of time

A forecasting model cannot train on December and validate on March of the same year if the real task is to predict future months from past data. Random shuffling lets future information influence training.

Rolling-origin or expanding-window validation fits on past observations and tests on later observations. Multiple cut points can show how performance varies through time.

Even time-aware validation can be optimistic if the future regime changes more than the historical period. Forecasting and Prediction owns regime shifts, calibration and backtesting in greater depth.

Stratification changes fold composition, not the target distribution

For classification, stratified folds preserve approximately similar class proportions across folds. This can reduce instability when classes are imbalanced and ensure that each fold contains examples of each class where possible.

Stratification does not solve target-population imbalance or class-prevalence shift. It is a partitioning strategy for evaluation, not a guarantee that deployment will have the same prevalence.

The metric is part of model selection

The model declared “best” depends on what score is optimised. Accuracy, log loss, mean absolute error, mean squared error, ranking metrics, calibration error and utility-based scores reward different behaviour.

If false negatives are far more costly than false positives, selecting by overall accuracy can optimise the wrong decision. If probability estimates will drive later decisions, calibration may matter even when classification accuracy is similar.

Choose the metric before inspecting which model it favours. Otherwise the metric itself becomes another adaptively selected hyperparameter.

Hyperparameters are learned by a higher-level search

Model parameters are fitted from training data inside a model, such as regression coefficients. Hyperparameters control the fitting process or model structure, such as regularisation strength, tree depth, number of neighbours or kernel settings.

Grid search, random search and Bayesian optimisation are strategies for exploring hyperparameter settings. Whichever search is used, the evaluation data used to choose settings become part of the training-development process.

The more alternatives explored, the greater the opportunity to select a model that benefited from random validation noise. This is model-selection bias.

The best validation score is optimistically selected

Suppose twenty models have identical true performance but noisy validation scores. The model with the highest observed score will tend to be one that received a favourable noise realisation. Its winning validation score therefore overstates its expected future performance.

This is the same selection principle behind winner’s curse phenomena elsewhere in statistics. Choosing the maximum of noisy estimates changes the distribution of the selected estimate.

An untouched test set or nested validation can provide a less biased estimate after selection.

Nested cross-validation separates model selection from performance estimation

Nested cross-validation creates an outer loop for performance estimation and an inner loop for selecting hyperparameters or models. For each outer training split, the inner process chooses the model using only that outer training data. The selected model is then evaluated on the outer held-out fold.

This recreates the full development pipeline inside each outer training set. The outer scores therefore evaluate the procedure “select a model using this inner process, then predict new data,” rather than evaluating one hyperparameter setting chosen with access to all folds.

Nested validation is computationally more expensive and still depends on the split design representing the target use. It is not a universal requirement for every low-stakes model, but it is a strong tool when model-selection bias matters.

Regularisation changes the objective, not merely the evaluation

Ordinary least squares minimises residual sum of squares. Regularised regression adds a penalty to the coefficient vector. The penalty discourages certain complex solutions and can improve stability when predictors are numerous or strongly correlated.

Scikit-learn’s current linear-model documentation describes ridge, lasso and elastic-net estimators and their regularisation parameters.

Ridge regression shrinks coefficients through an L2 penalty

Ridge regression adds a penalty proportional to the sum of squared coefficient magnitudes. In one common scaling, the objective is residual loss plus α||w||²₂. Larger α produces stronger shrinkage.

Ridge is especially useful when predictors are correlated. Ordinary least squares can produce large unstable coefficients that offset one another. Ridge accepts some bias to reduce variance and often distributes influence across correlated predictors rather than selecting one arbitrarily.

Its coefficients generally shrink toward zero without becoming exactly zero. The model remains dense unless another structure removes variables.

Lasso uses an L1 penalty and can set coefficients exactly to zero

The lasso adds a penalty proportional to the sum of absolute coefficient magnitudes. This geometry can drive some fitted coefficients to exactly zero, producing sparse solutions.

Sparsity can aid prediction, storage and interpretation when the underlying structure is suitably sparse. It should not be mistaken for proof that zeroed variables are scientifically irrelevant. With correlated predictors, the selected variable can be unstable across samples.

The stable scikit-learn Lasso documentation defines the current optimisation objective and explains that the regularisation strength is controlled by alpha.

Elastic net combines L1 and L2 penalties

Elastic net blends lasso-style sparsity with ridge-style shrinkage. It can be useful when groups of correlated features contain related information and pure lasso selection is unstable.

The mixing ratio and overall penalty strength are hyperparameters and therefore belong inside the selection process. If both are tuned on the same validation data used for final performance reporting, the reported score inherits selection optimism.

Regularisation strength is meaningful only relative to feature scaling

Penalties act on numerical coefficient magnitudes. If one feature is measured in thousands and another in fractions, equal coefficient penalties do not imply equal treatment of the underlying variables.

Standardising predictors is therefore common before penalised regression. The standardisation must be learned inside each training fold to avoid leakage. If original units are scientifically meaningful, interpret the fitted model carefully after any scaling transformation.

Regularisation is a bias-variance trade, not a free accuracy machine

Strong regularisation intentionally moves coefficients away from the unpenalised best fit to the training data. This introduces bias. The expected gain is reduced variance, improved numerical stability or lower out-of-sample error.

If the true relationship is simple and the sample large, heavy shrinkage can hurt performance. If the data are noisy and predictors numerous, an unregularised model can overfit badly. The appropriate strength is an empirical and decision-specific question.

Feature selection performed before cross-validation leaks selection information

Suppose ten thousand features are screened using their association with the outcome on the full dataset, the best hundred are retained, and only then cross-validation begins. Each validation fold helped decide which features were available. The score is optimistic.

Feature screening, lasso fitting, PCA and any other outcome- or data-dependent selection must occur within each training fold when the goal is to estimate performance of the whole procedure.

Imputation belongs inside the validation pipeline

Filling missing values using means calculated across the full dataset leaks validation information. More sophisticated imputation models can leak even more structure.

Fit the imputation model using the training subset and apply it to the validation subset. When missingness itself changes over time or across populations, even a correctly nested imputer may fail to transport. How Missing Data Analysis Works owns the missingness assumptions.

Threshold tuning is model selection too

A probabilistic classifier may output a risk score. Choosing the threshold that converts that score into an action can materially affect false positives, false negatives and utility.

If a threshold is chosen by inspecting the test set, the test set has participated in optimisation. Threshold selection belongs inside the validation process, ideally using a utility or constraint that reflects the actual decision.

Model selection for explanation and model selection for prediction are not identical

A prediction model is judged by performance on relevant unseen data. An explanatory model may prioritise interpretable parameters, causal identification, uncertainty and a prespecified estimand. The model with the lowest cross-validated prediction error is not automatically the model with the clearest causal interpretation.

Penalised coefficients can be excellent for prediction and difficult to interpret as unbiased estimates of causal effects. Variable selection changes the inferential problem. Causal Inference remains the canonical owner for causal claims.

Information criteria and cross-validation approximate different targets

AIC, BIC and related criteria combine fit with complexity penalties derived from different theoretical motivations. Cross-validation estimates predictive performance under a resampling design. They can choose different models because they are not identical procedures.

An analyst should not run several criteria and report whichever selects the preferred model without acknowledging the multiplicity. Choose the criterion to match the inferential or predictive job.

The one-standard-error idea encodes a preference for simpler near-winners

When cross-validation scores for several complexity levels are similar, a common heuristic chooses the simplest model whose estimated score is within roughly one standard-error-type band of the best. The exact implementation and uncertainty estimate depend on the method.

The principle is useful: do not assume a tiny validation advantage justifies substantial complexity. But simplicity is a preference or operational advantage, not proof that the simpler model is true.

A final model is usually refitted after selection

Once a model family and hyperparameters are chosen using the development data, it is common to refit the selected model on all available development observations before deployment. This uses more information to estimate the final parameters.

The untouched test set can then provide one final estimate if it was genuinely kept out of development. After inspecting that test performance, future redesign should be treated as a new development cycle with a new evaluation plan rather than repeatedly mining the same test set.

Cross-validation estimates a procedure, not an eternal property of an algorithm

A reported score depends on the data distribution, preprocessing, feature set, split strategy, metric, hyperparameter search and random state. Change any of these and the target procedure changes.

“Algorithm X has 92% accuracy” is therefore usually too vague. A stronger statement names the dataset population, evaluation design, metric and uncertainty, and separates internal validation from external validation.

Internal cross-validation is not external validation

Cross-validation partitions one dataset. If all folds come from the same institutions, devices and period, the procedure estimates performance under that source environment. It does not automatically establish performance in another country, school system, hospital, sensor generation or future regime.

External validation uses genuinely separate data representing the target domain. It can reveal failures caused by prevalence shift, measurement differences, new workflows or changed relationships that internal resampling cannot see.

External Validity and Evidence Transfer owns that next bridge.

High average performance can conceal subgroup failure

A model can perform well overall and poorly for a smaller group. Random folds that preserve the same population mixture reproduce the average and may hide consequential minority errors.

Evaluate relevant subgroups where sample size and governance permit, but do not create an endless fishing expedition of unplanned slices. Define groups based on deployment relevance and preserve uncertainty. Fair comparison also depends on whether outcomes and measurements have equivalent meaning across groups.

Measurement Invariance and Fair Comparisons provides the measurement route.

Calibration and discrimination answer different predictive questions

A risk model can rank cases well while its predicted probabilities are systematically too high or too low. Conversely, a model can be well calibrated on average and poor at ranking.

If the probabilities will drive decisions, evaluate probability quality, not only classification accuracy. Recalibration may be necessary after deployment when prevalence or measurement changes.

Performance drift is a post-deployment model-selection problem

Once deployed, a model enters a changing world. Input distributions shift, behaviours adapt, policies change and upstream systems are replaced. A model that passed cross-validation in 2026 can fail later without any software bug.

Monitor inputs, predictions, outcomes where available, calibration and operational consequences. Define review triggers before the model degrades enough to surprise users.

This connects to Data Observability and Monitoring and Models and Simulations.

Reproducibility requires preserving the selection path

To reproduce a selected model, preserve the data edition, feature definitions, preprocessing steps, fold assignments or split-generation rules, random seeds where relevant, software versions, candidate search space, metric, chosen hyperparameters and final refit procedure.

Recording only the final coefficients loses the path that determined them. A future analyst cannot tell whether the model was one prespecified candidate or the survivor of thousands of adaptive searches.

A worked conceptual pipeline

DEFINE TARGET USE
→ FREEZE OUTCOME AND METRIC
→ DEFINE INFORMATION AVAILABLE AT PREDICTION TIME
→ CHOOSE SPLIT UNIT AND TIME RULE
→ HOLD OUT FINAL TEST DATA IF NEEDED
→ INSIDE EACH TRAINING FOLD:
     FIT IMPUTATION
     FIT SCALING
     FIT FEATURE REDUCTION / SELECTION
     FIT CANDIDATE MODEL
→ SCORE HELD-OUT FOLD
→ SELECT HYPERPARAMETERS
→ IF PERFORMANCE ESTIMATION AFTER SELECTION MATTERS:
     USE OUTER/NESTED EVALUATION OR UNTOUCHED TEST SET
→ REFIT FINAL PIPELINE ON DEVELOPMENT DATA
→ VERIFY EXTERNAL / TEMPORAL TARGET WHERE POSSIBLE
→ DEPLOY WITH MONITORING AND REVIEW TRIGGERS

The pipeline makes one principle visible: evaluation is part of model construction. It cannot be bolted on after every modelling choice has already seen the answers.

A practical release checklist

State the deployment population, unit and time horizon.
Define the performance metric and decision threshold before comparing candidates.
Protect a final test set when the consequence justifies one.
Choose random, grouped, stratified or time-aware splits to match the target use.
Fit every learned preprocessing step inside training folds.
Audit feature availability to prevent target and future-information leakage.
Place feature selection and dimensionality reduction inside the validation pipeline.
Use nested validation or a clean test set when extensive model selection makes optimism consequential.
Standardise features appropriately before scale-sensitive regularisation.
Tune ridge, lasso or elastic-net penalties using development data only.
Report the search space, not only the winner.
Do not treat fold scores as independent replications.
Check subgroup, temporal and external performance relevant to deployment.
Preserve the complete pipeline and software/data versions.
Monitor post-deployment drift and define a revalidation trigger.

A learner’s mental model: the test set is evidence, not a scoreboard

The most important habit is to stop seeing validation as a competition in which models repeatedly peek at a scoreboard. Held-out data are evidence about performance the model has not yet had a chance to exploit.

Every time the result changes the model, that evidence has been consumed for development. This is not forbidden. It simply changes the role of the data. Honest evaluation requires fresh evidence or a design that accounts for the selection process.

Regularisation then becomes easier to understand. It is not magic complexity tax. It is one way to encode a preference for more stable fitted functions. Cross-validation is not magic truth detector. It is one way to simulate held-out prediction under a chosen partition. Model selection is the layer that connects them—and therefore the layer where discipline matters most.

Sources and further reading

Source documentation was checked for this edition on 5 September 2026. Scikit-learn’s stable documentation surfaced as version 1.9.0 during the check. This article explains principles; production model validation should follow the assumptions, governance and consequences of the specific domain.

Continue through eduKate: read Regression Models, Forecasting and Prediction, PCA and Factor Analysis, External Validity and Evidence Transfer, and return through the Research Collections Directory.

How Model Selection, Regularisation and Cross-Validation Work | Overfitting, Leakage, Ridge, Lasso, Elastic Net and Honest Generalisation