A model can achieve excellent performance on examples it has already seen and still fail on new ones. The central purpose of dataset splitting is to separate learning from evaluation so that we can answer the question that actually matters:
How well will this model perform on future data drawn from the target population?
The familiar train, validation, and test split is not merely an organizational convention. Each subset has a different statistical role. When those roles are mixed, information leaks into the learning process and the reported result becomes optimistic.
Suppose we begin with a dataset $D$ sampled from some population. We partition it into three disjoint subsets:
\[D = D_{\text{train}} \cup D_{\text{val}} \cup D_{\text{test}}.\]| Subset | Primary role | What may depend on it? |
|---|---|---|
| Training set | Fit model parameters | Weights, coefficients, learned representations, and fitted preprocessing |
| Validation set | Make development decisions | Architecture, hyperparameters, thresholds, feature choices, and stopping time |
| Test set | Estimate final generalization | Ideally nothing; it is used only after the model and evaluation procedure are fixed |
The key distinction is between model parameters and development choices. Gradient descent may fit millions of weights using the training set, while the validation set determines choices such as depth, learning rate, regularization strength, data augmentation, or the epoch at which training stops. Both are forms of learning from data.
The test set is different. It simulates data that remained unavailable throughout development.
A sufficiently flexible model can memorize its training examples. Its empirical training error,
\[\widehat R_{\text{train}}(h) = \frac{1}{|D_{\text{train}}|} \sum_{(x_i,y_i)\in D_{\text{train}}} \ell\!\left(h(x_i),y_i\right),\]measures fit to the sample used for learning. It does not provide an independent estimate of the population risk
\[R(h)=\mathbb{E}_{(X,Y)\sim P}\left[\ell(h(X),Y)\right].\]The gap between these quantities is the generalization gap. A validation or test set approximates population risk only when it has not already influenced the predictor in the way it is being evaluated.
Although no gradient may be computed directly from the validation set, repeated validation feedback guides development. Consider a workflow that tries 100 model configurations and keeps the one with the highest validation accuracy. The selected model benefits from information in that validation set: configurations that happened to match its noise were more likely to win.
This is overfitting to the validation set. It becomes more severe when:
Validation data are therefore development data. They should not be treated as an untouched estimate of final performance.
The test set is intended to answer one final question after the model, preprocessing, decision threshold, and metric have been fixed. If its results influence another round of development, it has begun to act as a validation set.
This does not mean a test set can literally be evaluated only once. The principle is that it should not provide an adaptive feedback loop. If we inspect the test result, change the system, and inspect it again many times, the final system can overfit the test set even without directly training on its examples.
The same problem occurs at the community level. A public benchmark can become a shared validation set after years of model selection, paper writing, and leaderboard tuning. Honest progress then requires fresh test data or a hidden evaluation server.
Rules such as 70/15/15 or 80/10/10 are starting points, not laws. The right split is determined by the number of examples required for two competing goals:
With millions of independent examples, even 1% may produce a large evaluation set. With a small dataset, fixed holdout sets may be too noisy, making cross-validation more useful.
For a fixed classifier evaluated with 0–1 loss, the empirical test error $\hat p$ is the proportion of misclassified examples:
\[\hat p = \frac{1}{N}\sum_{i=1}^{N} \mathbf{1}\!\left[h(x_i)\neq y_i\right].\]Each test outcome is therefore treated as a Bernoulli variable: 1 when the prediction is wrong and 0 when it is correct. On $N$ approximately independent test examples, the estimated standard error of this proportion is
\[\operatorname{SE}(\hat p) \approx \sqrt{\frac{\hat p(1-\hat p)}{N}}.\]This standard error measures finite-test-set sampling uncertainty. It describes how much the measured error would typically vary if we evaluated the same fixed classifier on another independent test set of size $N$ from the same distribution. A rough 95% confidence interval is $\hat p\pm1.96\operatorname{SE}(\hat p)$.
It does not measure variation caused by retraining on different training sets, uncertainty in an individual prediction, or distribution shift. When examples are grouped or correlated, the effective sample size can be far smaller than $N$, so a group-aware confidence interval or bootstrap procedure is more appropriate.
The most important decision is often not the percentage but the unit that must be independent. Splitting individual rows is wrong when several rows come from the same underlying entity.
Examples include:
If related examples appear in both training and test sets, the model can exploit entity-specific information. The measured performance may then describe recognition of known entities rather than generalization to new ones.
The safe rule is: split groups first, then construct examples or augmentations within each split.
A random split is appropriate when examples are approximately independent and identically distributed, and future data will come from the same stable population. It is simple and often effective, but it does not protect against hidden groups, duplicates, or temporal dependence.
A stratified split approximately preserves selected proportions—most commonly class frequencies—across subsets. It is useful when a random split might place too few minority-class examples in validation or test data.
Stratification should not be confused with balancing. Preserving a naturally rare class keeps the evaluation representative; oversampling that class changes the evaluation distribution unless metrics are reweighted appropriately.
A grouped split keeps all examples from an entity in one subset. For medical imaging, the split is usually by patient rather than image. For document chunks, it may be by source document. For federated learning, it may be by client.
This strategy evaluates the harder and often more relevant question: can the model generalize to unseen groups?
When a model predicts the future from the past, the split should respect time:
\[t_{\text{train}} < t_{\text{val}} < t_{\text{test}}.\]Randomly moving future observations into the training set leaks information that would not have existed at deployment time. Time-based evaluation also exposes temporal drift in user behavior, prevalence, policy, sensors, or market conditions.
If deployment involves new hospitals, countries, cameras, environments, or data providers, holding out entire domains can be more informative than randomly holding out examples within every domain. The correct split should resemble the intended generalization boundary.
Data leakage occurs when information unavailable at prediction time influences training or model selection. Leakage can make a weak model appear remarkably strong.
A feature directly or indirectly contains information about the target that would not be available when the prediction is made. For example, using a treatment prescribed after diagnosis to predict that diagnosis leaks the outcome timeline.
Any preprocessing step that learns from data must be fitted using the training set only. This includes:
The correct order is:
A pipeline abstraction is valuable because it binds preprocessing and model fitting into one procedure and reduces accidental leakage during cross-validation.
Exact duplicates, near-duplicates, or augmented versions of the same example must not cross split boundaries. Deduplication should consider semantics, not only identical file hashes: resized images, copied text with minor edits, and overlapping time windows can still reveal almost the same example.
Selecting features, thresholds, prompts, or checkpoints after observing test results leaks test information into the final system. Even an apparently harmless manual inspection can become selection leakage when it changes what is submitted or reported.
When data are limited, one validation split may give an unstable comparison. In $K$-fold cross-validation, the development data are divided into $K$ folds. Each fold serves as validation data once while the remaining $K-1$ folds are used for training:
\[\widehat R_{\text{CV}} = \frac{1}{K}\sum_{k=1}^{K}\widehat R^{(k)}_{\text{val}}.\]The average gives a more stable estimate for model selection, and variation across folds reveals sensitivity to the sampled data. The fold construction must still obey the correct unit: use stratified folds for class proportions, group folds for entities, and forward-chaining splits for time series.
Cross-validation does not eliminate the need for an independent test set when a final unbiased estimate is required. The usual workflow is:
When both model selection and performance estimation must use resampling, nested cross-validation places model selection in an inner loop and evaluation in an outer loop. It is more expensive but avoids evaluating a hyperparameter search on the same folds that selected it.
Independence is not enough. Validation and test data must also represent the distribution we care about.
Suppose a model is trained on studio photographs but deployed on mobile-phone images. A random split of the studio dataset can produce an accurate estimate of performance on more studio photographs while saying little about deployment. The estimate is statistically valid for the wrong population.
Useful questions include:
Sometimes training data are intentionally broader or rebalanced. That can be useful, but validation and test sets should still reflect the target population or be reweighted to a clearly defined target distribution.
If multiple deployment domains matter, report results separately rather than hiding failures inside a single average.
The previous article used training and validation errors as practical diagnostics. With a trustworthy split, their pattern provides an initial signal:
| Training result | Validation result | Likely interpretation |
|---|---|---|
| Poor | Poor and similar | High bias, weak features, optimization failure, or noisy labels |
| Strong | Much poorer | High variance, leakage in training evaluation, or distribution mismatch |
| Strong | Strong | Good fit on the validation distribution; final test evaluation is still needed |
| Poor | Surprisingly strong | Small-sample noise, inconsistent pipelines, or an implementation error |
Learning curves make the diagnosis more informative. Plot training and validation performance as the training-set size increases. A persistent high error in both curves suggests bias. A large gap that narrows with more data suggests variance.
These patterns are clues rather than proofs. Before changing model capacity, verify the split, metric, labels, and preprocessing pipeline.
A machine learning model must do more than fit its training data. It must learn a pattern that continues to work on examples it has never seen. Bias and variance describe two different ways this can go wrong:
The best model is not necessarily the most complex one or the one with the lowest training error. It is the model that generalizes best to the data we will encounter after training.
Here, bias means statistical bias in a learning procedure. It is different from social or demographic bias in data and model predictions.
Suppose examples are drawn from an unknown distribution
\[(X,Y) \sim P(X,Y),\]and a training set $D$ is sampled from this distribution. A learning algorithm uses $D$ to produce a predictor $\hat f_D$. If we drew a different training set, we would generally obtain a different predictor.
This gives us two questions:
The first question is about bias. The second is about variance.
Bias measures the systematic error introduced by the assumptions of the model and learning algorithm. A high-bias model cannot represent the important structure in the data, even when it is trained correctly.
Imagine fitting a straight line to a relationship that is strongly curved. More data will locate the best straight line more precisely, but it will not make that line curved. The model class itself is too restrictive.
Typical signs of high bias are:
High bias is therefore associated with underfitting.
Possible responses include using a richer model, constructing more informative features, reducing excessive regularization, or fixing an optimization problem that prevents the model from fitting even the training data.
Variance measures how much the learned predictor would change if it were trained on a different dataset sampled from the same population. A high-variance model learns not only the underlying pattern but also accidental details and noise in its training sample.
A very flexible curve may pass through every training point while behaving unpredictably between those points. Its training error can be extremely low, yet its error on new data can be much higher.
Typical signs of high variance are:
High variance is therefore associated with overfitting.
Possible responses include collecting more representative data, strengthening regularization, simplifying the model, using data augmentation, stopping training earlier, or averaging several models through an ensemble.
For squared-error regression, the expected prediction error at an input $x$ has the familiar decomposition
\[\mathbb{E}_D\!\left[(Y-\hat f_D(x))^2\mid X=x\right] = \operatorname{Bias}(x)^2 + \operatorname{Variance}(x) + \sigma^2(x),\]where
\[\operatorname{Bias}(x) = \mathbb{E}_D[\hat f_D(x)]-f(x),\]and
\[\operatorname{Variance}(x) = \mathbb{E}_D\!\left[ \left(\hat f_D(x)-\mathbb{E}_D[\hat f_D(x)]\right)^2 \right].\]and $\sigma^2(x)$ is irreducible noise in $Y$ given $X=x$.
Increasing model flexibility often reduces bias because the model can represent more complex patterns. The same flexibility can increase variance because the fitted model has more ways to respond to fluctuations in a finite training set. Regularization moves in the opposite direction: it usually increases bias while reducing variance.
The trade-off is not a law that one side must always worsen when the other improves. Better features, a more appropriate inductive bias, or more representative data can improve both. It is nevertheless a useful way to diagnose errors.
For classification and other losses, the clean squared-error equation does not carry over unchanged. Bias and variance remain useful concepts, but their exact decomposition depends on the loss and the definition being used.
Training and validation errors provide a useful first check:
| Observation | Likely problem | Useful next step |
|---|---|---|
| High training error and high validation error | High bias / underfitting | Increase suitable capacity, improve features, or reduce excessive regularization |
| Low training error and much higher validation error | High variance / overfitting | Add data or augmentation, regularize, simplify, or ensemble |
| Low training and validation error | Good fit on the current distribution | Test robustness, calibration, and distribution shift |
| High training error despite a powerful model | Optimization or data problem may dominate | Check labels, preprocessing, loss, learning rate, and convergence |
These are diagnostics, not proofs. A validation set drawn from the wrong distribution can hide both underfitting and overfitting. Label leakage can make validation error look excellent while real deployment performance is poor.
Even with unlimited data and a sufficiently capable model, classification error may not be zero. The input can be ambiguous, relevant information can be missing, labels can be noisy, or the outcome itself can be stochastic.
Assume we know the true conditional distribution $P(Y\mid X)$. Under 0–1 loss, the Bayes classifier predicts the most probable class:
\[f^*(x)=\arg\max_y P(Y=y\mid X=x).\]Its expected error is the Bayes error or Bayes risk:
\[R^* = \mathbb{E}_X\left[1-\max_y P(Y=y\mid X)\right].\]No classifier using the same input $X$, evaluated on the same distribution with the same 0–1 loss, can achieve a lower expected error. Bayes error is therefore the theoretical lower bound for that learning problem.
The qualification “using the same input” matters. Suppose two classes look identical in an image but can be distinguished with an additional sensor. The image-only task can have positive Bayes error even though the richer task does not. What appears to be irreducible error may be uncertainty created by an incomplete representation of the world.
In real problems, we do not know the true $P(Y\mid X)$. We only observe a finite sample from the world, so we generally cannot compute $R^*$ exactly. We can only construct proxies.
A direct approach is to train a probabilistic classifier—such as logistic regression or a neural network with a softmax output—to estimate
\[\widehat P(Y\mid X)=P(Y\mid X,\theta),\]where $\theta$ is learned from data. Given a large independent test set ${x_i}_{i=1}^N$, we can form the plug-in estimate
\[\widehat R_{\text{plugin}} = \frac{1}{N}\sum_{i=1}^N \left(1-\max_y \widehat P(Y=y\mid x_i)\right).\]This is intuitive for deep learning because the model already outputs an estimated class distribution. It is reliable only when those probabilities are close to the true conditional probabilities. Good calibration is important: a model that reports 90% confidence should be correct roughly 90% of the time on comparable examples. Calibration alone is not sufficient, however, because aggregate calibration can hide large local errors in $\widehat P(Y\mid X=x)$.
The test set must also be independent and representative of the deployment distribution. Otherwise, the estimate describes the wrong prediction problem.
For a simpler low-dimensional problem, we can estimate each class-conditional density $p(x\mid Y=k)$ and the class prior $P(Y=k)$. Bayes’ rule then gives
\[P(Y=k\mid x) = \frac{p(x\mid Y=k)P(Y=k)} {\sum_j p(x\mid Y=j)P(Y=j)}.\]Once this posterior has been estimated, it can be inserted into the Bayes-risk formula. Gaussian density models, kernel density estimation, and nearest-neighbor probability estimates are possible choices.
This approach becomes difficult in high-dimensional spaces. Density estimation in image or language input spaces usually requires an enormous amount of data and strong modeling assumptions, so the resulting Bayes-error estimate can be dominated by density-estimation error.
Collecting several independent labels for the same input gives more information than forcing a single “ground-truth” label. If 90 out of 100 reliable annotators label an example as class A and 10 label it as class B, we might estimate
\[P(A\mid x)\approx 0.9, \qquad P(B\mid x)\approx 0.1.\]The estimated local Bayes error for that example is then
\[1-\max\{0.9,0.1\}=0.1.\]This interpretation assumes that the variation in labels reflects genuine conditional uncertainty. Annotators can instead share systematic mistakes, use different labeling rules, or lack relevant context. Clear instructions, reliable annotators, independent judgments, and adjudication are therefore important.
Human error is not the formal Bayes lower bound: people can make systematic mistakes, and a classifier may outperform them. It is better understood as a practical measure of task difficulty. Repeated disagreement can reveal ambiguity, missing information, or label noise, while expert consensus can expose errors in the dataset labels.
In modern high-dimensional problems, exact estimation is usually impossible. If several powerful model families trained on large datasets converge to roughly the same test error, their performance provides a useful practical reference. For any particular classifier $h$,
\[R^*\leq R(h).\]The same applies to the strongest model we have trained:
\[R^*\leq R(h_{\text{best}}).\]If the best network obtains 3% test error, then—after accounting for finite-test-set uncertainty—we have evidence that $R^*$ is no greater than approximately 3%. We cannot conclude that the Bayes error is exactly 3%.
However, this achieved error is normally an upper bound on Bayes error, not proof of the lower bound: the model can still have approximation error, estimation error, optimization error, or distribution mismatch. A larger model is a better proxy only when its predictions are supported by sufficient data and evaluated properly.
None of these approaches recovers the true world model from limited data. They give increasingly useful approximations to the best achievable performance under a clearly defined input, target, distribution, and loss.
A helpful conceptual summary is
\[\text{generalization error} \approx \text{Bayes error} + \text{approximation error} + \text{estimation error} + \text{optimization error}.\]This is a conceptual accounting for classification, not a universal exact additive identity.
This distinction prevents a common mistake: blaming every stubborn error on model capacity. Sometimes a model underfits. Sometimes it overfits. Sometimes training is poor. And sometimes the input simply does not contain enough information to determine the label reliably.
The guiding question is:
What does a model’s error tell us about the data, the learning algorithm, and the task itself?
Why high bias leads to underfitting, why high variance leads to overfitting, and where Bayes error sets the irreducible limit.
Train, Validation, and Test Sets
How to measure generalization without leaking information, overfitting the evaluation, or testing on data that does not represent deployment.
Future notes will build on this foundation with topics such as:
The series will grow as I add concise explanations, mathematical intuition, and practical diagnostics for each concept.
]]>