What it means
Overfitting occurs when a model fits training examples or noise too closely and does not generalize to new conditions.
Why it matters
A model can appear excellent during development yet fail after deployment because evaluation data was reused, unrepresentative, or contaminated.
Practical focus
- Define the approved purpose, accountable owner, users, operating environment, and acceptable risk for overfitting and generalization before implementation or procurement
- Implement independent test data, cross-validation, regularization, early stopping, leakage prevention, and temporal or geographic holdouts through documented architecture, policy, configuration, and change control
- Collect learning curves, train-validation gaps, holdout results, leakage checks, and out-of-distribution tests and compare the results with approved requirements, risk thresholds, and expected outcomes
- Exercise this failure scenario as a planned test: repeated tuning against the same test set turns the test set into training data and inflates confidence
Common mistakes
- Using the label overfitting and generalization without defining the exact system boundary, decision, data, and people in scope
- Treating a model response or vendor claim as sufficient evidence without testing the surrounding application and operating process
- Failing to preserve learning curves, train-validation gaps, holdout results, leakage checks, and out-of-distribution tests in a form that lets reviewers reconstruct decisions and changes
- Testing only the normal path and never rehearsing what happens when repeated tuning against the same test set turns the test set into training data and inflates confidence
Certification relevance
This subject appears in or supports the following certification bodies of knowledge: