What it means
Model data leakage occurs when training or evaluation uses information that would not legitimately be available at prediction time, causing performance to appear better than it will be in practice.
Why it matters
Leakage can survive ordinary accuracy checks and lead decision-makers to deploy a model whose apparent signal comes from future, target-derived, or duplicated information.
Practical focus
- Define the scope, owners, acceptable risk, and required outcome for model data leakage before choosing a tool or platform feature
- Implement time-aware and entity-aware splitting, feature lineage, duplicate detection, pipeline isolation, and independent evaluation through documented, reviewable configuration and change control
- Collect feature availability dates, overlap tests, split rationale, preprocessing fit scope, and validation discrepancies and compare the results with approved baselines, service objectives, and policy requirements
- Exercise a target-derived, future, duplicate, or post-outcome feature entering training or evaluation as a planned test case, including detection, containment, rollback, escalation, and evidence retention
Common mistakes
- Treating model data leakage as a one-time technical setting instead of an operating control with owners and review cycles
- Using default thresholds or broad exceptions without connecting them to business, privacy, security, and availability requirements
- Keeping evidence in disconnected tools so reviewers cannot reconstruct who changed what, why it changed, or whether it worked
- Testing only the normal path and discovering during an incident that a target-derived, future, duplicate, or post-outcome feature entering training or evaluation was never rehearsed
Certification relevance
This subject appears in or supports the following certification bodies of knowledge: