Data leakage is any path by which information from your evaluation data reaches your model before evaluation. The result is always the same: the reported metric goes up, the deployed performance does not, and nothing in the pipeline throws an error. A model trained with leakage does not look broken. It looks like your best model.
The scale of the problem is documented. A survey by Sayash Kapoor and Arvind Narayanan, published in Patterns in 2023, found leakage errors across 17 scientific fields, collectively affecting 329 published papers. In their case study field, civil war prediction, every paper claiming that complex ML models beat logistic regression failed to reproduce, and the culprit in each case was leakage. Those are published, peer reviewed results that cleared review because leakage leaves no trace in a paper. It only leaves a trace in the pipeline.
What follows is a field taxonomy: six classes of leakage, the mechanism behind each one, the measured damage where a measurement exists, and the order in which to audit a pipeline. The most useful recent finding comes from a 2026 study that ran 29 controlled leakage experiments across 2,047 benchmark datasets and measured the AUC gap between leaky and clean versions of the same workflow. Its conclusion inverts the textbook: the leakage everyone warns about first, fitting a scaler on the full dataset, barely moves the needle. The leakage that dominates in practice is the kind where the labels get consulted.
The rule that generates the whole taxonomy
Every class below is a special case of one principle, formulated by Shachar Kaufman, Saharon Rosset, and Claudia Perlich in their 2012 work on leakage in data mining: a feature or a procedure is legitimate only if the information it uses would be available at prediction time, for the unit being predicted. Leakage is illegitimate information availability. The classes differ only in the channel through which the illegitimate information travels: through preprocessing statistics, through repeated entities, through time, through the feature itself, through the model selection loop, or through the arithmetic of the metric.
The channel determines the damage. That is the most practical fact on this page. Information about feature distributions leaks weakly. Information about labels leaks catastrophically. Keep that asymmetry in mind and the measured hierarchy below stops being surprising.
Class 1: Label-blind preprocessing leakage
This is the textbook example. You standardize features, impute missing values, or fit a PCA using statistics computed on the entire dataset, then split into train and test. The test fold’s means and variances have now influenced the training data’s representation.
The mechanism explains why this class is weak. A scaler transfers marginal feature statistics: means, variances, quantiles. It never touches the labels. For the test fold’s statistics to help the model, the marginal distribution of features would have to carry substantial information about the test labels beyond what the training folds already provide, and on datasets of ordinary size it does not. The 2,047-dataset study measured this directly: across nine estimation-leakage conditions, the AUC difference between leaky and clean pipelines stayed at or below 0.005. Real, detectable, and almost never the reason your model looks too good.
The caveat is dataset size. As the training set shrinks toward the tiny-sample regime, shared statistics carry proportionally more information, which is exactly the regime where the classic biology results live. Fix it anyway, because the fix is free: fit every preprocessing step inside each cross-validation fold, on that fold’s training portion only. A pipeline object does this by construction. But when your metric looks suspiciously good, this is not the first place to look.
Class 2: Label-aware selection leakage
This is the first place to look. Selection leakage occurs whenever a choice in the pipeline consults the evaluation data’s labels: selecting features by their correlation with the target on the full dataset, target-encoding categoricals before splitting, tuning hyperparameters against the same folds used for the reported score, picking the best of ten random seeds, or running the evaluation repeatedly and keeping the run you liked.
The founding demonstration is 24 years old. Christophe Ambroise and Geoffrey McLachlan showed in PNAS in 2002 that selecting genes on a full microarray dataset before cross-validating produced error estimates biased hard toward zero, on studies whose honest error was far higher. The rule was tested on samples that had already voted on which features exist. Hastie, Tibshirani, and Friedman turned it into the canonical classroom demonstration in The Elements of Statistical Learning, section 7.10.2: select predictors on the full data from thousands of pure-noise features, cross-validate the wrong way, and the pipeline reports a small single-digit error rate on data where the honest number is 50 percent, because there is no signal at all.
The 2026 measurements confirm the mechanism at scale and add a diagnosis: in the selection experiments, roughly 90 percent of the measured metric inflation was noise exploitation. The pipeline is not finding signal in the test data. It is memorizing the test data’s noise through the selection loop, one greedy choice at a time. This is also why the class is so dangerous: every act of model development, every tried-and-discarded architecture, every early stopping decision made against the evaluation set, is a small selection event, and they compound silently across a project.
The fix has a name: nested validation. Anything chosen must be chosen inside an inner loop, and the outer loop’s data must never have voted. If a choice was made by looking at a number, the data that produced the number is spent.
Class 3: Duplicate and group leakage
Rows are rarely independent. Multiple images from one patient, multiple sessions from one user, multiple sequences from one protein family, near-duplicate records from data collection. Split randomly at the row level and the same underlying entity appears on both sides of the split. The model then gets credit for recognizing the entity rather than learning the task.
The mechanism is memorization, and it scales with model capacity, because higher-capacity models memorize entities better. The 2,047-dataset study measured effect sizes from 0.37 for naive Bayes up to 1.11 for decision trees on the same duplication conditions. A domain-specific measurement makes it concrete: a 2026 leukemia classification benchmark by Nisreen Albzour and colleagues showed that the near-perfect results long reported on the C-NMC 2019 blood smear dataset were inflated by exactly this, cells from the same patient landing in both training and test folds. Under a strict subject-disjoint protocol with 28 fully unseen patients, the best honest AUROC was 0.913, and even in their most conservative frozen-feature setting, random splitting inflated AUROC by about 0.04. In a literature where papers compete over the third decimal place, 0.04 of free, fictional AUROC decides which method looks best.
The fix is to split on the dependence unit, not the row: GroupKFold on patient, user, molecule, or document identity. The hard prerequisite is knowing what the dependence unit is, which is a property of your data collection, not your model, and no library can infer it for you.
Class 4: Temporal leakage
If the task involves time, the split must respect time. Training on March to predict January means the model has seen the future, and features computed over windows that cross the split boundary smuggle the future in even when the split itself looks clean. Random cross-validation on temporal data commits this class by construction, which is why the boundary experiments in the 2,047-dataset study found it invisible under random CV: the evaluation procedure that causes the leak is structurally incapable of detecting it.
A 2026 PLOS One study of ML build-failure prediction dissected the temporal class into three subtypes worth stealing for any domain: direct outcome encoding, where a feature is the label wearing a costume; execution-dependent metrics, features that only come into existence while the predicted event runs; and future information leakage, features computed from chronologically later records. All three produce models with excellent papers and no deployment value, because the information they rely on does not exist at the moment a real prediction is needed.
The fix is a temporal split with an embargo gap, plus one question asked of every feature: at the timestamp of prediction, did this value exist yet?
Class 5: Target leakage in the features
Sometimes a feature simply encodes the outcome. The classic cases are administrative: a hospital dataset where a treatment code implies the diagnosis being predicted, a churn dataset containing the account closure reason, a fraud dataset with a field populated by the fraud team after investigation. The feature is legitimate data, collected honestly, present in production databases, and completely unavailable at prediction time, because it is generated downstream of the event being predicted.
This class does not yield to any splitting discipline. GroupKFold cannot save you, temporal splits cannot save you, because the leak rides inside individual rows. The only defense is feature provenance: for each column, establish when it gets written relative to the event you predict, and by what process. Suspicion should scale with performance. A single feature with enormous importance in a domain where prediction is known to be hard is a target leak until proven otherwise. This is the class where the mechanism-level question, what process generated this value, does the entire job.
Class 6: Derivative-metric contamination
This class is mostly absent from taxonomies, and it is where careful teams still get burned. Many reported quantities are not single predictions but arithmetic over several: the difference between two predicted stabilities, an improvement delta over a baseline model, a ratio, a rank ordering across predictions, a metric computed at multiple operating points. The rule for these is strict: a derived quantity is out-of-sample only if every parent prediction feeding it is out-of-sample. One leaked parent contaminates the derivative, and the contamination is invisible in the derivative itself, because the arithmetic launders the provenance.
Two variants recur. The first is the interpolator trap. Some model components are exact interpolators: one-nearest-neighbor, RBF interpolation through all points, splines constrained through the data, kernel regression with zero bandwidth. Their training error is identically zero by construction, so any in-sample metric that includes one is vacuous, and any derived metric with an interpolator among its parents inherits the vacuum. The second is correction circularity: a residual or correction model, fit on data that includes the evaluation point, makes the corrected metric partially a function of the answer. Corrections, calibrations, and stacking layers must be re-fit inside every fold, exactly like a scaler, except that unlike a scaler they consult the labels, which moves them from Class 1 damage to Class 2 damage.
The audit question for this class is genealogical: for every number in the results table, list its parent predictions, and demand that each parent name its validation. A number that cannot state its provenance, this fold, this holdout, this external set, is not a result yet.
The audit order
Ordering the audit by expected damage rather than textbook order changes what you check first. Start with the dependence unit: what is the entity whose repetition would let the model cheat, and does the split respect it? Second, the selection loop: list every decision made during development, features, hyperparameters, seeds, architectures, stopping points, and ask which data voted on each. Third, feature provenance: for every column, when is it written relative to the predicted event? Fourth, time: does any feature window cross the split boundary? Fifth, derivative genealogy: do all parents of every reported number carry independent out-of-sample provenance? Last, and only last, the preprocessing placement that tutorials put first.
The order matters because audit attention is finite. A team that spends its review budget confirming the scaler sits inside the pipeline, while the same patient appears in train and test, has audited the 0.005 problem and shipped the 0.04 problem.
Why leakage persists
Leakage survives because every incentive points the same direction. It inflates the metric, and inflated metrics get published, promoted, funded, and shipped. It produces no error message, no warning, no failed test. Its removal makes your numbers worse, which means the person who finds the leak delivers bad news about work that already looked finished. And it hides from readers completely: as Kapoor and Narayanan note, none of the 329 affected papers could have been flagged by reading them, because the leak lives in code and data handling that papers do not show.
This publication has documented the same structure elsewhere. Coding agent scores swing by double digits on the evaluation setup alone, with no change in the model. A flagship model’s hallucination rate fell 38 points without the model learning anything, because the measurement moved. Even counting lawsuits breaks the same way when the instrument changes underneath the phenomenon. Leakage is the ML-internal member of this family: the measurement and the thing being measured are entangled, and the entanglement always flatters the result.
What this taxonomy cannot do
Honest limits. First, the quantitative hierarchy comes primarily from one large study on iid tabular binary classification. The ordering of effect sizes is mechanistically motivated and consistent with domain results like the leukemia benchmark, but magnitudes will differ in vision, language, and graph settings, and in the small-sample regime the label-blind class grows.
Second, detection is not always decidable. Group leakage requires knowing the dependence structure of your data, and as Kapoor and Narayanan acknowledge, handling nonindependence without knowing that structure is an open problem, not a checklist item. Near-duplicates without shared identifiers, batch effects, and annotator overlap can all create groups nobody recorded.
Third, the boundary between leakage and distribution shift is genuinely blurry. A test set that fails to represent the deployment distribution inflates metrics through the same arithmetic as leakage, and some taxonomies count it in while others count it out. The line drawn here follows the mechanism: if illegitimate information travels from evaluation to training, it is leakage; if the evaluation itself measures the wrong population, it is a validity problem that no split discipline fixes.
Fourth, the six classes are a lossy compression. Kapoor and Narayanan’s taxonomy has eight types, the 2,047-dataset study uses four causal mechanisms, and the build-prediction work uses three temporal subtypes. The classes here are chosen for audit utility, not exhaustiveness, and edge cases will straddle them.
What happens next
Tooling is starting to encode the discipline. bioLeak, an R package by Selçuk Korkmaz, builds leakage-aware resampling and post-hoc leakage audits directly into the modeling workflow, with train-fold-only preprocessing and nested tuning as defaults rather than options. Kapoor and Narayanan propose model info sheets, structured claims about the absence of each leakage type that a reviewer can actually check. And leakage-aware benchmarks like the subject-disjoint C-NMC protocol replace contaminated leaderboards outright, which is the only correction that sticks, because it changes what winning means.
The direction of travel is clear: the burden of proof is shifting from the skeptic to the claimant. A reported metric increasingly needs to arrive with its provenance attached, named validation scheme, named dependence unit, named split discipline, or it gets discounted. That is the correct equilibrium. Until it arrives, the working posture for any ML engineer reading any result, including their own, is the one this taxonomy is built to serve: assume the number is inflated, and go looking for the channel.
Frequently asked questions
What is data leakage in machine learning?
Data leakage is any mechanism by which information from evaluation data influences a model before evaluation, producing metrics that overstate real performance. It includes preprocessing fit on full datasets, entities repeated across train and test splits, features containing future or outcome information, model selection performed against the reported test data, and derived metrics with contaminated inputs.
Does cross-validation prevent data leakage?
No. Cross-validation only controls leakage that operates at the row level, and only if every data-dependent step, preprocessing, feature selection, tuning, and corrections, is re-fit inside each fold. It does nothing against grouped entities split across folds, temporal information, or features that encode the outcome, and random cross-validation on temporal data causes leakage rather than preventing it.
How much does data leakage inflate model performance?
It depends on the class. Measurements across 2,047 benchmark datasets found that fitting scalers on full data shifts AUC by at most 0.005, while selection-type leakage produced the largest effects, with roughly 90 percent of the inflation traceable to noise exploitation. In medical imaging, patient-level leakage inflated AUROC by about 0.04 even in conservative settings, and memorization effects grow with model capacity.
How do I detect data leakage in my pipeline?
Audit by expected damage: verify the split respects the dependence unit (patient, user, entity), list every development decision and which data voted on it, establish when each feature is written relative to the predicted event, check that no feature window crosses a temporal boundary, trace the parent predictions of every derived metric, and confirm preprocessing is fit per fold. A metric that cannot name its validation scheme is not yet a result.
Sources for all quantitative claims are linked inline: the Kapoor and Narayanan survey (Patterns, 2023), the 2,047-dataset leakage study (arXiv, 2026), the subject-disjoint leukemia benchmark (arXiv, 2026), Ambroise and McLachlan (PNAS, 2002), The Elements of Statistical Learning section 7.10.2, the PLOS One build-prediction taxonomy (2026), and the bioLeak package paper (arXiv, 2026).