In brief
How should machine learning models be validated?
Design the test to mirror how the model will be used. If it predicts the future, train on the past and test on later periods. If it will meet new customers, sites or machines, keep those entities out of training. Check that no feature contains information unavailable at prediction time, and compare results with the current method, not only with other models.
Randomly splitting data into training and test sets is the first technique most practitioners learn. It is appropriate when records are independent and the future looks like the past. Many business problems violate both conditions, and random splits then produce results that are optimistic, sometimes dramatically so.
Three ways validation goes wrong
Time leakage
A demand forecasting model trained on randomly split data learns from records that occur after some of its test records. It effectively sees the future. In production, it will only ever see the past, so its real accuracy is lower than testing suggested.
Entity leakage
A model predicting equipment failure may have many records for each machine. If records from the same machine appear in both training and test sets, the model can learn that particular machine's quirks. When deployed to new machines, performance falls.
Feature leakage
Sometimes a feature quietly encodes the answer. A churn model might use a field updated when a cancellation request is logged. A claims model might use a status that is only set after investigation. These features make models look excellent and useless at the moment a prediction is needed.
Choosing the right validation design
| Situation | Validation design | Example |
|---|---|---|
| Predicting future events | Time-based split or rolling backtest | Train on January to June, test on July; repeat by rolling forward |
| New entities in production | Group-based split | All data for held-out stores, customers or machines kept for testing |
| Both of the above | Time and group combined | Test on later periods for entities not seen in training |
| Deployment at a new site | Site-level holdout | Train on several plants, test on a different plant |
| Independent, stable records | Random or stratified split | Classifying well-defined document types |
A leakage checklist
- For each feature, ask when its value becomes known relative to the moment of prediction.
- Check whether any feature is updated by the process the model is trying to predict.
- Confirm that data preparation steps such as scaling or encoding were fitted on training data only.
- Look for suspiciously strong single features; investigate before celebrating.
- Ensure duplicates or near-duplicates do not span training and test sets.
Compare with what exists today
A model's accuracy matters less than its improvement over the current method. For forecasting, that might be last year's sales plus a planner's adjustment. For risk scoring, the existing rules. Evaluate the baseline under the same validation design. A model that beats other models but not the current process is not ready.
Validation continues after launch
Even a well-validated model meets conditions that testing did not cover. Monitoring accuracy against actual outcomes, tracking changes in input data and comparing performance across segments turns validation into an ongoing practice. This is where machine learning research meets MLOps.
Sources and further reading
About the author: Dr. RohitCo-Founder, Director and Chief Technology Officer, AARHIT Systems. Writes on technology strategy, AI architecture, agents, evaluation and AI governance. More articles by Dr. Rohit