Every backtest you run lives in one of two states: in-sample (you optimized parameters using this data) or out-of-sample (you did not). The distinction is the single most important concept in quantitative validation.
The Definitions
In-sample (IS) data is the data the strategy was tuned on. Any parameter you chose by looking at backtest results — RSI period, threshold level, stop loss — is fitted to that data. The IS Sharpe is, in expectation, higher than the strategy's true edge because you have selected for what looks good in this particular sample.
Out-of-sample (OOS) data is data the strategy did not see during tuning. OOS performance is the unbiased estimate of what your strategy would have done in real trading.
You can run the same in-sample backtest a thousand times. You can only run an out-of-sample test once. The moment you tweak parameters in response to OOS results, that data becomes in-sample.
Why IS Always Looks Better
Suppose you test 100 random strategies with no real edge. By chance, the best of them will have a Sharpe of 1.5 or higher in-sample — pure noise selected for. Now you publish "the strategy" with its 1.5 Sharpe. On new data, that strategy will revert to its true Sharpe of zero, plus or minus noise. The gap is called selection bias.
This happens at every scale: choosing between MA(20) and MA(50), between two timeframes, between two confirmation filters. Every choice you made by looking at IS results contributes a little to selection bias. The total bias compounds.
The Standard Split
QuanterLab modules default to a 70% / 30% split: the first 70% of historical data is in-sample, the last 30% is out-of-sample. The OOS portion is hidden until you click "Reveal Out-of-Sample," and the platform tracks how many times you have revealed it for each strategy.
Why 30%? Enough OOS to be statistically meaningful (a 6-month OOS in a 2-year sample has roughly 125 trading days), small enough to leave most of the data for fitting.
The Single-Use Rule
The OOS reveal is one-shot. After you see it, your brain has updated, and any subsequent re-tuning is implicitly fitted to that OOS view. This is why QuanterLab has the Time Travel feature — to give you a way to do more than one OOS test by simulating "today" at different historical dates, each with its own clean OOS forward of that date.
Build and tune on IS data. Decide your final parameters. Lock them. Then reveal OOS. If OOS disappoints, do not re-tune — that disappointment is information. The strategy is what it is.
When OOS Is Not Enough
A single 30% OOS slice gives you one window into the future. If that window happens to be a particularly favorable or hostile market regime, your OOS verdict is misleading. Walk-forward validation generalizes the IS/OOS split into many sequential windows, each with its own clean OOS slice — far more robust than the single-shot version. See the Walk-Forward article for how to use it.
The Bottom Line
Treat IS results as hypothesis generation and OOS results as hypothesis testing. Confusing the two is the most common, most expensive mistake in quantitative research, and it is the mistake QuanterLab's entire validation stack is designed to prevent.
Further Reading
Foundational papers
- Bailey, D. H., Borwein, J. M., López de Prado, M. & Zhu, Q. J. (2014). Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance. Notices of the AMS, 61(5), 458–471.
- Harvey, C. R. & Liu, Y. (2015). Backtesting. Journal of Portfolio Management, 42(1), 13–28.
Textbook references
- Pardo, R. (2008). The Evaluation and Optimization of Trading Strategies (2nd ed.). Wiley.
- López de Prado, M. (2018). Advances in Financial Machine Learning. Wiley.
Related QuanterLab articles
- Walk-Forward Validation: Anchored vs Rolling
- The Time Travel Workflow
- p-Hacking and Curve-Fitting in Quant Research
Try it in QuanterLab
On any builder module, run a backtest with the default 70/30 split. Click "Reveal Out-of-Sample" only after you have committed to your parameters — that single click is the whole game.