Suppose an investment strategy produced high returns over the past few years. But what if dozens of rules were tested and only the best one was picked? It may be a rule that fit the past well by chance. The important question is whether the rule can hold up in a period it has never seen.
Rules that fit the past too well
Imagine a student who saw the exam questions and answers in advance and memorized the answers. They would score highly on those same questions, but you cannot tell whether they would also do well on new ones. Investment rules are similar. If you keep changing the conditions to match past price movements, even the chance features of that period can end up in the rules.
This problem, where rules are fitted so closely to past data that they do not work well in new situations, is called overfitting. You cannot conclude that a strategy is overfitted just because its past results are good. But you should ask how many rules were changed and tried while those results were being produced.
For example, if someone changes the buying criterion over and over, to 5 days, 10 days, 20 days and so on, and reports only the number with the highest return, you also need to see how the other numbers failed. The more conditions are tried, the more you need to consider the chance that a result that only looks good by luck was picked.
Separate the period that builds the rules from the period that checks them
Let us split the data in time order. The dates and results below are a hypothetical case for illustration. They are not a strategy that was actually tested, and they are not a stock recommendation.
| Stage | Hypothetical period | What you do |
|---|---|---|
| Training period | 2015–2019 | Compare several candidates and build the rules |
| Validation period | 2020–2021 | Compare the candidates and choose the detailed settings |
| Final test period | 2022–2023 | Check the rules set earlier without changing them |
The training period is the data used to build the rules. The validation period is the data used to check which of the candidates to choose. If, during this process, you change the conditions after looking at the validation results, that period no longer counts as fresh, unused data for the final assessment.
The final test period, kept aside until the end, is data you did not look at when you chose the rules. It is often called out-of-sample data. Here, the sample means the range of data used in the analysis. Strictly speaking, the validation period is also out of sample as far as training goes, because it was not used for training. But if you chose the rules after looking at its results, you should keep it distinct from an independent final test period.
A hypothetical case where the return changes
Assume a hypothetical rule was chosen in the training period and then applied unchanged in the final test period. Each period is a separate calculation, and the periods differ in length. Each one starts with $1,000, and we assume there are no deposits or withdrawals, commissions or taxes.
If $1,000 grew to $1,400 in the training period, the return for the whole period is as follows.
(1,400 - 1,000) ÷ 1,000 × 100 = 40%.
If $1,000 became $950 in the final test period, the return is as follows.
(950 - 1,000) ÷ 1,000 × 100 = -5%.
These figures are returns for each whole period, not average annual returns. Because the periods differ in length, you cannot compare 40% and -5% directly to judge which was better each year, or by how much. Still, this is an example of good results from the rule-building stage not showing up in the same way afterward.
You cannot conclude from this result alone that overfitting caused the failure. Market conditions may have changed, or the test period may have been short. Check not only the return but also how large the losses were, how the benchmark did and which costs were left out. Backtest results are not money earned in actual trading.
Moving forward in time and checking several times
With just one split, you may simply have picked a favorable period by chance. That is why some tests move forward in time one step at a time and evaluate at each step. This is called walk-forward testing.
For example, you set the rules using 2015–2019 and apply them in 2020. Next, you use 2016–2020 data to set the rules again, in a way decided in advance, and apply them in 2021. At each evaluation point, use only information that had been made public by then, and do not look ahead at results from later periods when you set the rules.
This process helps you see how a strategy behaves in different periods, but it does not guarantee future returns. If you keep looking at the results and adjusting the length of the training period and the criteria, the periods used for evaluation are in effect also being used for selection. Record which rules you changed and when, along with the candidates you tried, and at the end check the result on data you have not looked at before.
When you look at a strategy in practice
If someone shows you strong performance, first check the start and end dates of the results. Ask whether the strategy was built and changed during that same period, and whether there is a separate period that was not used for selection. When you compare strategies, match the market, the period, the amount invested, deposits and withdrawals, and how dividends and costs were handled.
For example, suppose a loss appeared in the final test period, and the buying criteria were changed after seeing that result and then tested again on the same period. What is wrong with that? The second result has been adjusted with reference to that period. To use it as a new report card, lock in the revised rules first, then check them on a separate period that has not been used for selection before.
Good testing shows how the rules were built and the risks across several periods, rather than a single high return. In the next lesson, we will look at how results change when the calculation includes trading costs and real execution conditions.