When traders set in-sample and out-of-sample periods for their backtests, it’s common just to pick some dates that split the data into a pre-defined percentage or range.
The trouble with this approach is that it often doesn’t take into account the type of market environments that exist in those periods. For example, your in-sample period may be very bearish, and if your out-of-sample period is more bullish then the backtest results could be out of balance, potentially yielding poor out-of-sample performance or giving unrealistic results.
You could even be potentially throwing away a good strategy purely because of the way you’ve split the data!
What can traders to do address this issue?
Last week I received this excellent question from BST listener Theo:
“I’ve been a great fan of your podcast and I have literally recommended it to everyone I know in the trading world.
I have a question I would truly appreciate if you could share some insight.
You have mentioned a couple of times the approach of stitching together bearish, bullish or sideways data for a backtesting.
I’m a bit confused on how this approach takes place. Does this approach mean you stick a whole insample period with bullish data and then leave the out of sample intact and repeat that for bearish and sideways market conditions as separated backtests?
Also does the out-of-sample has to be equally bullish/bearish or sideways in that case or just remains intact as your most recent data?”
Firstly, thanks for the support Theo, glad you’re enjoying the podcast!
Now, onto your question, which a really good one.
I originally heard about this approach from World Cup Trading Champion Michael Cook, who was a guest on episode 39 of the podcast, so I thought it would be best to get it directly from the source and asked Michael for more details on how he actually does this.
He very kindly shared it in a step-by-step process:
“I hope this will cover both the questions. The way I do it is like this:
- I take a specific product for the entire period to be tested and print its price history on a high definition graph, say 2006 to yesterday.
- Visually mark all the periods I would regard as being in one of six categories, as bullish, bearish, rangebound. Each being volatile or non volatile,
- Then I allocate each of those six market conditions to one of two ‘pools’. Pool 1 becomes in sample for testing, pool 2 is my out of sample. If there are 5 bullish volatile periods, 3 would probably go into the in-sample pool, the other two into out of sample,
- In this way, as a by-product, some of the most recent data will be in-sample and some of the most recent data will be out of sample,
- Then each of these periods is put into a single function which I call e.g. SoybeansDatesTest. By doing this, the function can be true or false and this function is incorporated into every entry rule of each strategy being developed. So of course an entry may be within one period and by the time that specific trade exits, it could be in a different period. But as the function is simply an entry filter, the system wont exit anything just because the entry condition has changed during the trade (incidentally, just as my system wouldn’t exit in the real world if underlying conditions suddenly changed as it probably wouldn’t realise until later!)
That’s the basic process.
Of course, it is unscientific in that I don’t think anything will be gained by generating the period categorisation with an algorithm – I am a great believer in being led by data, but not being its slave, so I am fine to do it manually. This exercise is a bit manual and laborious, but once, say SoybeansDatesTest is done, that probably wont need to be updated for a year and in exactly the same way I have NatGasDatesTest, CrudeDatesTest etc etc”
It’s an interesting approach to splitting data, which could potentially give you more ‘balanced’ backtest results by intentionally controlling the different types of market environments in a backtest, rather than being randomly subjected to whatever market environments just happen to occur between a date range.
If you’d like to read more about Michael’s approach to trading, along with other podcast guests too, check out the ‘Interviews with Pro Traders’ series here.
Cheers,
Andrew.
Released:
4 April 2018
