• iTunes
  • Twitter
  • Facebook
  • Google+
  • Instagram
  • Youtube
  • Email
  • Home
  • Courses
  • eBooks
  • Library
  • The Empowered Trader Club
  • BST LIVE
  • About
Better System Trader

Better System Trader

Better System Trader is the podcast and blog dedicated to systematic traders, providing practical tips from trading experts around the world.

Stitching data for a more ‘balanced’ backtest

When traders set in-sample and out-of-sample periods for their backtests, it’s common just to pick some dates that split the data into a pre-defined percentage or range.

The trouble with this approach is that it often doesn’t take into account the type of market environments that exist in those periods. For example, your in-sample period may be very bearish, and if your out-of-sample period is more bullish then the backtest results could be out of balance, potentially yielding poor out-of-sample performance or giving unrealistic results.

You could even be potentially throwing away a good strategy purely because of the way you’ve split the data!

What can traders to do address this issue?

Last week I received this excellent question from BST listener Theo:

“I’ve been a great fan of your podcast and I have literally recommended it to everyone I know in the trading world.

I have a question I would truly appreciate if you could share some insight.

You have mentioned a couple of times the approach of stitching together bearish, bullish or sideways data for a backtesting.

I’m a bit confused on how this approach takes place. Does this approach mean you stick a whole insample period with bullish data and then leave the out of sample intact and repeat that for bearish and sideways market conditions as separated backtests?

Also does the out-of-sample has to be equally bullish/bearish or sideways in that case or just remains intact as your most recent data?”

Firstly, thanks for the support Theo, glad you’re enjoying the podcast!

Now, onto your question, which a really good one.

I originally heard about this approach from World Cup Trading Champion Michael Cook, who was a guest on episode 39 of the podcast, so I thought it would be best to get it directly from the source and asked Michael for more details on how he actually does this.

He very kindly shared it in a step-by-step process:

“I hope this will cover both the questions. The way I do it is like this:

  1. I take a specific product for the entire period to be tested and print its price history on a high definition graph, say 2006 to yesterday.
  2. Visually mark all the periods I would regard as being in one of six categories, as bullish, bearish, rangebound. Each being volatile or non volatile,
  3. Then I allocate each of those six market conditions to one of two ‘pools’. Pool 1 becomes in sample for testing, pool 2 is my out of sample. If there are 5 bullish volatile periods, 3 would probably go into the in-sample pool, the other two into out of sample,
  4. In this way, as a by-product, some of the most recent data will be in-sample and some of the most recent data will be out of sample,
  5. Then each of these periods is put into a single function which I call e.g. SoybeansDatesTest. By doing this, the function can be true or false and this function is incorporated into every entry rule of each strategy being developed. So of course an entry may be within one period and by the time that specific trade exits, it could be in a different period. But as the function is simply an entry filter, the system wont exit anything just because the entry condition has changed during the trade (incidentally, just as my system wouldn’t exit in the real world if underlying conditions suddenly changed as it probably wouldn’t realise until later!)

That’s the basic process.

Of course, it is unscientific in that I don’t think anything will be gained by generating the period categorisation with an algorithm – I am a great believer in being led by data, but not being its slave, so I am fine to do it manually. This exercise is a bit manual and laborious, but once, say SoybeansDatesTest is done, that probably wont need to be updated for a year and in exactly the same way I have NatGasDatesTest, CrudeDatesTest etc etc”

It’s an interesting approach to splitting data, which could potentially give you more ‘balanced’ backtest results by intentionally controlling the different types of market environments in a backtest, rather than being randomly subjected to whatever market environments just happen to occur between a date range.

If you’d like to read more about Michael’s approach to trading, along with other podcast guests too, check out the ‘Interviews with Pro Traders’ series here.

Cheers,
Andrew.

Released:

4 April 2018

  • facebook
  • linkedin
  • twitter
  • google+
  • pinterest
  • stumbleupon
144 - Building dynamic trading s...
The Apple Portfolio Strategy

Related Posts

  • 054 – From trading ideas to robust strategies
  • 073 – Combining simple concepts to build robust strategies with Art Collins
  • 060 – Strategy Optimization with Robert Pardo
  • 096 – Could you be throwing away good trading strategies?
  • Sharpe Ratio – the right answer to the wrong question?

Trackbacks

  1. Quantocracy's Daily Wrap for 04/04/2018 | Quantocracy says:
    April 5, 2018 at 3:19 pm

    […] Stitching data for a more ‘balanced’ backtest [Better System Trader] […]

Next issue (Apr 2021):

Get the new "The Empowered Trader" monthly newsletter, direct to your door (yes it's printed!)

Click here for more info

Want to build robust trading strategies that actually work in live trading?

Click for more details

How do you REALLY know your strategies aren't just over-optimized rubbish?

Get this handy optimization checklist from veteran trader Bob Pardo, and make sure you haven’t made any of these mistakes that could lead to strategy failure.

Click to download now

What pro traders are saying about BST

"Andrew seems to have a knack for getting the world's most famous traders to share valuable insights about how they trade for the benefit of the trading community."

Andrew Gibbs halifaxonline.co.nz

"I really enjoy the wide variety of topics and perspectives shared in the podcasts and with the interviews done by someone with real life trading experience"

Tim Rea thetradingmasterclass.com

"Being a trader himself, Andrew drives insightful conversations that uncover gems of practical knowledge that can improve your trading today"

Kris Longmore robotwealth.net

"The key to better system trading is information.  As a system trader himself, Andrew has unique capability of asking insightful questions that extract the skills of each guest."

John Ehlers www.mesasoftware.com

"BetterSystemTrader is an amazing resource of valuable information, Andrew is always able to ask the right question at the right time to deliver the best content possible, a must listen for anybody interested in trading!"

Andrea Unger skilledacademy.com

"Better System Trader is an invaluable resource that I've gleaned tremendous amounts of information from over the years. I wouldn't be the trader I am today without having found this resource. Please keep up the great work."

David Bergstrom buildalpha.com

"I get some of my best ideas from listening to other market participants and Better System Trader is a great resource of conversations with trading professionals."

Michael Cook thetradingmasterclass.com

"The right questions to the right people at the right time."

Nick Radge thechartist.com.au

"Andrew is a master at asking difficult and relevant questions."

Perry Kaufman perrykaufman.com

"Andrew loves traders and draws the most informative and engaging responses out of them."

Art Collins artcollinstrading.com

"If you have been looking for that little extra nugget to transform your trading, you will find it at Better System Trader."

Laurent Bernut alphasecurecapital.com

"Professional, to the point and relevant content. Better System Trader should be followed by anyone with an interest in our field."

Andreas Clenow followingthetrend.com

"What really makes the BST podcast stand out is that Andrew is also an experienced trader and can therefore provide incomparable quality and depth in any interview. Thank you for endless inspiration."

Tomas Nesnidal systemsontheroad.com

"Andrew has interviewed some great people and I have picked up some useful new insights. His integrity, trading knowledge, light-hearted attitude, curiosity and enthusiasm shine through and enhance each of his podcasts. Given the quality of the information and the price (free!) any serious trader would have to be a fool to not take advantage. Keep it up my friend."

Bob Pardo pardocapital.com

"Knowledgeable and credible guests talk about useful topics in an interesting and engaging discussion. What's not to like?"

Robert Carver systematicmoney.org

"I really enjoy listening to the Better System Trader podcasts. Even though I am in the industry for over 15 years, there is always something new to learn from these. Thank you!"

Martin Lembak striker.com

"If I had to listen to one trading podcast only, it would be Better System Trader. I am always learning new things and getting research ideas from Andrew’s guests."

Cesar Alvarez alvarezquanttrading.com

Trading stocks, options, futures and forex involves significant risk of loss and is not suitable for everyone. Past performance is not necessarily indicative of future results.

© 2015 Better System Trader · Disclaimer & Terms · Privacy Policy · Sponsor