• iTunes
  • Twitter
  • Facebook
  • Google+
  • Instagram
  • Youtube
  • Email
  • Home
  • Courses
  • eBooks
  • Library
  • ET Club
  • BST LIVE
  • Leasing
  • About
Better System Trader

Better System Trader

Better System Trader is the podcast and blog dedicated to systematic traders, providing practical tips from trading experts around the world.

Stitching data for a more ‘balanced’ backtest

When traders set in-sample and out-of-sample periods for their backtests, it’s common just to pick some dates that split the data into a pre-defined percentage or range.

The trouble with this approach is that it often doesn’t take into account the type of market environments that exist in those periods. For example, your in-sample period may be very bearish, and if your out-of-sample period is more bullish then the backtest results could be out of balance, potentially yielding poor out-of-sample performance or giving unrealistic results.

You could even be potentially throwing away a good strategy purely because of the way you’ve split the data!

What can traders to do address this issue?

Last week I received this excellent question from BST listener Theo:

“I’ve been a great fan of your podcast and I have literally recommended it to everyone I know in the trading world.

I have a question I would truly appreciate if you could share some insight.

You have mentioned a couple of times the approach of stitching together bearish, bullish or sideways data for a backtesting.

I’m a bit confused on how this approach takes place. Does this approach mean you stick a whole insample period with bullish data and then leave the out of sample intact and repeat that for bearish and sideways market conditions as separated backtests?

Also does the out-of-sample has to be equally bullish/bearish or sideways in that case or just remains intact as your most recent data?”

Firstly, thanks for the support Theo, glad you’re enjoying the podcast!

Now, onto your question, which a really good one.

I originally heard about this approach from World Cup Trading Champion Michael Cook, who was a guest on episode 39 of the podcast, so I thought it would be best to get it directly from the source and asked Michael for more details on how he actually does this.

He very kindly shared it in a step-by-step process:

“I hope this will cover both the questions. The way I do it is like this:

  1. I take a specific product for the entire period to be tested and print its price history on a high definition graph, say 2006 to yesterday.
  2. Visually mark all the periods I would regard as being in one of six categories, as bullish, bearish, rangebound. Each being volatile or non volatile,
  3. Then I allocate each of those six market conditions to one of two ‘pools’. Pool 1 becomes in sample for testing, pool 2 is my out of sample. If there are 5 bullish volatile periods, 3 would probably go into the in-sample pool, the other two into out of sample,
  4. In this way, as a by-product, some of the most recent data will be in-sample and some of the most recent data will be out of sample,
  5. Then each of these periods is put into a single function which I call e.g. SoybeansDatesTest. By doing this, the function can be true or false and this function is incorporated into every entry rule of each strategy being developed. So of course an entry may be within one period and by the time that specific trade exits, it could be in a different period. But as the function is simply an entry filter, the system wont exit anything just because the entry condition has changed during the trade (incidentally, just as my system wouldn’t exit in the real world if underlying conditions suddenly changed as it probably wouldn’t realise until later!)

That’s the basic process.

Of course, it is unscientific in that I don’t think anything will be gained by generating the period categorisation with an algorithm – I am a great believer in being led by data, but not being its slave, so I am fine to do it manually. This exercise is a bit manual and laborious, but once, say SoybeansDatesTest is done, that probably wont need to be updated for a year and in exactly the same way I have NatGasDatesTest, CrudeDatesTest etc etc”

It’s an interesting approach to splitting data, which could potentially give you more ‘balanced’ backtest results by intentionally controlling the different types of market environments in a backtest, rather than being randomly subjected to whatever market environments just happen to occur between a date range.

If you’d like to read more about Michael’s approach to trading, along with other podcast guests too, check out the ‘Interviews with Pro Traders’ series here.

Cheers,
Andrew.

Released:

4 April 2018

  • facebook
  • linkedin
  • twitter
  • google+
  • pinterest
  • stumbleupon
144 - Building dynamic trading s...
The Apple Portfolio Strategy

Related Posts

  • 033 – Strategy Development & Money Management – Thomas Stridsman
  • Why do professionals usually win?
  • 006 – Trading System Development – Dr Howard Bandy
  • 073 – Combining simple concepts to build robust strategies with Art Collins
  • 129 – Does it actually make sense to ALWAYS use a stop loss?

Trackbacks

  1. Quantocracy's Daily Wrap for 04/04/2018 | Quantocracy says:
    April 5, 2018 at 3:19 pm

    […] Stitching data for a more ‘balanced’ backtest [Better System Trader] […]

Want to build robust trading strategies that actually work in live trading?

Click for more details

How do you REALLY know your strategies aren't just over-optimized rubbish?

Get this handy optimization checklist from veteran trader Bob Pardo, and make sure you haven’t made any of these mistakes that could lead to strategy failure.

Click to download now

What pro traders are saying about BST

"The right questions to the right people at the right time."

Nick Radge thechartist.com.au

"I really enjoy listening to the Better System Trader podcasts. Even though I am in the industry for over 15 years, there is always something new to learn from these. Thank you!"

Martin Lembak striker.com

"BetterSystemTrader is an amazing resource of valuable information, Andrew is always able to ask the right question at the right time to deliver the best content possible, a must listen for anybody interested in trading!"

Andrea Unger skilledacademy.com

"Andrew is a master at asking difficult and relevant questions."

Perry Kaufman perrykaufman.com

"Knowledgeable and credible guests talk about useful topics in an interesting and engaging discussion. What's not to like?"

Robert Carver systematicmoney.org

"If I had to listen to one trading podcast only, it would be Better System Trader. I am always learning new things and getting research ideas from Andrew’s guests."

Cesar Alvarez alvarezquanttrading.com

"Listening to BST is a requirement for everyone on our team."

Bert Mouler profluentcapital.com

"The Better System Trader podcasts have become an invaluable resource... There's no other podcast around that covers the subject in so much breadth and depth."

Alan Clement helixtrader.com

"I really enjoy the wide variety of topics and perspectives shared in the podcasts and with the interviews done by someone with real life trading experience"

Tim Rea thetradingmasterclass.com

"If you have been looking for that little extra nugget to transform your trading, you will find it at Better System Trader."

Laurent Bernut alphasecurecapital.com

"I get some of my best ideas from listening to other market participants and Better System Trader is a great resource of conversations with trading professionals."

Michael Cook thetradingmasterclass.com

"What really makes the BST podcast stand out is that Andrew is also an experienced trader and can therefore provide incomparable quality and depth in any interview. Thank you for endless inspiration."

Tomas Nesnidal systemsontheroad.com

"Andrew loves traders and draws the most informative and engaging responses out of them."

Art Collins artcollinstrading.com

"The key to better system trading is information.  As a system trader himself, Andrew has unique capability of asking insightful questions that extract the skills of each guest."

John Ehlers www.mesasoftware.com

"Being a trader himself, Andrew drives insightful conversations that uncover gems of practical knowledge that can improve your trading today"

Kris Longmore robotwealth.net

"This podcast provides unique information, from real traders, interviewed by an experienced trader."

Kevin Davey kjtradingsystems.com

"Better System Trader is an invaluable resource that I've gleaned tremendous amounts of information from over the years. I wouldn't be the trader I am today without having found this resource. Please keep up the great work."

David Bergstrom buildalpha.com

Trading stocks, options, futures and forex involves significant risk of loss and is not suitable for everyone. Past performance is not necessarily indicative of future results.

© 2015 Better System Trader · Disclaimer & Terms · Privacy Policy · Sponsor