• iTunes
  • Twitter
  • Facebook
  • Google+
  • Instagram
  • Youtube
  • Email
  • Home
  • Courses
  • eBooks
  • Library
  • ET Club
  • BST LIVE
  • Leasing
  • About
Better System Trader

Better System Trader

Better System Trader is the podcast and blog dedicated to systematic traders, providing practical tips from trading experts around the world.

051 – Strategy Evaluation with Dave Walton

http://traffic.libsyn.com/bettersystemtrader/BetterSystemTrader-Ep51-DaveWalton.mp3
Play in New Window|Download

davewalton

Today we’re covering a topic which can really be a concern for traders of all levels, from beginner to pro, and that is the topic of strategy evaluation.

  • Have you ever found that real-life performance does not match expected results?
  • Or perhaps you have a strategy that is stuck in a drawdown and wondering if it’s actually broken?

I’m sure we’ve all heard of data mining bias, over-optimization and curve fitting and the impacts this can have on our trading accounts.

We may be even using techniques such as Out Of Sample testing, Walk Forward Analysis, Monte Carlo analysis and a number of other measures to identify or reduce the impact of these issues, but do these approaches actually work? Are there limitations or dangers with these techniques? Are there better ways?

In this episode we talk to someone who evaluates trading systems for a living, plus his research into system evaluation techniques has won awards. The guest is Dave Walton.

Dave was the winner of the Wagner award in 2014 for a paper titled ‘Know your system – turning data mining bias to benefit through System Parameter Permutation’.

In our chat today we talk about the technique in his paper and how it can be applied to trading strategy evaluation. We also discuss some of the assumptions and limitations of the approach, and he shares with us some valuable insights he’s made since publishing the paper which have resulted in an updated approach he now considers a better alternative, so make sure you listen out for that.

‘The performance you can expect from your system is overstated – Dave Walton

Click To Tweet

Sponsor

Topics discussed

  • How the typical approaches to system development can introduce datamining bias without you knowing
  • The types of systems that can increase the chance of data mining bias and what to look for
  • How the method of splitting your out of sample data could be causing you to throw away good strategies
  • Out of sample, walk forward analysis and Monte Carlo – do they actually reduce data mining bias?
  • The problems with using Monte Carlo analysis to assess strategy performance and why it doesn’t protect from overfitting
  • System Parameter Permutation – how to use it, why use the median, parameter range selection and new insights since the SPP paper was published
  • How System Parameter Randomization solves some of the issues of System Parameter Permutation
  • Stochastic modelling and how it can be used to determine if a rule is adding value to your strategy

Resources mentioned in this episode

  • Dave can be contacted through his website statistrade.com
  • You can find a copy of Daves paper here: ‘Know Your System! – Turning Data Mining from Bias to Benefit Through System Parameter Permutation‘
  • Dave has been kind enough to provide us with an article that is essentially an SPP addendum, discussing some of the assumptions and limitations of SPP and explaining the System Parameter Randomization approach a little further, you can read it here.
  • Books mentioned in the show:

Watch this stochastic modelling presentation Dave made at the MTA symposium

In our chat Dave mentioned a presentation he made for the MTA, discussing the use of Stochastic Modeling in Trading System Development.

The MTA have kindly made it available for everyone to watch, you can view it here.

PresentationVideoScreenshot

Share a quote with your trading friends

When you fit historical data you’re fitting the noise as well

Click To Tweet

People don’t realize they’re inducing selection bias through their choice of instrument

Click To Tweet

When you fit historical data you’re fitting the noise as well

Click To Tweet

There are multiple factors that influence data mining bias

Click To Tweet

Transcript

Got A Question, Topic or Guest you want to see on the Podcast?

Do you have a specific question, topic or guest you’d like to see on a future podcast episode?

Click here and submit your suggestion for a chance to have it featured on an upcoming podcast episode.

Subscribe to Better System Trader and never miss another episode!

subscribe-with-itunes-button stitcher soundcloud android

Please support the podcast by giving an honest Rating/Review for the show on iTunes!

Episode Released:

12 June 2016

  • facebook
  • linkedin
  • twitter
  • google+
  • pinterest
  • stumbleupon
System Parameter Permutation - a...
052 - An 'unfair trading advanta...

Related Posts

  • 033 – Strategy Development & Money Management – Thomas Stridsman
  • 006 – Trading System Development – Dr Howard Bandy
  • 046 – Strategy Development with Perry Kaufman
  • 073 – Combining simple concepts to build robust strategies with Art Collins
  • System Parameter Permutation – a better alternative?System Parameter Permutation – a better alternative?

Comments

  1. Cesar Alvarez

    June 16, 2016 at 1:00 am

    Thanks for a great podcast. Lots of great information. I have been doing something like SPR for the last couple of years but it is great to see the concept formalized and more thought out. Is this the only documentation on SPR? I was talking to some students about my process a few weeks ago and I want to point them to more information on SPR.

    I have not had time yet to read the original paper or watch the MTA video. But I am very much looking forward to them.

    One question from the above. This quote:
    “For example, the system used in the SPP paper had four parameters and resulted in over 4,000 different combinations. It is unlikely you can get enough meaningful samples just using a single indicator.”

    I did a quick scan of the paper to find the rules. I am confused. Are you saying that 4000 permutations is not enough? Or just doing permutations on one value is not enough?

    The tricky part about SPR is deciding on the ranges. It is easy to fool oneself by picking narrow ranges.

    Thanks,
    Cesar

    • Dave Walton

      June 16, 2016 at 3:04 pm

      Hi Cesar and thanks for the feedback. As far as resources for SPR, I haven’t written or spoken much publicly about it until recently. The quick blog post here can be thought of as an addendum to my original NAAIM Wagner award paper which is referenced in the post. So if you want to point your students to a resource, they can download the original SPP paper and then read the blog post here on SPR which builds on that paper. Of course the MTA presentation is available also which is broader.

      As far as your question about 4000 permutations being enough, generally I would say yes. But you can test that in SPR by continuing to iterate until a reasonably steady state is reached. When I made that comment in the interview, I was trying to make the point that SPR should be used on a complete trading system with all rules involved so that the rule interaction effects are seen across the permutations/iterations.. People have tried to apply it to a single indicator like a moving average and end up with not many permutations and misleading results.

      And you are absolutely right that the tricky part of SPR is choosing the ranges. That’s why I mentioned that doing a sensitivity test is generally a good idea so as to not fool oneself. If the performance across the distribution of results really starts to drop off quickly when you widen ranges, that spells trouble.

      Thanks,
      Dave

Trackbacks

  1. Best Links of the Last Two Weeks | Quantocracy says:
    June 26, 2016 at 4:27 pm

    […] Strategy Evaluation with Dave Walton [Better System Trader] […]

Want to build robust trading strategies that actually work in live trading?

Click for more details

What pro traders are saying about BST

"Better System Trader is an invaluable resource that I've gleaned tremendous amounts of information from over the years. I wouldn't be the trader I am today without having found this resource. Please keep up the great work."

David Bergstrom buildalpha.com

"I get some of my best ideas from listening to other market participants and Better System Trader is a great resource of conversations with trading professionals."

Michael Cook thetradingmasterclass.com

"Professional, to the point and relevant content. Better System Trader should be followed by anyone with an interest in our field."

Andreas Clenow followingthetrend.com

"Andrew is a master at asking difficult and relevant questions."

Perry Kaufman perrykaufman.com

"This podcast provides unique information, from real traders, interviewed by an experienced trader."

Kevin Davey kjtradingsystems.com

"Knowledgeable and credible guests talk about useful topics in an interesting and engaging discussion. What's not to like?"

Robert Carver systematicmoney.org

"Listening to BST is a requirement for everyone on our team."

Bert Mouler profluentcapital.com

"BetterSystemTrader is an amazing resource of valuable information, Andrew is always able to ask the right question at the right time to deliver the best content possible, a must listen for anybody interested in trading!"

Andrea Unger skilledacademy.com

"The right questions to the right people at the right time."

Nick Radge thechartist.com.au

"If I had to listen to one trading podcast only, it would be Better System Trader. I am always learning new things and getting research ideas from Andrew’s guests."

Cesar Alvarez alvarezquanttrading.com

"Andrew has interviewed some great people and I have picked up some useful new insights. His integrity, trading knowledge, light-hearted attitude, curiosity and enthusiasm shine through and enhance each of his podcasts. Given the quality of the information and the price (free!) any serious trader would have to be a fool to not take advantage. Keep it up my friend."

Bob Pardo pardocapital.com

"What really makes the BST podcast stand out is that Andrew is also an experienced trader and can therefore provide incomparable quality and depth in any interview. Thank you for endless inspiration."

Tomas Nesnidal systemsontheroad.com

"Andrew seems to have a knack for getting the world's most famous traders to share valuable insights about how they trade for the benefit of the trading community."

Andrew Gibbs halifaxonline.co.nz

"I really enjoy the wide variety of topics and perspectives shared in the podcasts and with the interviews done by someone with real life trading experience"

Tim Rea thetradingmasterclass.com

"The key to better system trading is information.  As a system trader himself, Andrew has unique capability of asking insightful questions that extract the skills of each guest."

John Ehlers www.mesasoftware.com

"I really enjoy listening to the Better System Trader podcasts. Even though I am in the industry for over 15 years, there is always something new to learn from these. Thank you!"

Martin Lembak striker.com

"The Better System Trader podcasts have become an invaluable resource... There's no other podcast around that covers the subject in so much breadth and depth."

Alan Clement helixtrader.com

Trading stocks, options, futures and forex involves significant risk of loss and is not suitable for everyone. Past performance is not necessarily indicative of future results.

© 2015 Better System Trader · Disclaimer & Terms · Privacy Policy · Sponsor