Today we’re covering a topic which can really be a concern for traders of all levels, from beginner to pro, and that is the topic of strategy evaluation.
- Have you ever found that real-life performance does not match expected results?
- Or perhaps you have a strategy that is stuck in a drawdown and wondering if it’s actually broken?
I’m sure we’ve all heard of data mining bias, over-optimization and curve fitting and the impacts this can have on our trading accounts.
We may be even using techniques such as Out Of Sample testing, Walk Forward Analysis, Monte Carlo analysis and a number of other measures to identify or reduce the impact of these issues, but do these approaches actually work? Are there limitations or dangers with these techniques? Are there better ways?
In this episode we talk to someone who evaluates trading systems for a living, plus his research into system evaluation techniques has won awards. The guest is Dave Walton.
Dave was the winner of the Wagner award in 2014 for a paper titled ‘Know your system – turning data mining bias to benefit through System Parameter Permutation’.
In our chat today we talk about the technique in his paper and how it can be applied to trading strategy evaluation. We also discuss some of the assumptions and limitations of the approach, and he shares with us some valuable insights he’s made since publishing the paper which have resulted in an updated approach he now considers a better alternative, so make sure you listen out for that.
- How the typical approaches to system development can introduce datamining bias without you knowing
- The types of systems that can increase the chance of data mining bias and what to look for
- How the method of splitting your out of sample data could be causing you to throw away good strategies
- Out of sample, walk forward analysis and Monte Carlo – do they actually reduce data mining bias?
- The problems with using Monte Carlo analysis to assess strategy performance and why it doesn’t protect from overfitting
- System Parameter Permutation – how to use it, why use the median, parameter range selection and new insights since the SPP paper was published
- How System Parameter Randomization solves some of the issues of System Parameter Permutation
- Stochastic modelling and how it can be used to determine if a rule is adding value to your strategy
Resources mentioned in this episode
- Dave can be contacted through his website statistrade.com
- You can find a copy of Daves paper here: ‘Know Your System! – Turning Data Mining from Bias to Benefit Through System Parameter Permutation‘
- Dave has been kind enough to provide us with an article that is essentially an SPP addendum, discussing some of the assumptions and limitations of SPP and explaining the System Parameter Randomization approach a little further, you can read it here.
- Books mentioned in the show:
Watch this stochastic modelling presentation Dave made at the MTA symposium
In our chat Dave mentioned a presentation he made for the MTA, discussing the use of Stochastic Modeling in Trading System Development.
The MTA have kindly made it available for everyone to watch, you can view it here.
Share a quote with your trading friends
Got A Question, Topic or Guest you want to see on the Podcast?
Do you have a specific question, topic or guest you’d like to see on a future podcast episode?
Click here and submit your suggestion for a chance to have it featured on an upcoming podcast episode.
Get the Transcript
Subscribe to Better System Trader and never miss another episode!
Please support the podcast by giving an honest Rating/Review for the show on iTunes!