Timothy Masters holds a degree in statistics and spent his early career doing Department of Defense work on missile guidance systems. His entry into trading came through a friendship with an electrical engineer who connected him to a commodities trading firm, where he met David Aronson. The two formed a partnership that combined Aronson’s trading experience with Masters’ statistical expertise – and that combination has driven decades of research into what makes trading strategies actually work, and what makes them fail.
Masters is the author of several books on statistically sound indicators and permutation tests for financial markets. In this episode, he focuses on the single biggest cause of strategy failure: the reuse of out-of-sample data. His framework for testing – built around permutation tests – gives traders a way to verify whether their system found a real edge or simply memorised noise.
Watch the full episode below, then read on for the complete breakdown.
The Most Common Error – Reusing Out-of-Sample Data
When asked for the most common error strategy developers make, Timothy’s answer is immediate: reusing out-of-sample data. This one mistake is the root cause of the vast majority of strategies that test well in development but fail when traded live.
He illustrates with a hypothetical. Two developers, John and Mary, each build a system independently. You withhold the most recent year of data so they cannot use it. They each develop their systems and submit them. You test both on the withheld year and find Mary’s system performs better. You decide to trade Mary’s system.
The problem: the moment you compared John and Mary’s performance on the same out-of-sample data and chose the better one, you introduced bias. You are now trading “the better of John and Mary’s systems” – not Mary’s system. That out-of-sample period has been used to make a decision, which means it is no longer a clean test.
“The out-of-sample performance is the sum of two components: the intrinsic power of the system to find real patterns, and luck on that particular data set. The moment you introduce competition using the same out-of-sample data, you optimistically bias your results.”
The Contamination Problem Scales With Every Decision
Timothy extends this logic to every decision made using out-of-sample data. Every time you look at the out-of-sample results and decide to keep or discard a variant, you have used that data. It is no longer truly out-of-sample. This applies to parameter choices, indicator selections, market choices, and filter decisions.
The consequence: most published out-of-sample results in trading research are contaminated. The researchers have unconsciously reused the data through dozens of decision points during development. The result looks like a clean test but is not. This is why systems that perform beautifully in backtests often fail immediately when traded live.
Once out-of-sample data is used, it becomes training data for all practical purposes. The fix is to treat out-of-sample data as precious – to be spent at most once, on a final evaluation of a finished system that has not been modified at all during development.
Permutation Tests – How to Test Without Using Your Out-of-Sample Data
Timothy’s solution is the permutation test. Instead of using withheld data to evaluate whether a system found a real edge, you permute your training data – shuffle it in ways that destroy temporal patterns – and retrain the system on each permuted version.
If the system has genuinely found a real pattern in the market, it will outperform the permuted versions. The real data has predictive structure. The permuted data does not. A system that sees actual patterns will do better on real data than on noise.
If the system has simply overfit – memorised the noise of the training period – then the same overfitting will occur on the permuted data. There is no reason for the real system to outperform the permuted versions. The original performance will not stand out.
“If our system is really good, it’s going to have an edge because the permuted data sets don’t have real patterns. Our truly good system will stand out. If our good performance is due to overfitting and learning noise, the same overfitting will happen on permuted data too.”
Three Permutation Tests Every Developer Should Know
Timothy outlines three distinct applications of permutation testing, each addressing a different question in the development process.
The first is used during development, before touching out-of-sample data. You permute the training data many times, retrain the system on each version, and compare the original system’s performance to the permuted performance distribution. If the original system consistently outperforms the permuted versions, it has found something real. If not, it is overfitting – and you find this out early, before wasting time on a system that will fail.
The second is used when evaluating a finished system on out-of-sample data. Good out-of-sample performance is necessary but not sufficient. You also need to establish that this performance could not easily have occurred by chance. Permuting the out-of-sample period many times and comparing your system’s performance to the resulting distribution answers that question. If your system’s result is in the top few percent of all permuted results, the probability of the performance being purely lucky is low.
The third – the most complex – tests the quality of your development process itself, not just the resulting system. The logic: if your process is sound, applying it to permuted data should consistently fail to produce good systems. If your process produces good-looking systems even on random data, the process itself is overfitting, and any system it generates is suspect.
The Two Types of Model Error – and Why One Is Invisible
Timothy frames overfitting as one of two types of model error. The first is underfitting: the model is too weak to find the patterns in the data. The second is overfitting: the model is too strong and it finds spurious patterns in noise.
Underfitting is easy to detect – the performance will be poor and the developer will know something is wrong. Overfitting is dangerous precisely because it looks like success. The performance metrics are excellent. The equity curve is smooth. The Sharpe ratio is high. Nothing signals a problem until the system is traded live and immediately degrades.
This asymmetry is why permutation tests matter so much. They reveal overfitting during development, before any real money is at risk, using only the training data. No precious out-of-sample period is consumed.
When to Retrain vs When to Keep a Fixed System
A practical question Timothy addresses: if you have a trained system and performance starts declining in live trading, should you retrain it on updated data or keep trading the original?
His answer depends on the cause of the degradation. If the market has genuinely changed – the pattern the system was capturing no longer exists – retraining on recent data may help, but it consumes out-of-sample data that you cannot get back. If the degradation is just normal variance in a fundamentally sound system, retraining on the bad period will cause the system to learn the wrong lessons.
The permutation test framework can help distinguish these cases. A well-designed system should still produce stronger performance on real data than on permuted versions, even during a period of underperformance. If it no longer does, the edge may genuinely be gone. If it still does, the degradation is likely temporary.
Building the Statistical Intuition
Timothy’s broader message is that strategy development requires statistical discipline, not just market intuition. The failure modes – overfitting, out-of-sample contamination, selection bias – are statistical problems. They cannot be solved by better indicators or smarter pattern recognition. They require knowing what the numbers actually mean and what your test methodology actually tests.
His books on permutation tests and statistically sound indicators provide the technical framework. The core takeaway from this episode is simpler: treat your out-of-sample data as precious, test the system on permuted training data first, and do not trust performance metrics you cannot statistically validate.
Related episodes
- Perry Kaufman on Building Algorithmic Trading Strategies
- How to Build Mean Reversion Trading Strategies with Stefan Friedrichowski
- How Traders Can Compete in the Markets and Find Profitable Edges with Jason Strimpel
Want more on trading strategy development and avoiding degradation? Subscribe to the Better System Trader podcast for weekly interviews with the world’s top systematic traders and quantitative researchers.
