At some point, every systematic trader faces the same question: is my strategy broken, or is it just going through a rough patch? Getting this wrong in either direction is costly. Stop a good strategy too early and you miss the recovery. Keep trading a genuinely broken strategy and the losses compound.
Kevin Davey has thought about this problem more rigorously than almost anyone. Before he became a full-time trader and World Cup Trading Championship winner, he spent 15 years in aerospace engineering, overseeing quality control for jet aircraft fuel pumps. The standards in that field are extreme, rejecting parts that are out of tolerance by one ten-thousandth of an inch. The statistical tools he used there, particularly statistical process control, turned out to translate directly into trading system monitoring.
In this episode Kevin goes through six approaches for detecting when a strategy is failing, from simple drawdown rules to Monte Carlo simulation, with an honest assessment of where each one breaks down.
Watch the full episode below, then read on for the complete breakdown.
The aerospace analogy: statistical process control in trading
Kevin’s background in manufacturing quality provides the conceptual framework for his entire approach. In aerospace, every part produced is measured and plotted on a statistical process control chart. Rules define what “in control” looks like and what “out of control” looks like. When a part falls outside the acceptable range, action is taken before the problem gets worse. The system does not wait until the plane falls out of the sky.
Applied to trading, the same logic holds. You measure your strategy’s outputs (trade results), track them over time, and define in advance what “out of control” looks like. When the strategy’s live performance triggers your threshold, you act, whether that means pausing, reviewing, or stopping the strategy entirely.
The key word is “in advance.” The biggest mistake he sees traders make is not having any criteria defined until they are already in the middle of a drawdown. At that point, emotions are involved, the decision is reactive, and panic takes over. The time to decide what you will do is before the problem arrives.
Method 1: Drawdown-based rules
The most common approach, and the one Andrew admits he used in his earlier trading, is watching the drawdown figure from the backtest report. If the worst historical drawdown was $15,000, some traders stop when they hit that level. Others use 1.5x or 2x that figure. Others use half of it as a more conservative trigger.
Kevin considers this a valid approach, and he still uses it on some systems. The drawdown number from your backtest does tell you something real. If you are making a new all-time worst drawdown, that is meaningful information. It is genuinely a signal worth paying attention to.
The limitation is that the single drawdown figure from your backtest is one data point from one sequence of trades. The trades could have come out in any order, and in a different order they would have produced a completely different drawdown figure. That number is not a law of physics. It is a historical artifact.
Method 2: Consecutive losing trades
Another approach is setting a threshold on consecutive losses. If your backtest never produced more than five consecutive losing trades and you suddenly hit seven, that is a signal worth taking seriously.
Kevin mentioned this as one of the simpler metrics to monitor in real time. You do not need any special software. You just track your trade results and watch the streak. If the longest losing streak in your backtest was five, and you are now at seven with no signs of improvement, something may have changed.
The limitation here is similar to the drawdown approach: you are comparing live performance to a single historical outcome. The Monte Carlo analysis discussed below shows why this single-path view understates the uncertainty.
Method 3: Monte Carlo simulation for probability-based thresholds
Monte Carlo simulation is the technique Kevin discusses in the most depth, because it is the most rigorous. The concept is straightforward even if the implementation requires some work.
Your backtest produces one equity curve, one sequence of trades in one order. That sequence could have happened in any order. If you take all the trade results, write each one on a piece of paper, throw them in a hat, and draw them out randomly, you get a different equity curve each time. Do this 1,000 times and you get 1,000 possible equity curves, all built from the same underlying trade distribution.
The range of those curves tells you something important. Some will show much better performance than your backtest. Some will show much worse. The worst-case outcomes from this range are more relevant for setting your stop criteria than the single historical drawdown from your backtest. If 5% of the simulated curves result in a loss of $20,000 or more, then a $20,000 drawdown is not necessarily proof your strategy is broken. It is within the expected distribution.
Kevin applies this to the statistical process control framework from his engineering background. The Monte Carlo output gives you a probability distribution of outcomes, and you can define your “out of control” threshold based on that distribution rather than on a single historical number.
The limitations of Monte Carlo analysis
Kevin is direct about where Monte Carlo breaks down. There are three main issues:
- Garbage in, garbage out: Monte Carlo analysis is only as good as the data you feed it. If your backtest is over-optimised or uses in-sample data without proper walk-forward validation, the trade results going into the simulation are already too good. The Monte Carlo output will be correspondingly optimistic and set thresholds that are too loose.
- Market regime changes: The simulation assumes the future trade distribution resembles the past. If the market undergoes a fundamental shift, historical trades may no longer be representative. This is not a reason to avoid Monte Carlo, but it is a reason to revisit your assumptions when market structure changes materially.
- Selection bias in backtests: The strategy you chose to trade was probably the best of several you tested. It may have outperformed partly due to good luck in the test period. Monte Carlo does not correct for this. Your actual future performance will likely be somewhat below what even a clean backtest shows.
Method 4: Performance metrics out of norm
Beyond drawdown and consecutive losses, Kevin monitors a range of metrics for deviations from historical norms. The idea is the same as statistical process control: define what “normal” looks like, and flag anything that falls outside that range.
Metrics to watch include:
- Win rate deviating significantly from historical average
- Average win or average loss shifting in size
- Profit factor declining over a rolling window
- Number of losing months in a row exceeding historical maximums
Any one of these moving out of the expected range is not necessarily a cause to stop trading. But several of them moving in the same direction at the same time is a stronger signal.
The most important rule: decide before you need to
The theme running through every method Kevin discusses is the importance of pre-commitment. It does not matter much which specific method you choose, as long as you choose one, write it down, and commit to it before you start trading.
The traders who get into trouble are the ones who have no criteria defined. When the drawdown arrives, they try to make the decision in the moment, under pressure, with real money at stake. That is when panic wins. That is when traders either hold on too long because they cannot accept the loss, or stop out right before the recovery because the pain becomes unbearable.
Having a written plan does not make the decision easy. Kevin acknowledges that turning off a strategy is always hard. But it makes the decision clear. When the trigger condition is hit, you act. You do not negotiate with yourself. You do not wait to see if the next few trades are better. You act on the plan you made when you were thinking clearly.
Method 5 and 6: Walk-forward results and out-of-sample validation
Two additional monitoring approaches involve tracking live performance against walk-forward expectations. If your strategy was developed using walk-forward optimisation, you have an out-of-sample performance estimate for each walk-forward window. Tracking how your live results compare to those estimates gives you a benchmark that is more honest than in-sample backtest figures.
A strategy that is consistently underperforming its walk-forward estimates is telling you something. It may still be within normal variation, which Monte Carlo can help you assess. But persistent underperformance relative to a properly conducted walk-forward analysis is a more concerning signal than simply underperforming against a naive backtest.
The broader point is that monitoring a live strategy should use the same rigour that went into building it. If you built your strategy using out-of-sample testing and statistical validation, your monitoring criteria should reflect that level of rigour too.
Related episodes
- Kevin Davey on system development, the best systems, and backtesting
- Market regime techniques for systematic traders with Cesar Alvarez
- How to optimize strategies for robustness with John Ehlers
- Consistent equity growth through diversification with Nick Radge
Want to build more robust trading systems that stand up to real market conditions? Subscribe to the Better System Trader podcast for weekly interviews with the world’s top systematic traders and quantitative researchers.
