Perry Kaufman is one of the most prolific authors in systematic trading, with over 14 books to his name and decades of practical experience building algorithmic trading systems. His foundational text, Trading Systems and Methods, is a comprehensive reference used by practitioners across the industry. In this episode – his second appearance on BST – he discusses the book he wrote specifically for strategy developers: A Guide to Creating a Successful Algorithmic Trading Strategy.
The book grew out of questions Perry received consistently at presentations – the same questions every time, regardless of what the presentation was about. Position sizing, testing methodology, optimization, evaluation of results. He gathered them, organised them into a logical sequence, and wrote the guide he wished had existed earlier. This episode covers the most important of those questions in detail.
Watch the full episode below, then read on for the complete breakdown.
The Most Common Question – Equal Risk Across All Positions
The single most common question Perry receives is about position sizing, and it always traces back to the same misunderstanding: traders do not ensure that every position carries the same risk.
“You need to be sure that every position you put on has the same risk as every other position. If one has more risk than another, it has to have a higher return for your return-to-risk ratio to be maintained. I can’t predict which of my next 10 trades will be most profitable. So I use equal risk.”
His approach: equal risk, equal everything. Not equal dollar size, which ignores volatility differences between instruments, but equal risk – the same expected loss on each position if the stop is hit. For stocks, the simplest implementation is dividing your investment equally across all positions and dividing by the stock price. For futures, where volatility varies significantly across contracts, adjusting position size by ATR or dollar volatility is feasible and appropriate.
Start With a Sound Premise – Before Any Testing
Perry is explicit about what separates good strategy development from random data mining: you must have a sound premise before you start testing. A sound premise is a reason to believe a particular pattern or relationship exists in markets – rooted in economic logic, not data fitting.
“I believe trend following works in the long term, not the short term. The real trends are created by central banks when they raise or lower rates slowly. Those trends are long-term. A five-day moving average is not likely to work. You need to decide first: am I a long-term or short-term trader? Then pick your parameter ranges accordingly.”
Examples of sound premises: long-term trend following (tracks economic policy cycles through interest rates, then FX, then equities), pairs trading (two competing companies in the same sector tend to move together), and seasonality (recurring calendar patterns driven by consistent fundamental cycles).
Without a premise, optimisation is just curve fitting. With a premise, testing becomes validation – you are checking whether your prior belief is confirmed by the data, not searching for whatever happens to have worked historically.
How to Select Parameter Ranges for Testing
Perry’s approach to parameter range selection follows directly from the premise. If you believe trend following works over the long term and not the short term, you define your moving average test range as something like 60 to 150 days – not 5 to 250. The five-day results are not relevant to your premise and should not be included.
He also challenges the standard practice of testing parameters in linear increments. The difference between a 65-day and a 70-day moving average is small in percentage terms. The difference between a 10-day and a 15-day is 50%. When you test in equal absolute increments, the longer end of the range appears falsely stable because adjacent results are more similar to each other.
Perry’s solution: test in percentage increments. Double each successive parameter – 30, 60, 120. Or step by 50%: 30, 45, 60, 90, 120. Fewer tests, but a more realistic distribution of results. The slow end does not look artificially better just because adjacent parameter values produce similar outcomes.
What to Look for When Analysing Optimisation Results
The single most important point Perry makes about analysing optimisation outputs: do not look for the best result. The best result is usually the one that happened to benefit from the most price shocks or lucky timing in the historical period. That performance will not repeat.
His alternative: use the average of all test results as your benchmark. If the average of all parameter combinations produces a good return at acceptable risk, the strategy is robust. If only a narrow cluster of parameters looks good, the strategy is overfitted.
“I judge my performance by the average of all the test results in my optimisation. If the average of all tests went up when I added a new filter or rule, it was a success. If only a few good results improved, that’s not what I’m looking for. I’m looking for something generalised.”
The average also becomes his realistic expectation for live trading. He does not expect to achieve the performance of the best-parameter combination in the test. He expects to achieve roughly the average across parameters – because he does not know in advance which parameter set will perform best in the next period.
Using Multiple Parameter Sets to Build a Portfolio of One Strategy
Perry extends this logic further: rather than picking a single optimal parameter set, he trades multiple parameter sets from the same optimisation simultaneously. This is portfolio diversification within a single strategy.
A single-parameter strategy is like a single-stock portfolio – maximum exposure to the specific risks of that parameter set. Four or five different parameter sets from the same optimisation are less correlated with each other and produce a smoother combined result. The individual sets need not be from the same sector – in strategy terms, they need not be tuned to the same speed or frequency.
This approach also eliminates the decision problem of parameter selection. Instead of trying to pick the best parameter, you spread across a set of reasonable parameters and let the combination do its work over time.
Price Shocks and Why the Best Backtest Is Misleading
Perry flags one of the most insidious sources of overfitting: price shocks. The parameter set that produced the best backtest result is often the one that was on the right side of the most significant one-day price moves in the historical data. Those shocks – news-driven, unexpected, unrepeatable in their exact form – cannot be predicted and cannot be expected to favour the same parameter in the future.
The rigorous approach is to go back through the trades and remove the profits and losses from one-day price shocks, then re-evaluate which parameter sets are genuinely best. Most traders will not do this. The practical alternative is the average-performance approach: if the average across all parameters is good, the strategy’s edge is not dependent on any single cluster of lucky trades.
When a New Rule Actually Improves a Strategy
As you add filters and rules to a strategy during development – volatility filters, regime filters, entry conditions – the test for whether the addition is genuine improvement or noise is straightforward using Perry’s framework. Run the full optimisation with and without the new rule. Compare the average performance across all parameter combinations.
If the average goes up and the distribution of results improves, the new rule added something real. If only the peak result improved but the average did not, the new rule is fitting to noise. It made the best case better but did not help the overall strategy. That is a red flag, not a green light.
Get the show notes & transcript
Related episodes
- How to Avoid Trading Strategies That Degrade Quickly with Timothy Masters
- Trading Champion Andrea Unger on Creating Trading Strategies That Work
- Ralph Vince on Position Sizing, Optimal f, Diversification and Risk
Want more on algorithmic strategy development and systematic trading? Subscribe to the Better System Trader podcast for weekly interviews with the world’s top systematic traders and quantitative researchers.
