Every systematic trader has a graveyard of backtests that looked brilliant in Excel and died on the first day of live trading. The Proving Grounds is where you learn to tell the difference between a robust edge and a statistical phantom.
What Is a Backtest?
A backtest is a simulation. You apply a set of rules to historical price data and measure what would have happened if you had traded those rules in the past. It is not a prediction. It is not a guarantee. It is a hypothesis test — and like all hypothesis tests, it can be rigged.
The purpose of a backtest is not to prove your strategy works. The purpose is to fail fast. A good backtest kills bad ideas before they cost real money. A bad backtest seduces you into deploying a strategy that never had an edge — it only had parameters that happened to fit the past.
The difference between a backtest and a lie is about ten minutes of honest scrutiny.
The Four Sins of Backtesting
There are four ways to corrupt a backtest. Every ruined strategy commits at least one. The disciplined captain checks for all four before risking a single dollar.
Sin 1: Curve-Fitting (Over-Optimisation)
Curve-fitting is the original sin. You test 500 parameter combinations. You pick the one with the highest return. You declare victory. What you have actually done is fit the noise.
Imagine you flip a coin 100 times and build a model that predicts heads based on the phase of the moon, the temperature, and what you had for breakfast. If you test enough variables, one combination will appear to predict the coin flips perfectly. It is not magic. It is spurious correlation — and it will fail on the next 100 flips.
Markets are noisier than coins. A strategy with a 50-day moving average might look brilliant while a 49-day or 51-day version looks mediocre. If you optimise to the 50-day because it backtests best, you have curve-fitted. The 50-day has no special properties. It just happened to catch a few lucky trends in your sample.
The defence: Use out-of-sample testing. Split your data in half. Optimise on the first half. Test on the second half. If the strategy fails on data it never saw, it was curve-fitted. A robust edge should work across a range of parameters, not a single magic number.
Sin 2: Look-Ahead Bias
Look-ahead bias is using information you would not have had at the time of the trade. It is the backtest equivalent of time travel.
Common forms include:
- Using today's close as today's entry price (you cannot know the close until the day ends).
- Rebalancing a portfolio based on end-of-month rankings calculated using end-of-month data (you cannot trade at the close using the close).
- Filtering stocks by market cap using today's market cap (a stock that was large-cap in 2010 might have been small-cap in 2000).
Look-ahead bias is insidious because it often produces spectacular results. A strategy that knows the future always wins. The captain who catches this sin early saves himself from a very public failure later.
The defence: Trade on a lag. Use yesterday's close for today's signal. Use last month's data for this month's rebalance. If your data vendor provides point-in-time databases, use them. If not, build your own lag logic and audit every variable.
Sin 3: Survivorship Bias
Survivorship bias is testing your strategy only on stocks that still exist today. It is like evaluating a hospital by only interviewing the patients who survived.
If you backtest a strategy on the current S&P 500 constituents going back to 2000, you are ignoring the companies that left the index — Enron, WorldCom, Lehman Brothers, Bear Stearns, Circuit City. Your strategy looks brilliant because it never had to hold the disasters. In real time, it would have.
Survivorship bias inflates returns by 2–4% annually in most equity backtests. That is the difference between a viable strategy and a fantasy.
The defence: Use historical constituent lists. If you cannot access them, at least test on a broad index (like the Russell 3000) and assume your strategy would have held some delisted stocks. Delistings and bankruptcies are part of the cost of doing business.
Sin 4: Transaction Cost Amnesia
The most dangerous backtests ignore friction. Commissions, slippage, borrow costs, market impact, and taxes turn a 20% annual return into a 12% return — or a loss.
A strategy that trades 50 times a day on mid-cap stocks will generate enormous slippage. The backtest assumes you get the mid-price. In reality, you get the ask on entry and the bid on exit. The spread on a $50 mid-cap stock might be 5 cents. On 50 trades a day, that is $2.50 per day, $625 per year, per stock. Multiply by a portfolio of 100 stocks and you have erased your edge before the market opens.
The defence: Model costs explicitly. Use half the bid-ask spread as your slippage estimate. Add commission per trade. If shorting, add borrow cost. If rebalancing monthly, add market impact (larger for bigger portfolios). A conservative cost model is the difference between a live strategy and a dead one.
The Proving Grounds Protocol
A backtest is not a single number. It is a process. The Proving Grounds Protocol has seven stages. Skip any stage at your peril.
Stage 1: Define the Hypothesis Before Touching Data
Write down your strategy in plain English before you open a spreadsheet. What is the economic rationale? Why should this edge exist? Who is on the other side of the trade, and why are they willing to lose?
If you cannot answer these questions, you do not have a strategy. You have a pattern you noticed. Patterns without rationale are the raw material of curve-fitting.
If you torture the data long enough, it will confess to anything. — Ronald Coase
Stage 2: Build the Rules, Not the Results
Code your strategy with fixed parameters. Do not optimise yet. Use round numbers that make intuitive sense: a 20-day moving average, a 2× ATR stop, a 5% position size. The goal is to see if the concept works, not if the numbers are perfect.
Stage 3: Run the In-Sample Test
Test on your full historical dataset. Record everything: win rate, average win, average loss, Sharpe ratio, max drawdown, profit factor, number of trades, and the equity curve. This is your baseline. Do not celebrate. Do not despair. Just record.
Stage 4: Walk-Forward Analysis
Walk-forward analysis is the gold standard. Divide your data into segments. Optimise on segment 1, test on segment 2. Optimise on segment 2, test on segment 3. Repeat until you reach the present.
This mimics real life: you optimise your strategy based on recent data, then trade it forward. If the walk-forward results are stable, your edge is likely real. If they collapse, you have curve-fitted.
A simpler version: use a single out-of-sample period. Optimise on 2000–2015. Test on 2016–2024. If the strategy works in both periods with similar parameters, it is robust.
Stage 5: Parameter Sensitivity Analysis
Vary your parameters across a wide range. Plot the results. A robust strategy should show a plateau — a range of parameters where performance is stable. A curve-fitted strategy will show a spike — one perfect parameter surrounded by mediocrity.
If your 50-day moving average works but your 45-day and 55-day versions lose money, the 50-day is not magic. It is lucky. Walk away.
Stage 6: Monte Carlo Simulation
Monte Carlo simulation reshuffles your trade sequence 1,000 times and measures the distribution of outcomes. It answers a critical question: How much of my result was luck?
If your backtest shows a 20% annual return but Monte Carlo shows a 15% chance of a 30% drawdown, you have a risk problem. If Monte Carlo shows a 5% chance of break-even over five years, your edge is thinner than you think.
Monte Carlo does not predict the future. It shows you the range of possible futures. The captain who knows the range sails with appropriate ballast.
Stage 7: Paper Trade Before Live Capital
Paper trading is not a backtest. It is a execution test. Can you follow the rules when real money is not at stake? Does the signal arrive when you expect it? Are the fills reasonable? Does the slippage match your model?
Paper trade for at least three months. If you cannot follow the rules on paper, you will not follow them with real money. And if the paper results diverge from the backtest, investigate immediately. Something in your execution pipeline is different from your simulation.
Reading the Results
A backtest produces numbers. Here is how to read them without delusion.
Win Rate: A high win rate (above 60%) is suspicious. Most robust strategies win 40–55% of the time. If your win rate is 80%, check for look-ahead bias or data errors.
Profit Factor: Gross profit divided by gross loss. Above 1.5 is acceptable. Above 2.0 is strong. Below 1.2 is fragile — one bad month can erase a year.
Sharpe Ratio: Return per unit of risk. Above 1.0 is good. Above 1.5 is excellent. Below 0.5 suggests the edge is too thin for the volatility. Be wary of Sharpe ratios above 3.0 on short backtests — they are usually curve-fitted.
Max Drawdown: The worst peak-to-trough decline. Compare it to your expected return. A strategy that makes 15% annually with a 30% max drawdown is riskier than one that makes 12% with a 10% drawdown. The second strategy has a better Calmar ratio (return divided by max drawdown).
Number of Trades: A strategy with 12 trades in 10 years is not testable. You need at least 30–50 trades to draw statistical conclusions. Preferably 100+. The more trades, the more confident you can be that the result is not luck.
Equity Curve: Look at the shape. A smooth upward slope is ideal. A jagged line with long flat periods suggests the edge is intermittent. A curve that rises steeply in one period and flatlines in all others is curve-fitted.
The Red Flags
Some backtest results are automatic rejections. No second opinion required.
From the Logs
In Episode 05 — Charting Unknown Waters, Captain Elena spent six years as a breakeven discretionary trader before discovering systematic trading. Her first system — a 50-day / 200-day moving average crossover — worked because she backtested it to 1950 and found positive expectancy. But she also paper-traded it for eight months before going live. The Proving Grounds would have shown her whether her Sharpe ratio improvement was real or curve-fitted — and whether her eight-month paper test was long enough to validate the edge.
Related Guides
Celestial Mechanics
Build systematic, rules-based strategies from first principles.
Dead Reckoning
Track your fleet's true performance with the right metrics.
The Astrolabe
Expectancy, profit factor, and the statistical validity of your edge.
The Helm Response
Rolling Beta, R-squared, and how your portfolio moves with the index.
The Moral
A backtest is a mirror, not a crystal ball. It shows you what the past looked like through your strategy's eyes. It does not show you the future. The captain who treats backtests as hypothesis tests — who seeks to disprove rather than confirm — builds strategies that survive contact with the market. The captain who treats backtests as validation builds strategies that survive only in Excel. Choose your mirror carefully.