Building a Trading Edge: Backtesting Best Practices

The Overfitting Problem

Given enough parameters and enough optimisation cycles, any strategy can be made to fit historical data perfectly. A 20-parameter model with 5 years of daily data has approximately 1,250 data points — enough degrees of freedom for the model to memorise every twist and turn of the equity curve without learning anything about the underlying market structure.

Overfitting manifests as a strategy that performs spectacularly in-sample and collapses out-of-sample. The classic sign is a suspiciously smooth equity curve with no significant drawdowns in the backtest period. Real trading strategies have drawdowns. If the backtest doesn't, it has been curve-fitted.

The rule of thumb: the number of free parameters should be at most 1/10th the number of independent observations. A strategy with 5 parameters needs at least 50 independent trades in the backtest to have any statistical validity — and even that is marginal. The AI Trading Copilot's 80+ strategies from Kakushadze & Serur are each defined by 2–4 parameters, keeping the overfitting risk low.

Walk-Forward Analysis

Walk-forward analysis is the gold standard for strategy validation. Instead of optimising on the full dataset, the data is divided into sequential windows. The strategy is optimised on window 1 (in-sample), tested on window 2 (out-of-sample), then re-optimised on windows 1+2 and tested on window 3, and so on.

This mimics how the strategy would actually be deployed: you develop it on historical data, trade it forward, periodically re-optimise, and trade again. If performance degrades significantly in each out-of-sample window, the strategy is overfitted. If performance is consistent across windows (even if lower than in-sample), the edge is likely genuine.

The critical metric is the Walk-Forward Efficiency Ratio: the ratio of out-of-sample return to in-sample return, averaged across all windows. A ratio above 0.5 indicates a robust strategy. Below 0.3 suggests overfitting. The AI Trading Copilot's backtest engine implements walk-forward analysis with configurable window sizes and reports the efficiency ratio prominently.

Out-of-Sample Testing: The Final Exam

Walk-forward analysis validates parameter stability. Out-of-sample testing validates the strategy concept itself. The most rigorous approach is to reserve 20–30% of the available data as a 'hold-out set' that is never used during development — not for idea generation, not for parameter tuning, not for any purpose until the strategy is finalised.

This is harder than it sounds. Human nature leads to 'peeking' at the hold-out period, even unconsciously. If you know that 2022 was a bear market and your strategy is designed to perform in bear markets, you've already contaminated the hold-out set with survivorship of the strategy concept.

The best practice is to define the strategy concept and parameters on the in-sample period, document them, and then run the hold-out test exactly once. If the results are acceptable, proceed to paper trading. If not, do not go back and re-optimise — that turns the hold-out set into a second in-sample set and destroys its validity.

Survivorship Bias and Look-Ahead Bias

Survivorship bias occurs when your backtest universe contains only assets that survived to the present day. If you backtest a stock-picking strategy on the current S&P 500 constituents, you exclude every company that was in the index but went bankrupt, was acquired, or was delisted during the test period. These are precisely the stocks that would have generated the largest losses — and excluding them inflates the strategy's apparent performance.

The correction is to use point-in-time constituent lists. For any given date in the backtest, the universe should contain exactly the stocks that were available on that date, including those that subsequently failed. This data is harder to obtain and more expensive, but it is essential for honest backtesting.

Look-ahead bias is equally insidious: using information in the backtest that was not available at the time of the decision. Common sources include using adjusted close prices (which incorporate future dividend information), using the final revision of economic data (GDP, jobs numbers are revised multiple times), and using corporate data filed weeks after the reporting period.

• ·Use point-in-time universe constituents, not current-day lists
• ·Use unadjusted prices or explicitly handle dividend adjustments
• ·Use first-release economic data, not final revisions
• ·Ensure indicator calculations use only data available at bar close (barstate.isconfirmed in Pine Script)
• ·Account for data publication delays — SEC filings, earnings, economic releases

Transaction Costs and Slippage

A strategy that is profitable before costs may be unprofitable after them. Transaction costs include commissions, bid-ask spread, market impact (the price movement caused by your own order), and borrowing costs for short positions.

For retail traders, the minimum realistic cost assumption is the bid-ask spread plus commission. For liquid large-cap stocks, this might be $0.02–0.05 per share. For less liquid instruments, it can be $0.10–0.50 or more. For forex, the spread itself is the primary cost.

Slippage — the difference between the expected fill price and the actual fill price — is hardest to model. A backtest that assumes fills at the close price systematically understates slippage, especially for strategies that trade at the open, during news events, or in less liquid markets. The AI Trading Copilot's backtest engine adds configurable slippage (default: 0.1% for equities, 0.05% for major forex) to all trade entries and exits.

The Metrics That Matter

Many traders focus on total return or win rate. These are among the least useful metrics for evaluating a strategy. A strategy with 90% win rate can still lose money if the average loss is 10x the average win.

The metrics that matter for systematic trading evaluation are Sharpe Ratio (risk-adjusted return — above 1.0 is acceptable, above 1.5 is good, above 2.0 is excellent), Maximum Drawdown (the worst peak-to-trough decline — should be tolerable psychologically and financially), Calmar Ratio (annualised return divided by maximum drawdown — above 1.0 is good), and Profit Factor (gross profit divided by gross loss — above 1.5 indicates a robust edge).

The AI Trading Copilot's backtest engine reports all four metrics prominently, along with the number of trades, average trade duration, and the walk-forward efficiency ratio. No single metric tells the full story — a strategy with a high Sharpe but few trades may have insufficient statistical significance, while a high-frequency strategy with marginal Sharpe may generate reliable returns through volume.

• ·Sharpe Ratio: risk-adjusted return; target > 1.0
• ·Maximum Drawdown: worst peak-to-trough loss; must be psychologically tolerable
• ·Calmar Ratio: return / max drawdown; target > 1.0
• ·Profit Factor: gross profit / gross loss; target > 1.5
• ·Walk-Forward Efficiency: OOS return / IS return; target > 0.5
• ·Number of trades: minimum 50 for statistical significance

Frequently Asked Questions