Backtesting

Backtesting is the process of simulating a trading strategy on historical data to evaluate how it would have performed in the past. It is the cornerstone of quantitative strategy development — the quant equivalent of a clinical trial in medicine. Before risking real capital, every serious quant firm tests its strategies extensively against historical market data.

The basic process is straightforward: define your trading rules (when to buy, when to sell, how much to trade), feed in historical price and volume data, simulate the trades, and measure the results. The output is a simulated track record showing returns, risk metrics, and other performance statistics.

However, the apparent simplicity of backtesting masks significant methodological challenges. A poorly conducted backtest can make a worthless strategy look like a money machine — only for the strategy to fail spectacularly when deployed with real money. Understanding what can go wrong in backtesting is just as important as understanding how to do it.

A rigorous backtest follows these steps:

1. Define the strategy rules clearly: Every decision the strategy makes must be fully specified — entry signals, exit signals, position sizing, and risk limits. There should be no ambiguity or room for subjective interpretation.
2. Gather high-quality historical data: This includes prices, volumes, corporate actions (splits, dividends), and ideally order-book data. Data quality matters enormously — errors in historical data can create phantom profits.
3. Split data into training and testing sets: Use one portion of data (e.g., 2010-2020) to develop the strategy and a separate, untouched portion (e.g., 2020-2025) to test it. This out-of-sample test is critical for detecting overfitting.
4. Simulate with realistic assumptions: Include transaction costs (bid-ask spreads, commissions, exchange fees), market impact (large orders move prices), and borrowing costs for short positions.
5. Evaluate performance metrics: Calculate the Sharpe ratio, maximum drawdown, win rate, profit factor, average trade duration, and turnover. No single metric tells the full story.
6. Stress test: Run the strategy through historical crises (2008 financial crisis, March 2020 COVID crash, 2022 rate hiking cycle) to understand tail risk behavior.

Most backtesting errors lead to results that are too optimistic — making you think a strategy is better than it actually is. Here are the most dangerous pitfalls:

• Overfitting (curve fitting): The most common and most dangerous error. If you test hundreds of parameter combinations and pick the best-performing one, you are fitting to historical noise, not a genuine signal. The strategy will almost certainly fail out of sample. Rule of thumb: the more parameters a model has, the more likely it is to be overfit.
• Look-ahead bias: Using information that would not have been available at the time of the trade. Example: using a company's full-year earnings to make a trade in January. This is sometimes subtle — even using the closing price to make a trading decision at the close constitutes a form of look-ahead bias.
• Survivorship bias: Only testing on securities that exist today, ignoring those that were delisted or went bankrupt. This makes mean-reversion strategies look better than they are because the worst performers are excluded.
• Ignoring transaction costs: A strategy that trades frequently and earns small profits per trade can look amazing before costs but terrible after realistic spread and slippage assumptions.
• Ignoring market impact: Large orders move prices. A strategy that assumes you can buy $10 million of a small-cap stock at the current price is unrealistic — your own buying would push the price up significantly.
• Data snooping: Running many statistical tests on the same dataset and reporting only the significant results. If you test 100 random signals, 5 will appear statistically significant at the 5% level by pure chance.

Let's backtest a simple moving average crossover strategy on the S&P 500:

Rules: Buy the S&P 500 ETF (SPY) when the 50-day moving average crosses above the 200-day moving average ("golden cross"). Sell and go to cash when the 50-day crosses below the 200-day ("death cross").

Period: January 2005 to December 2024 (20 years).

Results (hypothetical):

• Annual return: 8.2% (vs. 10.1% for buy-and-hold)
• Maximum drawdown: -18% (vs. -55% for buy-and-hold)
• Sharpe ratio: 0.65 (vs. 0.55 for buy-and-hold)
• Number of trades: 14 round trips over 20 years
• Win rate: 43% (most signals were false, but the winning trades were large)

Analysis: The strategy underperformed buy-and-hold on raw returns but had better risk-adjusted returns (higher Sharpe) and significantly less drawdown. However, we should be cautious: this is a single backtest with no out-of-sample validation, and the parameters (50/200) were chosen because they're popular — a form of implicit data snooping.

A more robust approach would test the strategy across multiple parameter combinations, multiple assets, and multiple time periods to verify that the result is not specific to this particular configuration.