A Taxonomy of Backtest Lies: Survival Bias, Lookahead Bias, and the Rest

I am reorganizing where my quantitative writing lives. Working notes on essays, ideas, and identity stay here on susanpotter.net. Longer-form technical work that maps directly onto the tooling my company builds is moving to referentiallabs.com/blog , where it sits alongside the services and platform pages it informs.

The full taxonomy of backtest biases has moved to Referential Labs: A Taxonomy of Backtest Lies .

What the article covers

Every backtest I have written was biased. The useful question is not whether biases exist but which ones are present, how large they are, and whether any signal survives after accounting for them. The article catalogs the biases I have encountered, organized by source:

• Data biases: survivorship, corporate action handling, staleness, index reconstitution, and the momentum trap that emerges when survivorship interacts with directional strategies.
• Methodology biases: lookahead in its three forms (direct, data revision, knowledge), the pandas index alignment trap that silently leaks one bar of future information, time-period bias when your window captures a favorable regime, and transaction cost bias that flatters strategies with unrealistic execution.
• Reporting biases: overfitting under multiple hypothesis testing, and the maths from Bailey, Borwein, Lopez de Prado, and Zhu (2014) showing that even zero-signal parameter search produces expected Sharpe ratios of 3.7 across 1,000 combinations.

Each section covers the mechanism, direction, magnitude from the academic literature, detection methods, mitigation strategies, and code snippets for automated checks. There is a scorecard at the end for producing a de-biased Sharpe estimate from a reported one, and integration notes for slotting the audit into a pre-deployment validation pipeline.

Why the reorganization

The commercial site is where the tools and services these articles inform actually live. Colocating the writing with the tooling makes the site do a job it should always have been doing, and it lets me use susanpotter.net for the kind of writing that does not belong on a company site: essays, ideas, incidental notes.

Continue to the full article at Referential Labs .