Skip to model research
Systematic Trading ModelsSpecification / Evaluation / Release

Backtest audit checklist

A detailed experiment contract covering data availability, preprocessing, execution, costs, population and reserved evaluation; includes a blank JSON template.

Original editorial revision: 9 October 2026. Expanded edition: 14 October 2026.

Research design and arithmetic, not model recommendations. Classification scores are not monetary returns; a registered artifact and a passing software test are not independent strategy validation.

Start with the experiment specification. A performance chart is the output; the specification explains which information the model could see, when it could act, how orders were filled and which decisions were counted. If those rules change after the result is inspected, the result belongs to a different experiment. The procedure below is an audit of that chain, with hypothetical examples rather than a claim about any live model.

1. Freeze an experiment contract

Write one versioned record before testing. It should name the dataset and its timestamp convention, the instrument universe, the decision rule, the exit policy, the sizing rule, costs, the split dates and the success criteria. Keep enough detail that two people running the same version would generate the same decisions. A phrase such as “buy an oversold market” leaves too much room; a defined input, threshold, decision time and order instruction can be reproduced.

Download the blank experiment contract (JSON). Its null fields mark decisions still to be specified; it contains no sample performance or implied approval. Preserve the completed version with the data manifest and trial ledger.

What the frozen research record must identify
PartRequired detailFailure it catches
DataSource version, timezone, bar convention, missing rows and instrument identifiersA result changing because the input file changed
DecisionAvailable features, lookback, threshold and earliest action timeA signal using prices that were not known yet
ExecutionOrder type, fill reference, latency and ambiguous-bar policyA touched price being treated as a guaranteed fill
Exit and sizeOne declared exit policy and a risk or unit rulePicking a different exit after seeing each outcome
EvaluationDevelopment, validation and holdout boundaries; required metricsReporting tuning results as independent confirmation
TrialsEvery candidate version, including rejected and failed runsPresenting the best of many attempts as the only experiment

2. Rebuild the information clock

For each feature, ask when its value became available. A bar labelled 09:30 might describe a bar that opens at 09:30 or one that closes then. Those conventions produce different possible decision times. If a one-minute bar closes at 09:31, a rule using its close cannot also claim a fill at an earlier price inside that same bar without a separately justified execution convention.

Check three clocks in order: observation completed, signal calculated, order eligible to execute. A daily high calculated at the end of the session cannot be an input to a morning decision. A revised fundamental value must be tied to the version available at the time. These checks apply even when the code has no obvious forward-looking index.

Preprocessing can leak information too. Fit scaling, imputation and feature selection on the training segment, then apply the fitted transformation to later segments. The scikit-learn leakage guidance explains why learning transformations from the full dataset gives overly optimistic evaluation.

3. Keep the split chronological

Use the holdout planner to sketch one boundary. Its default 1,200 observations and 100 warmup rows leave 1,100 scored rows: 660 for development, 220 for validation and 220 for holdout. Development occupies rows 101–760, validation 761–980 and holdout 981–1,200. The warmup supplies earlier context and is excluded from performance.

Suppose each label uses the next five observations. A development decision at row 760 would use validation prices through row 765. In this simple example, the last eligible development decision is row 755, whose label ends at row 760. Apply the equivalent boundary check before the holdout, and exclude end-of-series decisions without a full outcome window. The appropriate exclusion depends on the actual label and holding horizon; five rows is an illustration, not a universal gap.

A full walk-forward study repeats train-then-test windows on later data. Declare whether its training window expands or rolls, how often refitting occurs and which parameters may change. The TimeSeriesSplit documentation describes an expanding-window approach and an optional gap. A chronological split still needs the feature and label checks above.

4. Inspect fills and costs before interpreting the score

A price appearing inside a bar establishes a traded range, not that your particular order filled. Record the fill reference and handling of spread, latency, partial execution and order size. If both target and stop lie inside one bar, the bar alone may not identify which happened first. Use a declared rule for that ambiguity or sufficiently detailed data; choosing the favourable sequence after inspection changes the experiment.

Work costs in the same units as the gross result. A hypothetical average gain of 0.10 per unit becomes an average loss of 0.02 when round-trip friction is 0.12. Show the gross and net figures alongside each other and stress less favourable assumptions. The SEC fees bulletin explains the general effect of expenses on returns; the cost level for a specific experiment needs its own evidence.

5. Preserve the decision, including failure

Retain counts of eligible decisions, filled trades, cancelled instructions and rejected data rows, with reasons. Report the declared period, losses, drawdown, exposure and execution convention. If several strategies overlap, a sum of their percentages does not by itself describe a funded portfolio. Reconstruct the cash or equity series using the actual sizing and overlap rules.

Freeze the final candidate before opening the holdout. Record whether it passed the predeclared criteria and preserve a failed result. Changing a threshold after seeing the holdout makes that period part of development; a later untouched sample is needed for a new evaluation. A reproducible failure is a useful research output because it rules out one specified approach.

Keep the data manifest, experiment version, split boundaries, complete decision ledger and calculation code together. Then use the track-record guide to review the resulting evidence rather than judging only the best chart.

Related

Audit the probability stage separately

Add raw forecast ownership before any thresholded trade summary. Identify every eligible probability, positive-label meaning and when the outcome became resolvable. If the source supplies group means only, do not compute an exact individual Brier loss as though all probabilities were constant. The new lens explicitly requires that constant-per-group declaration. A threshold study also needs its selection record, and financial evaluation still needs native exits, costs and weights. Preserve earlier checklist requirements; probability diagnostics supplement rather than replace causal data, independent evaluation or subscriber execution evidence. A good score on an authored fixture belongs to software checks, never the claimed market validation population.