How to Backtest a Risk Model (Not a Trading Strategy): A Practical Framework for Crypto

Strategy backtests ask one question: would this have made money? A risk model does not trade. It outputs limits: maximum position size, whether leverage is allowed, which actions are blocked. So 'did it make money' is the wrong test.

The right questions are two, and they are different.

Signal validation: do the model's scores carry information about what happens next? For example, is a high risk score followed by higher volatility or worse drawdowns?

Policy validation: do the model's limits improve outcomes compared with a naive baseline? For example, does sizing by the model reduce drawdowns more than it costs in returns, compared with a fixed size?

A model can pass one and fail the other. Report them separately.

Pick the target before you look at results. Good targets for a risk model are risk outcomes, not returns.

Forward realized volatility over the next day or week.

Maximum adverse excursion (MAE): the worst drawdown a position would have suffered during the holding window.

Tail events: did a drop of more than X% happen within the window?

Forward returns are worth tracking, but a risk model that tightens limits before volatile periods is doing its job even if it says nothing about direction. Writing the hypothesis down first, including the exact contrast you will test, is the single best defence against fooling yourself.

If you record the model every 4 hours and measure 7-day forward outcomes, consecutive observations share almost all of their future. A thousand snapshots might contain only a few dozen independent 7-day windows.

Treat raw N as a lie. Estimate an effective sample size from the autocorrelation of the outcome, or count non-overlapping windows. Report it next to every result. A few dozen independent windows is enough to reject large claims, and not enough to confirm small ones.

Standard bootstrap and permutation tests shuffle observations independently. With autocorrelated financial data, that destroys the dependence and makes noise look significant.

Use a block bootstrap: resample contiguous blocks (circular blocks are a good default) with block length at least the outcome horizon, often twice it.

For permutation tests, shuffle in blocks, or shift the outcome series in time against the signal, rather than shuffling single rows.

Correct for multiple testing. If you test four horizons and two assets, the best of eight p-values is not a p-value. Report a family-wise result.

The difference is not cosmetic: on autocorrelated data, a result that looks significant under a naive shuffle can disappear entirely once the test respects the dependence.

An out-of-sample period only counts if the model was not changed while it ran. Every tweak after looking at results turns test data into training data.

Freeze the model version, with weights, thresholds and formulas. Record the version with every observation. Collect the out-of-sample window before analysing it. Then compare in-sample and out-of-sample results on the same pre-registered test.

This is expensive in time, and it is why we freeze our scoring for six months between versions. The alternative is a model that looks great on every chart and has never been tested.

Most failed backtests fail in the data pipeline, not in the statistics. Checks worth running before any analysis:

Constant inputs. A feature with zero variance over part of the sample usually means a data source silently failed and a default was written. Check the standard deviation of every input by period and by source.

Entry price provenance. Forward returns must be computed from the same price source at entry and exit. An entry price from a stale or different feed introduces fake returns.

Unmatured labels. A '7-day' outcome must have a full 7 days behind it. Interpolating from neighbouring observations is not the same thing.

Column alignment. When exporting assets with different feature sets, make sure each column means the same thing on every row.

Version mixing. Scores computed under different model versions are not comparable. Rescore history under one version or split the sample.

A useful backtest report states: the pre-registered hypothesis, the raw and effective sample sizes, the block length and number of resamples, the in-sample and out-of-sample results side by side, results per asset, the multiple-testing correction, and the failures.

Negative results are results. A risk model that is tested honestly, and corrected when a test fails, is worth more than one with a perfect chart that was never exposed to data it had not seen.