Why Most Crypto Trading Signals Fail Out of Sample: What 2,000 Live Snapshots Taught Us

Since February 2026 our risk engine has recorded a snapshot of BTC and ETH market state every few hours: the inputs, the scores, the policy it produced, and later the price moves that followed over 24 hours, 72 hours, 7 days and 30 days. By mid-October we had 2,069 of them.

When we tested the engine's composite market score against those forward returns, with methods that account for overlapping data, we found no detectable signal. A block permutation test gave p-values above 0.56 at every horizon, and the family-wise p-value across horizons was 0.98. An earlier headline result, a +21.7 percentage-point win-rate gap between score buckets, did not survive more data. We no longer cite it.

This article explains why that happened, what did hold up, and why a negative result like this is exactly what the snapshot record exists to produce.

In May we froze the scoring logic for six months: no weight, threshold or formula changes. The reason is simple. Every time you tweak a model after looking at its results, you contaminate the test. If we adjusted the weights each time a backtest disappointed, we would end up with a model fitted to the past and no honest evidence about the future.

Freezing forced the engine to face data it had never seen. That is what out-of-sample means in practice: not a held-out slice of an old spreadsheet, but decisions recorded live, scored later against what the market actually did.

The most important finding was not that the score was noisy. It was that it was looking backwards. Our structural score correlated about +0.45 with the previous 7 days of price movement, and the trend-strength input about +0.77. The scores were largely describing the move that had already happened.

That has a predictable consequence. After a rally the score reads strong; after a drop it reads weak. In the up-legs of our sample, the layers that looked strongest were negatively related to the next 7 days. Any system that sizes up when such a score is high ends up largest just after the move, which is pro-cyclical sizing.

This is the most common way crypto indicators fail. Momentum, sentiment and many composite scores are built from recent price, so they look predictive in a backtest of a trending period and quietly invert when the regime changes.

Before testing any model, you have to test the pipeline. Ours had three defects, all found by auditing the snapshot record itself.

Some server-side inputs were constants. An exchange blocked requests from our hosting provider, and several inputs (RSI, long/short ratio, MVRV, DXY) silently fell back to default values on most server snapshots for months. Momentum, one of the highest-weighted inputs, was effectively a constant across three quarters of the sample.

Some entry prices were stale. Browser-recorded snapshots sometimes stored a price that was not the market price at that moment, up to 21.7% off for ETH. Returns computed from those prices were wrong. We now compute entry and exit from the same exchange candles and migrated every historical label.

Some labels were premature. The export could attach a "7-day return" to a snapshot that was only five days old. It now requires the horizon to have actually closed.

None of these were scoring bugs, and fixing them did not rescue the signal. But each one would have produced a misleading result in either direction. If you have not audited where every number in your backtest came from, you do not yet know what you are testing.

2,069 snapshots sounds like a lot. It is not, for a 7-day question. Snapshots taken a few hours apart share almost the same future, so they are not independent. Our clean BTC daily cohort works out to roughly 18 independent 7-day windows, all in one fairly calm regime with no panic or euphoria periods.

This matters for the statistics. A standard permutation test that shuffles observations as if they were independent gave one of our results a p-value of 0.036, which looks significant. The same test with blocks that preserve the time structure gave 0.696. Same data, opposite conclusion. If a crypto backtest reports thousands of observations and a tiny p-value without saying how it handled overlap, be sceptical.

Two things survived the audit.

Volatility pointed size in the right direction. Of the sizing rules we tested, a rule driven by realized volatility was the only one whose size moved the right way relative to subsequent drawdowns. Its tail benefit was not statistically significant in 2026's low-volatility market, but an eight-year reconstruction across multiple regimes supports it, and so does decades of evidence on volatility targeting in traditional markets.

The level of exposure did most of the protective work. Simply operating at a conservative average size reduced the frequency of deep weekly losses far more than any attempt to time when to be bigger.

The lesson is unglamorous: in our data, a modest base size adjusted by measured volatility beat the sophisticated layers built on top of it.

It would be easy to read this as a failure. We read it as the system working. The reason we record every decision with its inputs, a hash and its forward returns is so that the engine can be checked against reality, and so that we know precisely what to change when it falls short.

The negative results tell us where to act: remove the price echo from the structural layer, rebuild sizing around volatility, stop counting the same condition several times in the rules engine, and recalibrate inputs that sat pinned at one extreme. Since mid-October the candidate fixes have been computed in shadow alongside every live decision, recorded but never used, so they are measured on fresh data before they ship.

The week of 23 November, when the six-month freeze ends, we ship one calibration upgrade. The API contract does not change. The scoring_version field will, and so will the policy_hash values. The changelog will list what moved and why. Then the new version is frozen and tested forward the same way. Recording, measuring, admitting what does not work and recalibrating in the open is not a detour from building a reliable risk engine. It is the process.

Whether you are testing an indicator, a bot or an AI agent, these checks catch most of what we found:

1. Trace every input to its source and confirm it varies. A column with zero variance is a default, not data.

2. Compute entry and exit prices from the same source and the same candle convention.

3. Only label a horizon once it has fully closed.

4. Count independent windows, not rows. Use block bootstrap or block permutation for overlapping horizons.

5. Check whether your signal correlates with the trailing move. If it does, it may be describing the past rather than predicting the future.

6. Compare against a simple baseline at the same average exposure, such as a volatility-scaled position. If you cannot beat it, the complexity has not earned its place.

7. Freeze, then test forward. Decide in advance what result would make you change the model, and record it before you look.