82% of our backtest fills were trades we could not have taken live

We had a model we liked. It passed walk-forward, it passed out-of-sample, and the equity curve looked like something you would want to trade. Then we audited whether the trades could have existed at all, and most of them could not. This is a report on that audit, because the failure mode is not specific to us. Any backtest that reads a watchlist from a database has to answer one question: did that list exist, in that form, at the moment the strategy claims to have acted on it.

1. What "eligible" means

Our engine does not see the whole market. It subscribes to a rotating watchlist, and a symbol becomes tradable only after the engine has subscribed to it. That gives four constraints a backtest must respect: - The candidate list at time t must be built only from trades that happened up to t. A list of "symbols that were top movers at some point that day" is future information, even if every price in it is historical. - A ranking built from the window [t, t+30) is only known at t+30. Labelling it t hands the strategy up to 29 seconds of the future. - No entry before first_seen — the moment the engine actually subscribed to that symbol. - Local accumulators (trade strength, day buy amount, running maxima) restart at first_seen, because live they start at zero when the subscription opens. None of these are exotic. They are just the difference between "the data exists in my database" and "my program could have known it".

2. What the audit found

We rebuilt the dataset under those constraints and re-ran the same model family. Original fills that broke at least one constraint: 82.2% Share of total profit carried by those fills: 48% The second number is the one that matters. Even if you assume the violating trades were only mildly optimistic, removing them removes half the result. When we applied the constraints properly and retrained, the edge did not survive. It held up in only 2 of 11 walk-forward windows, and the out-of-sample result turned negative.

3. We checked whether anything was salvageable

A reasonable objection: maybe the model still ranks eligible candidates well, and we only lost the impossible trades. We tested that directly over the full set of eligible candidate rows. Rows examined: 564,303 Correlation between model score and realised return: -0.0075 Deciles: no monotonic relationship Top 1% by score: still negative So the ranking carried no information on the trades we could actually take. The model had not learned an edge that happened to include some impossible fills; the impossible fills were the edge.

4. Where the violating trades actually came from

We also checked the shape of the 1,714 violating trades, because the natural assumption is that they were obscure names the engine picked up hours later. That is not what happened. At the moment of purchase, their median rank across the whole market was 206 (p25 107, p75 429). Symbols ranked inside the top 500 accounted for 75.7% of the violating profit, and the "discovered hours later, thousands of ranks away" pattern accounted for 0.5%. In other words, the trades were not exotic. They were ordinary active names that our engine simply had not subscribed to yet. That is a coverage problem, not a data-cleanliness problem, and it points at a different fix: watch more of the market, earlier.

5. What we changed

The rules above are now enforced when the dataset is generated, not checked afterwards. Recording starts at first_seen. Snapshots are stamped at the end of their bucket. Local accumulators reset on subscription. An audit pass runs over the generated rows and fails the build if a single row violates a rule. We would rather have a smaller result we can trust than a larger one we cannot reproduce live. Conclusion The useful question about a backtest is not how good the result is. It is whether every input was available at the moment the strategy claims to have used it. We asked that question late, and it cost us the result. Asking it first is cheaper.