Why your backtest does not match live trading, and how much each gap costs

A backtest says one thing and the account says another. When results do not match, the usual explanations offered are overfitting and psychology. Those are real, but they are also unfalsifiable, which is why they get reached for first. There are four gaps that are not vague at all. Each one is a specific assumption a backtest makes that live trading does not honour, each one can be measured, and we have measured all four on our own data. Two of them are large enough on their own to turn a good backtest into a losing account.

1. The four gaps, in order of how much they cost

The rest of this guide is one section per gap: what it is, what it costs, and how to test whether it applies to you.

2. Gap one — data you could not have had

This is the biggest and the least visible, because nothing about it looks like cheating. Your engine does not see the whole market at once. It watches a list. A symbol only becomes tradable after your system has actually started watching it. That produces four constraints a backtest has to respect: The candidate list at time t must be built only from what happened up to t. A list of "symbols that were big movers that day" is future information, even though every price in it is historical. A ranking computed over a window is only known at the end of that window. Stamping it at the start hands the strategy the length of the window in free foresight. No entry before the moment you would actually have subscribed to that symbol. Running totals restart when the subscription opens, because live they start at zero, not at the day's true cumulative value. None of these are exotic. They are the difference between "the data is in my database" and "my program could have known it." When we audited one of our own models against those constraints — an audit over 564,303 eligible candidate rows — 82.2% of the original fills broke at least one of them, and those fills carried 48% of the total profit. Applying the constraints properly and retraining, the edge did not survive. How to test yours: take any single trade the backtest took and ask what your program knew one second before entry. Not what the database contains — what the running program had received. If the answer requires a list built from the whole day, the backtest is reading the future.

3. Gap two — the fill price

Every backtest assumes a fill price. Most assume the last trade, some assume the mid, the careful ones subtract a fixed number. All three are guesses, and the size of the guess decides the result. We took one model's 2,084 trades and re-priced every fill under six cost assumptions, changing nothing else: One tick removed 44% of the edge. Two ticks removed 87%. Look at the win rate column while you read the edge column. The edge falls to nothing while the win rate moves about five points. A strategy can keep winning the same share of trades while every dollar of profit disappears, which is why traders watch a healthy-looking win rate and still lose. How to test yours: re-run with a fill penalty of one tick round trip, then two. If the result changes character, you did not have an edge, you had an assumption.

4. Gap three — the fill size

The price is only half the fill. The other half is how much of your order gets it. We walked market orders against ten recorded levels of book and measured what the quoted price actually supports: about $1,155 in $1-5 stocks, $1,909 in $5-20, and $5,150 in $20-200. Beyond that you are buying the second and third best offers. At larger size it stops being a rounding error. A $25,000 order could not be filled inside ten visible levels 28.7% of the time. A backtest that fills $25,000 instantly at the quote is not modelling a trade that existed. How to test yours: compare your typical order size against those numbers. If it is bigger, your backtest is quoting you a price for a quantity that was never available.

5. Gap four — the universe moves

This one surprises people who build a watchlist at the open and work from it. We recorded which symbols a 30-slot, change-ranked universe actually held at every instant. A median of 140 symbols passed through those 30 slots in one session — a turnover of 4.7 times. Of the symbols present at 09:35, only 57% were still there at 10:00, and 33% at the close. If your backtest ranks the whole market continuously but your live system watches thirty names, you are not testing the same strategy. You are testing a version with a bigger field of view than you will ever have. How to test yours: cap the backtest's candidate set to the number of symbols you can actually subscribe to, chosen by a rule that only uses past data, and re-run.

6. Do these in order

The four gaps are not equally worth your time. Fix gap one first. It is the only one that can invalidate everything rather than shrink it. A strategy with future information in it is not a strategy with an optimistic cost model, it is a different strategy that does not exist. Then gap two. It is one line of code to add a tick of penalty, and it is the fastest way to find out whether you have anything. Then gaps three and four, which mostly determine how much of a surviving edge you actually capture. If you want the mechanics of running tests on your own recorded data rather than vendor bars, that is covered in how to backtest on your own recording, and the cost side is in what is slippage and what does it cost.

7. What this does not mean

It does not mean backtesting is useless. It means an untested backtest is a hypothesis about a hypothesis. It also does not mean these four numbers are yours. 82.2% was our model on our data with our subscription limits, not a universal constant, and quoting it as one would be exactly the sloppiness this guide is arguing against. The numbers are here to show the size these gaps can reach, and to give you four concrete tests. The results that matter are the ones you get running them on your own.