my first profitable backtest was a bug

Mine was three weeks and I told people about it first. That part stung more than the code.

A good habit: before celebrating, run the same code on shuffled data. It should lose. If it still makes money, you have a bug.

Betty's shuffle test is neat, but how do you decide what "should lose" looks like exactly? Shuffled data still has drift baked in if you shuffle returns and rebuild the series. Do you shuffle the returns or the bar order?

One thing that tipped me off on my own bug years back was the sizing side of it. If your win rate and payoff imply a Kelly fraction near 1, the strategy is claiming it's almost never wrong, and nothing real is almost never wrong. Compute f = W - (1-W)/R on your backtest output. If f comes out huge, that's the same alarm Mike is describing, just in a number instead of a curve.

dee - shuffle bar order. keeps the distribution, kills the sequence. if the edge survives that it was never about sequence. also check trade count. mine had 40 trades over 6 years. should have stopped there.

Three weeks is rough, Dave. Least you found it before it found you with real size. I hold for days at a time and honestly the slow pace means bugs show up in the boring way, position sits there doing nothing and you go check why.

THE TOO CLEAN CURVE THING IS SO REAL!! mine had like two down days in the whole sample and I remember being thrilled instead of suspicious. now if the drawdown chart looks flat I assume I broke the drawdown chart lol

- log every fill with the timestamp of the bar it used - diff that against when the data was actually available - most look-ahead shows up right there - takes 10 min, saves the week

The trade count point deserves more weight than it got. With 40 trades the standard error on your mean trade is wide enough that a nice curve is barely evidence of anything, bug or no bug, the sequence test and the sanity check both get drowned in noise. What I'd rather see is the shuffle run a few hundred times so you get a distribution of outcomes under the null, then ask where the real result falls in it; a single shuffle that loses tells you very little. Does anyone here keep those null distributions around, or do you regenerate them each time?