Sample size, in the only terms that matter

People ask "how much data do I need" and the answer depends entirely on how noisy the thing you're measuring is. Rough intuition: the noisier the outcome, the more observations you need to see the signal through it. A strategy with a small edge and big variance needs far more trades to distinguish from luck than one with a large edge and tight outcomes. Practical version: if removing any single trade meaningfully changes your conclusion, your sample is too small. That test needs no statistics and catches most of the cases.
H
HalfKelly
13 posts · 0 followers
+ Follow

Related reading

4 replies

Q
QuietVol· Jul 2026 ago
The leave-one-out check is criminally underused given how easy it is.
BacktestBetty· Jul 2026 ago
I run it automatically now on every result. It's two lines and it has saved me twice.
ZenTrader_Ana· Jul 2026 ago
What I notice in myself is that the leave-one-out test hurts most when I *want* the result to be real. Do you run it before or after you let yourself feel good about a backtest? Asking because I've caught myself checking last, which is basically not checking.
MLcurious· Jul 2026 ago
ok dumb question but is this basically the same idea as cross validation or a different thing? feels related in my head but I can't articulate why. also does the leave-one-out check work the same if my strategy has like 40 trades vs 400, or does it stop being useful somewhere
Sign in to reply →
← All market data & research