Is overfitting why my ideas die out of sample?

MLcurious· Jun 2026 ago· 👍 2· 💬 5

Every idea I have looks great on the data I built it with and mediocre-to-bad afterwards. I know the word "overfitting" but I don't think I understand it in a way I can act on. Concretely: if I try eight variations and pick the best one, have I already overfit even if I never touched the out-of-sample period? It feels like the answer is yes but I don't know how to do research without trying variations.

MLcurious

18 posts · 0 followers

+ Follow

5 replies

QuietVol· Jun 2026 ago

Yes, and this is the honest version of the problem. Trying variations is fine; the cost is that your best result is now biased upward and you need a held-out period you genuinely haven't looked at to measure the real number.

BacktestBetty· Jun 2026 ago

Practical version: write down how many variations you tried. If it's eight, expect the live result to look like the fourth-best, not the best.

MLcurious· Jun 2026 ago

"Expect the fourth-best" is the most useful thing anyone has told me about this.

GrandpaGrizzly· Jun 2026 ago· edited Aug 2026 ago

Old trick from the pit days: after you pick your best variation, look at the ones right next to it. If 20 days works great and 18 and 22 are garbage, you didn't find an edge, you found a pothole. A real edge is a plateau, not a spike.

RiskFirstRita· Jun 2026 ago

Add one line to your research checklist: whatever number you'd size on, size on the fourth-best instead. The overfit usually doesn't kill people, the position size they took because the backtest looked so pretty does. And write the held-out period rules down before you run anything, or you'll negotiate with yourself later.

Is overfitting why my ideas die out of sample?

Related reading

5 replies