my rules for not fooling myself in backtests (long, sorry)

Been asked a few times in chat how I set up a test before I let anything run live, so dumping my checklist here since it's the faq board. Not advice, just what stops me personally from lying to myself. 1. Write the hypothesis down BEFORE touching data. One sentence, plain english. "Mean reversion after N down days works better in high volatility regimes." If I can't say it in one sentence I don't understand it yet. 2. Split the history first. I chop off the most recent chunk and physically do not look at it until the end. If I peek, the test is dead and I have to pick a different holdout. Sounds dramatic but I've cheated on this before and the live results were... educational. 3. Count my parameters. Entry threshold, exit threshold, lookback, position size, filter = 5 knobs. On a few hundred trades, 5 knobs is already pushing it. Every knob you add, the curve gets prettier and the future gets worse. 4. Look at the parameter surface, not the best value. If lookback=14 prints great and 13 and 15 are garbage, that's noise wearing a costume. I want a broad boring plateau, and I'll take the middle of the plateau over the peak every time. 5. Costs in from the start, not bolted on later. Spread, fees, and a pessimistic fill assumption. Strategies that only work with perfect fills are just fee generators. 6. Trade count minimum. Under ~100 trades I basically don't believe anything I'm seeing, and even at 100 the error bars are embarrassing. 7. Drawdown is the number I actually plan around, not CAGR. I ask myself if I'd still be running it three months into the worst stretch in the test, because the real one is usually worse. Two things I'm genuinely unsure about and would like other opinions on: - Walk-forward vs one big holdout. I do the single holdout because it's simple, but I keep reading that rolling re-optimization is more honest. My worry is that every re-fit is another chance to overfit, just spread out. Anyone actually run both on the same idea and compare? - How do you handle a strategy that passes the holdout but you tested nine other variants first? Feels like the holdout is used up at that point even though I only looked once per variant. I've started keeping a tally of how many ideas I've thrown at a given dataset and it's a depressing number.