In-sample, out-of-sample, and the discipline nobody keeps

The rule is simple: build on one period, verify on another you haven't touched. The discipline is where it falls apart, because "haven't touched" degrades every time you peek. Once you've looked at the out-of-sample result and gone back to adjust, it isn't out-of-sample anymore. It's just a slower version of fitting. Most people, including me, do this at least once and then keep calling the number honest. What works for me: split the data before I start, and write the out-of-sample date range on a sticky note so that "just checking" requires me to physically ignore my own instruction.