A backtest that takes over a minute to load a single day is not a backtest you will run twice. You end up testing fewer ideas, which is the opposite of the point.
We profiled where the time went and found it was not the strategy engine.
1. Where the time actually went
Loading one recorded session took 73.8s. The engine itself ran in a couple of seconds. Almost all of the wall clock was spent pulling rows out of SQLite and turning each stored value into a number, one value at a time, across hundreds of millions of values.
That is a storage format problem, not an algorithm problem. Row-oriented storage with per-value parsing is a poor fit for something that reads whole columns.
2. What we changed
We write each recorded session a second time in a columnar binary layout: fixed-width float64 values, one block per column, with a checksum and a header describing the schema. Loading becomes a read into memory rather than a parse.
| Path | Time |
|---|
| SQLite load | 73.8s |
| Binary load | 0.9s |
That is 81x faster on the same data.
3. Proving it did not change anything
Speed is only useful if the numbers are identical. Two checks had to pass before we would use it:
- Every cell compared bit for bit: 641,261,891 values, all identical.
- The same strategy run over both formats produced the same 11,494 trades — same times, prices, quantities and P&L.
Not "close enough". Bit-identical, because a float that differs in the last place can move a value across a decision boundary and change a trade.
4. What it costs
The binary file is larger than the compressed SQLite original, because raw float64 does not pack as tightly as SQLite's integer encoding. We accepted that: disk is cheaper than the time you lose waiting. For transfer we compress it, which recovers most of the difference.
Conclusion
Before optimising a backtest engine, measure where the time goes. Ours was almost entirely in reading data, and the fix was a storage format rather than faster code. Whatever you change, verify bit-identity before you trust the faster path.
