the tick loading writeup — worth it for a small symbol list?

morning all, coffee number two in hand ☕ just read the research post about cutting the load time for a day of recorded ticks by swapping the SQLite reader for a columnar binary thing. the bit that stuck with me wasn't the speed number itself, it was the point that if a single day takes over a minute to load you just... stop rerunning stuff. which is painfully true. i've absolutely made a habit of "eh, close enough" on a parameter because i didn't want to sit through another load. but here's what i'm unsure about for my own setup. their test was a couple hundred symbols and a fairly chunky recording. i record maybe a dozen names, mostly the ones i actually watch in the morning. does the win scale down, or is most of the gain coming from the sheer row count and i'd barely notice? i'd guess the per-row overhead is per-row so it should still help proportionally, but i genuinely don't know if there's a fixed cost that dominates at small sizes. second thing they didn't dwell on much: disk. columnar formats usually compress nicely but i'd like to know if i'm trading a slow read for a fat folder, since i keep a rolling archive going back a while. and third — they verified every cell came out bit-identical and the trade list matched, which is exactly the check i'd want to see, and i respect that they bothered. but does that verification cost anything at runtime, or is it a one-off thing you do when you migrate and then trust it? anyone here actually converted their own recordings over? curious what the migration morning looked like in practice 😄
C
CoffeeAndCharts
12 posts · 0 followers
+ Follow

Related reading

3 replies

CryptoKarl· Aug 2026 ago
the "you stop rerunning things" line is the whole post honestly, i've definitely shipped a size on a strategy i only backtested once because reloading was a chore. bad reason to risk money.
H
HalfKelly· Aug 2026 ago
On your small-list question — if the cost is dominated by per-row deserialisation then it's roughly linear and you'd keep the same ratio, just with a smaller absolute saving. The thing to watch is whether the saving is still big enough to change your *behaviour*: going from a long wait to a short one changes how many ideas you test, going from short to shorter often doesn't. Worth timing your own dozen-symbol day before and after rather than assuming, since your bottleneck might genuinely be somewhere else entirely.
SlowSwing_Sam· Aug 2026 ago
meanwhile i load one daily bar file and go make lunch 😌 but yeah, the archive-size question is the one i'd want answered before converting anything.
Sign in to reply →
← All market data & research