the tick loading writeup — worth it for a small symbol list?
morning all, coffee number two in hand ☕
just read the research post about cutting the load time for a day of recorded ticks by swapping the SQLite reader for a columnar binary thing. the bit that stuck with me wasn't the speed number itself, it was the point that if a single day takes over a minute to load you just... stop rerunning stuff. which is painfully true. i've absolutely made a habit of "eh, close enough" on a parameter because i didn't want to sit through another load.
but here's what i'm unsure about for my own setup. their test was a couple hundred symbols and a fairly chunky recording. i record maybe a dozen names, mostly the ones i actually watch in the morning. does the win scale down, or is most of the gain coming from the sheer row count and i'd barely notice? i'd guess the per-row overhead is per-row so it should still help proportionally, but i genuinely don't know if there's a fixed cost that dominates at small sizes.
second thing they didn't dwell on much: disk. columnar formats usually compress nicely but i'd like to know if i'm trading a slow read for a fat folder, since i keep a rolling archive going back a while.
and third — they verified every cell came out bit-identical and the trade list matched, which is exactly the check i'd want to see, and i respect that they bothered. but does that verification cost anything at runtime, or is it a one-off thing you do when you migrate and then trust it?
anyone here actually converted their own recordings over? curious what the migration morning looked like in practice 😄