so how long do i have to record before it's actually usable?

ok that guide on recording your own crypto data has been rattling around my head all week!! The part that got me was the bit about volume being reported wildly differently between venues for the SAME symbol. Not a little different. Enough that any threshold I tuned on one source would be nonsense on the other. I have absolutely been guilty of pulling free history from wherever and pretending it's the market I actually trade on. Guilty! Moving on! So I set up a recorder on a cheap little box last weekend and it's been ticking away. But here's where I'm stuck, and the guide sort of stops before answering it for me: 1. How much do I actually need before I stop treating it as a toy? A week of my own feed is my own feed, sure, but it's also one week of one regime. Momentum stuff is all about how it behaves when things get loud, and quiet weeks tell me nothing about loud days. 2. Gaps. My box dropped connection twice already (my fault, not the venue). Do I patch those, mark them and skip them, or throw the whole session? Feels like silently patching is how you end up backtesting a fantasy. 3. The depth argument — I get why order book history is the interesting bit and why nobody sells it. But I mostly hold for minutes, not seconds. Is depth recording worth the storage for someone on my timeframe or am I hoarding data I'll never open? Anyone actually done this for a few months and looked back at their early files? What did you wish you'd recorded from day one?!
M
MomoQueen
14 posts · 0 followers
+ Follow

Related reading

4 replies

N
NightOwl_Yuki· Aug 2026 ago
been running mine since spring, and honestly the thing i wish i'd captured from the start is timestamps from both sides — when the venue says it happened and when my box actually received it. found out later my "fast" reactions were sometimes just clock drift lol
DataDrivenDee· Aug 2026 ago
On the depth question — how would you actually test whether it's worth it for your timeframe? Seems like you'd need to record it for a while anyway, then check if any feature built from it survives when you're holding minutes rather than seconds. Which is a bit circular, but at least the cost of finding out is just disk.
BacktestBetty· Aug 2026 ago
A few things from having done exactly this and made the mistakes: 1. Mark gaps explicitly in the data, never patch. A flag column costs you nothing and saves you from a result you can't explain six months later. 2. Your "how long" question doesn't have a time answer, it has an events answer. You don't need N weeks, you need enough distinct loud days that you're not fitting to two of them. That could be a month or a season depending on what shows up. 3. Related warning: early on you'll have a small sample and be tempted to tune thresholds on it. Don't. Let it accumulate boring and untouched for a while, because the temptation to fit the first thirty sessions is enormous and I fell straight into it. 4. Record the raw stream, not your derived features. You will change your mind about the features. You cannot re-derive from something you threw away.
P
PremarketPete· Aug 2026 ago
- checksum or row count per session file, check it daily, takes seconds - log your recorder's own restarts separately from market data - keep the raw, build bars later - agree with betty, don't touch it for tuning yet
Sign in to reply →
← All brokers & apis