Keep the raw data, always

BacktestBetty· Jun 2026 ago· 👍 5· 💬 13

Whatever processing you do, keep the original. Not because you'll need it. Because you will eventually discover a bug in the processing, and without the raw version every conclusion you drew is unverifiable. I learned this after finding an error in how I was aggregating. Everything downstream of it was suspect, and the only reason I could rebuild rather than start over was that the untouched source was still there. Storage is cheap. Redoing six months of research is not.

BacktestBetty

16 posts · 0 followers

+ Follow

13 replies

QuietVol· Jun 2026 ago

Also keep the processing code version alongside the output. Reproducibility needs both.

DrawdownDave· Jun 2026 ago

I did not keep the raw data. That is all I'd like to say about that.

GrandpaGrizzly· Jun 2026 ago· edited Aug 2026 ago

Learned this one on paper tapes and floppies, back when storage wasn't cheap and we still knew better. Kept a shoebox of printouts for exactly this reason.

SlowSwing_Sam· Jul 2026 ago

Dave's reply is the whole thread in two lines honestly. I just dump everything into a dated folder and never touch it again, takes zero effort.

LurkerLee· Jul 2026 ago

Do you keep the raw at the vendor's timestamp or do you normalise timezone before you file it? I've gone back and forth on whether that counts as processing.

BacktestBetty· Jul 2026 ago

Timezone question is a good one and my answer is: (1) raw stays exactly as received, byte for byte, including whatever weird timestamp format they use. (2) normalisation is the first step of the pipeline, not part of storage. (3) if you normalise on ingest you've already made an assumption you can't audit later, and timezone/DST bugs are the single most common thing I've had to go back and fix.

FiveMinFiona· Jul 2026 ago

Agreeing with the raw-stays-raw answer. My folder structure is source / date-received / untouched file, then a separate derived tree that I'm free to delete entirely and rebuild. Knowing the derived side is disposable makes me much less precious about rewriting the pipeline.

DataDrivenDee· Jul 2026 ago· edited Aug 2026 ago

QuietVol's point about code versions is the part people skip. Though I'd ask. Has anyone actually tried rebuilding an old result from raw plus tagged code and had it come out identical? Mine drifted once because a library version had changed underneath me, and I only found out by accident.

HalfKelly· Jul 2026 ago· edited Aug 2026 ago

Dee. That drift is almost always the environment, not your code. Pin the library versions in a lockfile and store it next to the tagged code, otherwise "same code" just means same text, not same behaviour. I had a rounding change in a stats package shift a vol estimate by a hair, which is nothing until it's feeding a sizing formula and the fraction moves.

MomoQueen· Jul 2026 ago

Dated folder, never touch, exactly like Sam said! Zero thinking required and thinking is what gets me in trouble at 3pm!!

CryptoKarl· Jul 2026 ago

Extra reason for crypto folks: exchanges quietly restate history. Candles get backfilled, a pair gets delisted and the whole endpoint vanishes, or they change how they stamp a bar. If your only copy is "whatever the API gives me today" you are trusting someone else's memory of the past, which historically has gone about as well as my 20x longs. I keep the raw json dumps ugly and unparsed for exactly this.

SpreadWatcher· Jul 2026 ago· edited Aug 2026 ago

Same applies to your own fill records. Keep the raw execution reports, not your cleaned-up P&L summary, fee schedules change and a rebuilt cost model applied to old trades will lie to you.

NightOwl_Yuki· Jul 2026 ago

Yeah raw stays raw for me too, my downloader just writes whatever it got and doesn't even look at it. Sorting it out is a daytime problem lol.

← All market data & research

Keep the raw data, always

Related reading

13 replies