How to record your own crypto market data for backtesting, and what to check first

Sample data is enough to learn a tool. It is not enough to research a strategy: one day is one sample, and the venue and symbol list are someone else's. This guide covers setting up your own recording so that every day from now on becomes data you own, and, more importantly — how to verify it is worth trusting.

1. Why record rather than buy history

Nobody sells what order-book strategies need. Historical crypto data is usually trades and bars. Depth history is either unavailable or expensive, and it is the input that says something about the next few seconds rather than the last few. A recording matches the feed you trade on. If you research on one source and trade on another, your thresholds are calibrated to a market you do not participate in. We measured a seven-fold difference in reported daily volume for the same symbol between two venues on the same day. That is not a rounding difference, that is a different market. You cannot go back and record yesterday. This is the asymmetry that decides the whole question. Storage is cheap and reversible; a day you did not record is gone.

2. What gets recorded

Per interval, per symbol: price, the ten-level order book on both sides, buy and sell volume, turnover, and cumulative daily figures — 68 columns in total. The one-minute file is built from the same stream as the one-second file. Sizing, from our own recordings: A year of one-second recording across five venues runs into the terabytes. Plan for it before you start rather than discovering it in month three.

3. Setting it up

1. Connect a venue Settings → the venue you want. For recording you do not need trading permissions. Create an API key with read access only. A key that cannot trade cannot be misused if it leaks, and this is the single cheapest security decision available to you. 2. Choose your universe Two approaches, and the choice matters more than it looks. Scanner — the app ranks by turnover or change rate and follows the top names. Good for finding movement. The list changes through the day, which means the contents of your file are not comparable day to day. Manual list. You name the symbols. Better as a starting point. A small deliberate list produces files you can reason about, and a question like "did volume pick up before the move on my names" is answerable. There is a trap in the scanner approach worth naming. Scanners pick names that are moving, and names that move sharply are disproportionately thin the rest of the time. In our measurement, between 8.5% and 19% of recorded symbol-minutes had no quote update for five minutes or more, and the worst offenders were exactly the small and meme pairs a scanner surfaces. Your recording will contain them. 3. Start recording and leave it Files are written per day and rotate at 20:00 ET, which is the boundary our own recordings use. Nothing needs attention while it runs.

4. What to verify on day two

Before trusting a week of data, spend ten minutes on the first day. Did every symbol you expect appear? Open the file and count tables. A symbol that was in the universe but never traded still produces a table with a quote that never changes. In our recording one symbol did not move once in 1,441 minutes. It was in the universe, it had a quote, and it was not a market. Does the size look right? A one-minute file far below a few megabytes usually means the universe was smaller than intended, or the connection dropped early. Do the timestamps cover the whole day? A gap in the middle is a disconnect. Whether it matters depends on your strategy, but you should know it is there rather than discover it inside a backtest result. How much of it is stale? Count runs where price and both sides of the quote are unchanged. This number is a data quality metric for your own feed, and tracking it session over session turns "the strategy is worse on volatile days" from an opinion into a testable claim — because "my feed drops more on volatile days" produces an identical equity curve and requires a completely different fix.

5. Keeping it

Keep the raw file. Once you convert or aggregate, the original is the only thing that lets you ask a question you did not anticipate. Minute bars can be rebuilt from seconds; the reverse is not true. Back up off the machine. A recording is not reproducible. Store what you did alongside what you recorded. Which venue, which universe rule, which version of the app. A file from four months ago is uninterpretable without it, and you will not remember.

6. What becomes answerable

The questions change once the data is yours, and they get sharper: - How often does the spread on *my* symbols exceed the cost assumption in my backtest? - At my clip size, how often is the visible book too thin to fill me? - How much of my session is my feed actually stale, and does it correlate with bad days? - Does volume really pick up before the move I trade, on my names, not in general? Each of those is an afternoon's work with your own recording and unanswerable without it. That is the whole argument for recording: not that the data is better, but that the questions become yours.