limitbookLive demo

ITCH 5.0 Parser & Limit Order Book Engine

  • Rust
  • zero-copy
  • no_std
  • WASM
  • criterion
  • CI
01

Overview

NASDAQ TotalView-ITCH 5.0 is the exchange's binary market-data feed: every order added, executed, cancelled, and replaced, broadcast as a stream of tightly packed big-endian records with no padding and no self-description. A single trading day is 268,744,780 messages — 3.5 GB gzipped, 8.25 GB decompressed. limitbook parses that feed and reconstructs the full limit order book from it.

Zero-copy parsing means the decoder never allocates and never copies: each message is a view over the caller's byte slice, with fields read in place. The payoff is not only speed but placement — the core crate, limitbook-core, is no_std with no file or OS I/O at all, so the same parser that chews through a day on the CLI also targets wasm32-unknown-unknown without modification. The workspace splits into limitbook-core, limitbook-cli, and limitbook-wasm along exactly that line.

The bar for this project was correctness, not throughput. Anyone can parse quickly if the book is allowed to drift. limitbook's standard was replaying the entire day and finishing with zero invariant violations — no crossed book, no negative quantities, no orphaned order IDs, no reference-count leaks — and only then asking how fast it went.

02

The Measurement

These four numbers measure four different pipelines and conflating them would be meaningless, so each is labelled with exactly what it includes. The gap between parse-only and parse-plus-book is the finding: the book dominates by more than an order of magnitude, so the parser was never the thing worth optimising.

pipelinethroughputwhat it includes
parse-only31.9M msg/sframe + decode, from decompressed bytes in RAM (≈0.98 GB/s)
parse + book1.38M msg/sfull limit order book reconstruction, from RAM
end-to-end1.15M msg/sincluding gunzip, streaming from the 3.5 GB .gz
verification replay0.70M msg/sfull replay with invariant checking — 0 violations, CLEAN

4-vCPU Xeon at 2.80 GHz, single core via taskset, warm · ±5–10% run variance on a shared cloud VM. The full day is 268,744,780 messages: 3.5 GB gzipped, 8.25 GB decompressed.

Flamegraph finding: a crossed-book re-arming check (refresh_armed) burned 23.0% of the pass on provably no-op work; one early return took it to 7.5%, with byte-identical replay output verified against the pre-change baseline.

All 23 spec message types decode in 22–43ns.

03 —

See It In Action

The repository has the full write-up — the message-type decode table, the criterion harness, the flamegraphs, and the invariant checker that had to come back CLEAN before any of the throughput numbers counted.

04

Links