Token-Native Storage

Looking for TL;DR? Check key takeaways

In the compression post I showed that storing vector-DB text payloads as BPE token IDs with a static entropy coder gives ~3.35× lossless compression (and ~4.7× with a bigram model). That post is about the ratio. This one is about everything else that falls out of the representation change — consequences that need no new benchmarks, or that turned out to be more interesting than the ratio itself.

Does ~3.35× Hold Outside WikiText-103?

The original number came from WikiText-103, and that corpus turned out to be unusually compressible — curated encyclopedia prose, not the messy web/code/multilingual mix a real deployment actually stores. So I reran the full comparison (byte codecs, raw token-ID packing, and tokenizer+static-ANS, for all three tiktoken vocabularies) on three real, held-out, non-overlapping corpora instead: prose (C4), code (codeparrot-clean, Python), and Hindi (Wikipedia) — at chunk sizes that actually matter for retrieval: 256 and 512 tokens (typical embedding-model chunk sizes — mixedbread and most RAG setups chunk well under 1,000 tokens), plus 2,000 as a longer-context reference point. Every number below is measured, not estimated: real tokenizers, a real compiled ANS coder (constriction), real gzip/zstd/lz4/brotli, with a train/test split so the static frequency table never sees the documents it's evaluated on.

Two findings that change the story from the WikiText-only version:

• WikiText overstated the ratio badly. Real prose (C4) at 2,000 tokens gets 3.38× (o200k+ANS) — a real win, but nowhere near the number a curated Wikipedia corpus implied. Domain matters enormously: code compresses better (up to 3.21×) and Hindi Wikipedia best of all (6.13× with o200k), but that last one is partly an artifact — see the latency section below.
• Raw token-ID packing alone is not a free lunch. Look at the "raw" rows before ANS: on code, r50k raw packing barely breaks even (1.06-1.09×); on Hindi, r50k and cl100k raw packing are worse than doing nothing (0.84-0.87×, because those two tokenizers fall back to near-byte-level tokens for Devanagari). The compression only shows up reliably once the static entropy coder is added — "tokenization gives you free compression" is really "tokenization plus ANS gives you compression," and that distinction matters if you were planning to skip the entropy-coding step.

"LZ4" above compresses each document alone — that's not what a real search engine does. Elasticsearch/Lucene's actual stored-fields codec batches documents into ~16KB (or 128-doc) blocks and compresses the whole block as one LZ4 stream, giving it a much bigger dictionary to find cross-document redundancy in. I added that as its own method ("LZ4 (ES blocks)") to all three charts, simulated the way real Lucene actually does it — verified against the CompressingStoredFieldsWriter source, not guessed: documents grouped in corpus order, batched to the 16KB/128-doc limit, and critically, a trailing partial block still gets compressed at segment flush rather than left raw (Lucene marks it a "dirty chunk" for later merge-time cleanup, but never skips compressing it — my first pass here wrongly discarded those instead).

That fix changes the story for prose specifically: at 2,000-token chunks, each document is already ~9KB, so two of them don't fit in one 16KB block — almost every block ends up holding exactly 1 document, and the ratio (1.44×) converges to exactly what plain per-document LZ4 already gets (also 1.44×). Batching only helps once several documents actually fit in a block together; on code and Hindi, where individual documents are smaller, the benefit holds up (code: 1.86-2.29× vs 1.51-2.06× per-document; Hindi: 1.86-2.01× vs 1.34-1.85×). The mechanism is exactly what you'd expect: batching only pays off when multiple documents fit in one block, and that gets less likely as documents approach the block size themselves. It still falls well short of tokenizer+ANS at every size, and even loses to plain per-document brotli. The real catch is on the decode side: retrieving a single document means decompressing its entire block, a tax that scales with the block's byte size, not the number of documents packed into it — batching helps bulk-write throughput and ratio, not point-read latency, which is the access pattern that matters for RAG.

Reading the two latency charts together against the ratio chart is the real point: brotli gets closest to matching tokenizer+ANS on ratio, but its encode cost is 20-100×+ higher and grows super-linearly with chunk size (256→2,000 tokens is 8× the input, but ~18-35× the brotli-encode time — its quality-11 search doesn't amortize). Tokenizer+ANS latency, by contrast, stays close to linear in token count. So the smaller the chunk — i.e. the more realistic the embedding-chunk scenario — the worse brotli's already-bad latency story gets relative to token-native storage, while the ratio gap barely moves. Hindi's 5.5-6.1× ratio needs its own caveat too: r50k and cl100k fall back to near-byte-level tokens for Devanagari (no real merges learned), so o200k's win there is a real, measured effect of having actual Devanagari BPE merges, not a fluke of the entropy coder.

The bar charts above only show three snapshots (256/512/2,000 tokens); here's the same data as a continuous trend so the scaling behavior is easier to read at a glance — toggle between decode/encode/ratio on the right, and prose/code/hindi on the left, independently:

Three things jump out switching between the tabs (defaults shown are prose — flip the left toggle to check code or Hindi, the shapes aren't identical across datasets). Decode: "LZ4 (ES blocks)" is the one line that doesn't fit the pattern — on prose it falls as chunk size grows (20.5→19.1→11.3µs) while everything else climbs, because its decode cost tracks the block's byte size (capped at ~16KB), and prose documents get so close to that cap at 2,000 tokens that almost every block holds exactly one of them — smaller blocks, faster decode, but no batching benefit left either (see the ratio tab). Code shows the same falling shape but less dramatically (18.0→16.6→14.4µs, smaller documents leave more batching headroom); Hindi dips at 512 instead of falling monotonically — same underlying mechanism, different byte sizes per document, different curve. Encode: brotli's line isn't just the highest, it's the steepest on this log-log plot — every other method is roughly straight (near-linear cost in token count), but brotli visibly bends upward, confirming its quality-11 search cost grows faster than the input itself, in all three datasets. Ratio: tokenizer+ANS (the top three lines) barely moves across a nearly 8× range of chunk sizes, while every byte codec climbs with more context to work with — extrapolate leftward past 256 tokens, toward the very small chunks some embedding models push toward, and the byte codecs would keep dropping while tokenizer+ANS stays roughly where it is, since the static table never depended on a per-chunk dictionary window in the first place.

Does the Faster-Than-ANS Alternative Hold Outside WikiText Too?

The main post proposed frequency-sorted token IDs + streamvbyte as a cheaper alternative to ANS, but only measured it on WikiText. So does it hold up on real prose/code/Hindi too?

Same corpora, same chunk sizes, same train/test split as everywhere else in this post. I measured decode as agent-decode: streamvbyte-decode plus a rank lookup straight to token IDs, no detokenize, because a model reading a retrieved chunk wants IDs, not text. Native tokenizer per domain (r50k/prose, cl100k/code, o200k/hindi):

It holds, and more consistently than I expected. Freq-remap decodes 1.5-5.7× faster than ANS everywhere, for about 15-26% less compression. Encode is basically a wash since both spend most of their time in the shared tokenize step anyway. The usual Hindi catch shows up here too: r50k/cl100k only get 1.3-1.4× on Hindi, which is a tokenizer problem, not a remap problem.

Which one you'd actually reach for depends on what you're optimizing for. Care about decode latency (agents on the hot path)? Freq-remap. Care about ratio (cold storage, egress)? ANS. Nothing stops you running both either, ANS at rest and freq-remap for a hot cache, since they're the same token IDs underneath.

What About zstd --train Beyond WikiText?

The main post's fairest byte-codec competitor is zstd --train, a dictionary trained on the corpus itself rather than the generic zstd tables. On WikiText it reached 2.61x, a real number but still 30% behind r50k+ANS's 3.37x. I trained the same 112KB dictionary on each domain's own train split and ran it at 256/512/2,000 tokens:

The WikiText gap doesn't hold everywhere. On prose it's about the same story as WikiText (2.87x vs r50k+ANS's 3.29-3.45x, ANS still ahead by a real margin). But on code, zstd-dict actually beats cl100k+ANS at 512 and 2,000 tokens (3.24x and 4.43x vs roughly 3.19x and 3.13x). Code has a lot of repeated structure, imports, indentation, common function signatures, and a trained dictionary's LZ77 matching picks that up better than a static token-frequency table does. On Hindi it closes most of the gap too (5.09x vs o200k+ANS's 6.13x), and it does that without needing a tokenizer that actually has Devanagari merges in the first place.

Decode stays cheap regardless, single-digit to double-digit microseconds, since attaching a dictionary doesn't change how fast zstd decompresses. Encode is the cost here, and it's steep: prose at 2,000 tokens takes almost 12ms, by far the slowest encode of anything in this post. So zstd-dict isn't a blanket win, it's a domain-dependent one. For code specifically, it's a legitimate alternative to tokenizer+ANS, and one that doesn't require picking a tokenizer at all.

The Multiplier Hits Every Copy of the Bytes

Compression headlines always quote primary storage: "31.5 GB becomes 9.4 GB." But a database never stores bytes once. The same 3.35× applies mechanically to:

• Snapshots and backups — every copy is 3.35× smaller
• WAL and replication traffic — 3.35× less data shipped per write
• Shard rebalancing — scale-out moves 3.35× less data over the network
• Page cache density — 3.35× more payloads fit in the same RAM, an effective memory upgrade for payload-heavy read workloads
• Egress — at $0.09/GB, every bulk transfer, cross-region sync, and client response shrinks proportionally

None of this needs separate measurement; it's the same bytes multiplied through the system. For a distributed deployment, rebalancing 1.3 TB instead of 4.5 TB is an operational win that doesn't show up on the storage bill at all.

The same logic makes token-native storage most attractive where storage is scarcest: edge and on-device RAG. A 3× payload reduction on a phone or an embedded deployment buys more than a few hundred dollars a month does on a server — and the static frequency table ships once with the binary.

A Free Byproduct: Out-of-Distribution Detection

The ANS encoder computes -log2 P(token) for every token it codes — so every compressed document gets a bits/token score for free. Documents that compress badly under the static table are documents that don't look like the training corpus. That's an ingest-time data-quality filter with zero extra compute.

I tested it: WikiText test articles (clean, 10.8 ± 0.5 bits/token) against length-matched probe classes, reporting AUC (1.0 = perfectly separable from clean, 0.5 = indistinguishable):

Two honest observations:

1. The byte-codec baseline is not just worse — it's directionless. Repetitive junk (spam, lorem ipsum, HTML, code) compresses better than clean prose under zstd, so a compression-ratio threshold flags it in the wrong direction (AUC ≈ 0). Even hex dumps slip through (16-character alphabet packs well at byte level). The token-entropy score separates every class cleanly — including marketing spam, whose words are common English but rare in encyclopedic text.

2. It's an out-of-distribution detector, not a junk detector. The last row is the catch: legitimate Python code also scores AUC 1.000, because the table was trained on Wikipedia prose. If your corpus legitimately mixes domains, you either train the table on a matching mix or maintain per-domain tables (which codec versioning already requires you to support). For a single-domain corpus, "out of distribution" and "junk" coincide — and the filter is free.

The same score works as a drift monitor: if the corpus-average compression ratio degrades over time, your data distribution moved — worth knowing, because your embedding model is now out-of-domain too.

LLM Output Is Born as Tokens

The compression pipeline also runs in reverse. LLM output is born as token IDs — the sampler emits them before anything detokenizes to UTF-8. Today the standard path is: model emits tokens → detokenize to text → store → compress with a byte codec. Storing generated text (chat logs, summaries, agent traces) token-natively has zero encode cost: you persist what the model already produced instead of detokenizing and re-compressing it.

With agent traces becoming a major storage category, this is the cleanest version of the argument that storage and inference should share a representation. The caveat is coordination: it only works when the storage tokenizer matches the model's, and model tokenizers churn across versions — so treat it as an optimization for systems that control both ends, not a portable guarantee.

The Inference-Side Continuation

The storage layer speaking tokens is also the prerequisite for inference-side wins — skipping re-tokenization at serving time, and KV-cache reuse for hot documents. That thread has its own experiments and its own surprises (the hot-doc opportunity turns out to hinge entirely on how concentrated your query traffic is): The Hot Doc Problem: KV Cache Tiering for RAG.

Key Takeaways

• The compression multiplier applies to every copy of the bytes — snapshots, WAL, replication, shard rebalancing, page-cache density, egress — not just primary storage. The operational wins (rebalancing, cache density) may matter more than the storage bill.
• Edge and on-device RAG benefit most: storage is scarcest there, and the frequency table ships once with the binary.
• The per-document bits/token score is a free out-of-distribution detector: AUC 1.0 on every junk class tested, where zstd-ratio thresholds fail in both directions. But it flags anything unlike the training corpus — including legitimate code — so the table must match the corpus domain. It doubles as a corpus drift monitor.
• LLM output is born as token IDs — storing chat logs and agent traces token-natively has zero encode cost, if you control both the storage codec and the model tokenizer.
• Scope it honestly: in vector-heavy collections the vectors, not the payloads, often dominate total bytes. These wins are largest for payload-heavy workloads.

How Far Can Token Compression Go?

The static frequency table only knows how often each token appears globally, and it's small: ~196 KB for r50k (50,257 tokens × 4 bytes). The next step up is a bigram table (P(token | previous token)), which is a genuinely bigger structure, not a rounding error on the unigram table. Pruned to the top 175,000 (prev, cur) pairs by frequency, it costs about 1.34 MB, roughly 7x the unigram table's size, and gets you to about 4.35x (analytic estimate). A full bigram table (69 MB) reaches ~4.7x.

I built a real bigram+backoff codec (absolute discounting, d=0.75, backing off to the unigram table for unseen pairs) and ran it end to end on WikiText-103, encode, decode, and roundtrip, rather than trusting the analytic estimate:

• Ratio: 4.094x measured on 30 real test articles, close to the 4.35x analytic estimate, and every article round-tripped exactly.
• Latency: unlike the unigram table, which is looked up once and reused for the whole document in one vectorized ANS call, a bigram model needs a different distribution for every token (whichever token preceded it). Once each distinct prev_tok distribution is cached in memory, steady-state cost is about 1.0µs/token to encode and 0.7µs/token to decode, roughly 10x slower per token than unigram ANS. The first time a given prev_tok is seen, building its distribution costs ~240µs; in a shared production table this is a one-time cost per distinct token (there are at most ~50K possible, and typically only ~1,000 are common enough to show up), so it amortizes away almost immediately.
• Domain mismatch: a bigram table inherits the same sensitivity to corpus mismatch as the unigram table. A WikiText-trained bigram model applied to JSON gets 1.126x, essentially identical to WikiText-trained unigram ANS on JSON (1.10x), because most JSON bigrams aren't in the pruned pair table and the model backs off to the same domain-biased unigram distribution. Higher order doesn't buy any extra robustness to mismatch, only a better fit when the table matches your data.

Past that is LM territory. Chinchilla 70B achieves 0.664 bits/byte on Wikipedia (roughly 12× compression) because a language model is literally computing P(token | all previous tokens), which is the optimal entropy coder by definition. The gap between 3.3× and 12× is entirely conditional probability.

And because the stored format is just token IDs, swapping the unigram table for a bigram table later is a codec upgrade, not a data migration.