Token-Native Storage: Read and Write in the Language Models Already Speak

Looking for TL;DR? Check key takeaways

Every vector search engine stores two things for each point: the vector and the payload. Vectors get real compression effort with quantization and MRL. The text payload gets raw UTF-8, or LZ4 at best, even though the systems that actually consume it, embedders, rerankers, LLMs, all operate on tokens, not bytes.

Token IDs alone, with no compression algorithm running at all, already achieves 2.33× compression over raw UTF-8 bytes. A static entropy based compression on top reaches 3.35×.

This post argues for token-native storage: persist text as BPE token IDs, not UTF-8 bytes. This mismatch of language isn't unique to vector search engines, it shows up anywhere models read and write: web, search, LLM chat history, or agent traces from RL rollouts.

How Token-Native Storage Works

The top path is what databases do today: store UTF-8 bytes, compress with a generic codec. The bottom path is token-native storage: tokenize first, then compress (here entropy-code) the token IDs.

Step 1: BPE Tokenization

BPE (Byte Pair Encoding) starts with raw bytes and repeatedly merges the most frequent adjacent pair into a new token, building up a vocabulary of common subwords. OpenAI ships two BPE tokenizers: r50k (used by GPT-2/GPT-3, 50,257 tokens) and cl100k (used by GPT-3.5/GPT-4, 100,277 tokens). cl100k has a larger vocabulary so it learns longer merges, which usually (but not always) means fewer tokens per document, especially for code and non-English text. Either way, each token ID needs 4 bytes (uint32) since the vocab exceeds the uint16 limit of 65,535. r50k fits in 2 bytes (uint16).

BPE merges frequent byte sequences into single tokens. "storage" is one token. "Token-native" is three (Token + - + native). On average one BPE token covers 3/4 of a word. An average English word is ~5 characters plus a space, so:

6 bytes/word × 3/4 words/token = 4.5 bytes/token (raw text)
2 bytes/token (r50k uint16 token IDs)
ratio: ~2.25x

This napkin math is what gave me the motivation to run the experiment: just packing token IDs into uint16 already gives compression, even before any true compression algorithm starts. ~2.25× from token IDs alone, before an entropy coder even runs, is confirmed by the benchmark below.

Here's another way to look at this: every token costs a flat 2 bytes (uint16). A word almost always comes with a leading space in a real sentence, and BPE merges that space straight into the next word for free, no extra token spent on it. So packing wins as soon as a word is longer than 1 character. " cat" is 4 raw bytes (space + 3 letters) but just 1 token, 2 bytes, a clean win.

Step 2 (Optional): ANS Entropy Coding

Step 1 alone already gets you the ~2.25× above, raw uint16 packing with no compression algorithm running at all. This step is for squeezing further: gzip and LZ4 see opaque bytes and can't know that [0x01, 0x7F] is token 383. An entropy coder working directly on token IDs can give frequent tokens short codes: "the" is 40x more common than "embeddings", so it gets a much shorter code. I used ANS (Asymmetric Numeral Systems, the entropy coder inside zstd) via the constriction library.

One design choice matters here: the frequency table is trained once on a large corpus and shared, not built per document. A per-document table compresses better on paper, but the decoder needs the table to reconstruct the document, so you'd have to store it alongside each payload. That's ~900 bytes for a typical 400-token chunk, which erases the entire gain. A shared static table costs nothing per document.

import constriction, numpy as np, tiktoken

enc   = tiktoken.get_encoding("r50k_base")
VOCAB = 50_257

# ── one-time setup: train on corpus ──────────────────────────────────────────
corpus_ids = enc.encode(corpus_text)
counts = np.bincount(corpus_ids, minlength=VOCAB).astype(np.float64) + 1  # Laplace
probs  = counts / counts.sum()
model  = constriction.stream.model.Categorical(probs, perfect=False)

# ── per-document encode (store this) ─────────────────────────────────────────
def compress(text):
    ids   = enc.encode(text)
    coder = constriction.stream.stack.AnsCoder()
    coder.encode_reverse(np.array(ids, dtype=np.int32), model)
    return len(ids).to_bytes(4, 'big') + coder.get_compressed().tobytes()

# ── per-document decode ───────────────────────────────────────────────────────
def decompress(data):
    n   = int.from_bytes(data[:4], 'big')
    buf = np.frombuffer(data[4:], dtype=np.uint32).copy()
    ids = constriction.stream.stack.AnsCoder(buf).decode(model, n).tolist()
    return enc.decode(ids)

What Compression Buys You

The Benchmark

Measured across 1,773 WikiText-103 test articles (≥30 words). The ANS frequency table was trained on the WikiText-103 train split, a shared static model with no per-document overhead. All methods below are lossless (0% character error rate on full round trip).

Two things stand out. r50k raw uint16 (no compression algorithm at all) already beats every byte codec except brotli. And r50k + static ANS beats everything by a wide margin. cl100k raw uint32 sits near 1.13x, worse than even gzip, yet cl100k + ANS recovers to 3.30x: the fixed-width storage penalty vanishes once ANS allocates bits by frequency instead of by vocab size.

The fairest competitor is zstd with a corpus-trained dictionary (zstd --train), since it also learns from the corpus. It reaches 2.61x median, still 30% behind. The difference is that the token model operates on units of language: "embeddings" is one symbol with a known global frequency, where a byte-level model has to rediscover that pattern from scratch.

What does it cost in latency?

Measured across 200 WikiText-103 articles (50–500 words), static corpus ANS model:

LZ4 decompresses in 1µs, designed entirely for speed. zstd dict's encoder is slowest at 128µs (best of three dictionary sizes tested: 32KB, 64KB, 112KB), though it decompresses in 2µs. The tokenizer-only bars isolate what enc.encode/enc.decode alone cost, and the "+ ANS" segment stacked on top shows what entropy coding adds. The split flips between encode and decode. Encode is tokenizer-dominated: BPE's merge search costs 43-54µs, ANS only adds 9-10µs on top. Decode is ANS-dominated: detokenizing is a 4µs table lookup, but ANS decoding costs 15-17µs, most of the 19-21µs total. Either way the tokenizer you'd run regardless (to get token IDs to embed or feed to an LLM) already accounts for most of the encode cost.

Note that today's embedding models use different tokenizers (WordPiece, SentencePiece) than LLM BPE tokenizers, so you can't skip the tokenization step by reusing embedding tokenization. The real benefit runs the other way: if you store token IDs in LLM format (r50k/cl100k), you avoid re-tokenizing retrieved documents at inference time when feeding them to the LLM, since the token IDs are already there.

What It Saves at Scale

English Wikipedia is ~7M articles averaging 708 words. At the 3.35x median ratio, that's 31.5 GB of raw UTF-8 down to 9.4 GB, and at a billion documents, 4.5 TB down to 1.3 TB. At $0.08/GB for SSD and $0.02/GB for S3-class object storage:

One scoping note so the claim doesn't overreach: how this shakes out depends heavily on chunk size and vector compression.

Compression doesn't just shrink primary storage. Every copy of your data, from snapshots and backups to write-ahead logs, replication, shard rebalancing, cache density, and egress, benefits from the same 3.35× reduction. That means less data shipped across the network, more payloads fitting in RAM, and faster reads and indexing, all multiplying the operational impact well beyond the headline storage savings. This effect is especially valuable in storage-constrained contexts like edge devices or on-device RAG, where shrinking the payload pays off far more than marginal savings on a central server.

How Far Can Token Compression Go?

The static frequency table only knows how often each token appears globally. The next step up is a bigram table (P(token | previous token)) which costs 1.4 MB and gets you to about 4.35x. A full bigram table (69 MB) reaches ~4.7x.

Past that is LM territory. Chinchilla 70B achieves 0.664 bits/byte on Wikipedia (roughly 12× compression) because a language model is literally computing P(token | all previous tokens), which is the optimal entropy coder by definition. The gap between 3.3× and 12× is entirely conditional probability.

And because the stored format is just token IDs, swapping the unigram table for a bigram table later is a codec upgrade, not a data migration.

What This Looks Like in Practice

When you insert a document into Qdrant, you typically send something like:

// Current approach for inserting data:
PUT /collections/my_collection/points
{
  "points": [
      {
        "id": 123,
        "vector": [0.12, -0.34, ...],
        "payload": { "text": "Token-native storage persists text as token IDs, not UTF-8 bytes..." }
      },
  ]
}

Qdrant compresses each payload value independently with LZ4. Elasticsearch instead shares an LZ4 window across a block of documents, trading that independence for a better ratio. Either way, the codec has no learned model of the language, just whatever byte patterns happen to repeat within its window.

But there is a component that already knows a lot about the structure of natural language, and it's part of every embedding and language model stack: the tokenizer. Token-native storage makes token IDs the on-disk representation instead of throwing them away right after tokenizing:

// One time: Set tokenizer on collection level for the 'text' field
PUT /collections/my_collection/index
{
  "schema": { "text": { "type": "token", "tokenizer": "r50k" } }
}

The insert call barely changes: text still takes plain text, or a client with token IDs on hand can send those instead, as a plain array:

// Insert with token IDs directly instead of text
PUT /collections/my_collection/points
{
  "points": [
    {
      "id": 123,
      "vector": [0.12, -0.34, ...],
      "payload": { "text": [1858, 6427, 20272, 318, 257, 6997, ...] }
    }
  ]
}

A string or an array is easy to tell apart, so the database picks the right path: tokenize-then-compress for text, compress directly for token IDs. Either way, ANS compression happens inside the database, on write, and gets reversed on read, so the interface never exposes compressed bytes. The tokenizer itself is a collection-level choice, not a per-document one: mixing tokenizers in the same collection would mean mixing vocabularies, and the static ANS table only makes sense for one vocabulary at a time.

POST /collections/my_collection/points/search
{
  "vector": [...],
  "limit": 10,
  "with_payload": { "text": "tokens" }
}

The same write path applies when the writer is a model instead of a client. An LLM's sampler produces token IDs before anything detokenizes them to text, so storing generated output (chat logs, summaries, agent traces) as tokens has zero encode cost: persist what the model already produced instead of detokenizing and re-compressing it.

Limitations

• No shared vocabulary across models yet. Embedding models use WordPiece or SentencePiece, LLMs use BPE variants (r50k, cl100k, o200k), and different LLM families don't share one either. A token-native payload only pays off end-to-end for a consumer using the same tokenizer it was stored with. That's exactly the case for a common token vocabulary standard, the way ASCII and UTF-8 standardized bytes, rather than treating token-native storage as a single-vendor optimization.
• Hosted LLM APIs don't accept or return token IDs today. Even when your storage tokenizer matches the model's exactly, the read path only closes the loop end to end if you own the inference stack: hosted chat APIs take text prompts and return text, not raw token arrays, so a detokenize/retokenize step still happens somewhere. See this follow-up proposing a token-ID-native LLM API for what closing that gap would actually take.
• Raw packing efficiency depends on the tokenizer and the script. cl100k's larger vocabulary doesn't always mean fewer tokens: on long digit strings and emoji-heavy text it can produce more tokens than r50k, since its extra vocabulary budget went toward code and non-English scripts. And for scripts a tokenizer has no merges for, uint16 packing can lose outright: the Thai word การ is 9 raw UTF-8 bytes but 6 single-byte tokens, 12 bytes as uint16, 33% worse than doing nothing.
• ANS is one entropy coder, not the point of this post. It's what got the benchmark from 2.33x (raw packing alone) to 3.35x, but it comes with real costs of its own: decode is slower than LZ4 (see latency), the frequency table is only as good as the corpus it's trained on (a WikiText-trained table gets 1.10x on JSON, a JSON-trained table gets 2.24x), and it only shares well at the corpus level, not per block (a table trained fresh on each 16KB block costs 900-4,000 bytes of overhead, losing to a static table even at 1MB blocks). A different entropy coder could trade some of this differently, and a tokenizer that assigned token IDs by frequency rank in the first place could shrink or remove the need for a separate entropy-coding step altogether. The core argument, store tokens instead of bytes, holds regardless of which coder, if any, you layer on top.

Key Takeaways

• Store text as tokens, not UTF-8 bytes. Vector databases compress payloads with LZ4 (~1.15× per document). Token-native storage — BPE token IDs + static ANS — gives 3.35× lossless compression and aligns the on-disk format with how embedders, rerankers, and LLMs actually consume text.
• The compression multiplier applies to every copy of the bytes — snapshots, WAL, replication, shard rebalancing, page-cache density, egress — not just primary storage.
• Raw uint16 token IDs alone reach 2.4×, beating every byte codec except brotli, with no compression algorithm at all.
• LLM output is born as token IDs — storing chat logs and agent traces token-natively has zero encode cost, if you control both the storage codec and the model tokenizer.
• The frequency table must be static and shared. Per-document tables cost ~900 bytes each and erase the gain.
• Latency is a non-issue: encode costs ~50-65µs (mostly the tokenizer, ANS adds only ~9-10µs), decode costs ~19-21µs (mostly ANS, the tokenizer's share is just ~4µs).
• The same idea stretches to ~4.7× with a bigram table (1.4 MB of table already gets 4.35×). Past ~5× you need an actual language model.
• The trade-off to respect: the vocabulary becomes part of your on-disk format forever, so version your encodings.

Acknowledgements

The benchmarks use tiktoken, zstandard, lz4, and constriction. The "language modeling is compression" framing comes from DeepMind's 2023 paper.

Citation

If you find this useful, please cite:

@misc{kumar2026tokenstorage,
  author       = {Kumar Shivendu},
  title        = {Token-Native Storage: Read and Write in the Language Models Already Speak},
  year         = {2026},
  url          = {https://www.kshivendu.dev/blog/token-storage},
  note         = {Blog post}
}