Latest

OpenAI's and Anthropic's chat APIs take text in and give text out, even though the model underneath samples token IDs at every step. There is precedent for token-ID input (OpenAI's legacy Completions prompt field, vLLM's prompt_token_ids), but no major hosted chat API exposes it end to end. This post analyzes the current input/output shapes and proposes a concrete, opt-in token-ID mode using the same string-or-array interface pattern from the token-native storage post.

Cross-tokenizer translation only comes up on a rare path (fallback provider, degraded model, version migration), and decode-then-re-encode already handles it: 100% lossless at 121.6µs median round trip, negligible next to network or inference latency. I also tried to make it faster with a static lookup table and a splice-based translator. It failed on both correctness (16.2% exact match) and speed (0.97x). The naive path remains the right answer, and the deeper case for a shared vocabulary standard is about not having to maintain a tokenizer-coupled storage layer at all.

I tried to build fake embeddings that match real ones. Matching all five popular structure metrics was easy — and the result was still a useless fog. The thing that actually worked was copying the geometry locally, which also tells you what a transformer's 24 layers are each doing.

Uniform 4-bit quantization costs an embedding model real retrieval quality. Quantizing only the layers that are not doing real work costs nothing — and a layer-by-layer look at the model tells you which layers those are. Validated on mxbai-embed-large + BEIR scifact.

High-dimensional random vectors are nearly equidistant — the 'nearest neighbor' is barely nearer than nothing, and a search index has nothing to navigate by. I unpack what that means, what 'structure' is the cure for, and what structure actually buys: mostly compute, not recall.