One Analyzer, Every Language

In Token Search I ran BM25 on a model's own BPE token IDs and matched a Porter-stemmed English analyzer, with no stopword list, no stemmer, no config. That's the credibility floor: it doesn't lose where good tools already exist. The payoff is here. The same tokenizer, with zero per-language setup, beats the tools built for each individual language.

Looking for TL;DR? Check key takeaways

A search engine tokenizes text per language to index it: a stemmer for English, a segmenter for Chinese, a dictionary tokenizer for Thai. Splitting on \w+, which is what a word analyzer does, only works for languages that put spaces between words. Chinese doesn't. Thai barely does. This is exactly where a word analyzer reaches for a language-specific tool, and a BPE tokenizer doesn't.

I ran the same experiment on MIRACL dev sets, with jieba (the standard Chinese segmenter) as the "proper analyzer" baseline, plus a code tab (CodeSearchNet, docstring → function). These are single dev sets on judged-passage pools, so the numbers are directional. Toggle the metric and dataset — for the languages, the zero-rate view lands hardest:

The Chinese result is the whole argument in one bar chart:

• \w+ word tokenization collapses to 0.012 NDCG@10. With no spaces, the regex swallows entire runs of characters into single terms, so queries and documents almost never share a term. Word-BM25 simply does not work on Chinese without a segmenter (and it's shaky on Thai too, another barely-spaced script: 0.35, nothing back for 46% of queries).
• jieba fixes it to 0.40 — that's the cost of a language-specific dependency.
• GPT-4o's o200k tokenizer beats jieba with zero language configuration. Normalized o200k hits 0.53 (13 points over jieba); even the fully config-free naive o200k, the version that comes straight off the model with no \w+ split at all, gets 0.49 — still 9 points over jieba. BPE segments Chinese into shared subwords the same way it does English, and a good multilingual vocabulary does it better than the dedicated segmenter — or than a production engine's own Chinese analyzers (Elasticsearch's cjk 0.49, its default character segmentation 0.52), both still under o200k's 0.53.

Chinese is where token IDs win outright. The sharper test is what happens once you add Elasticsearch's built-in per-language analyzers (cjk/hindi/thai, the production version of "reach for the language-specific tool"):

• One BPE setting lands within a whisker of the hand-built analyzer everywhere. Normalized o200k needs zero per-language configuration and scores 0.53 / 0.60 / 0.59 on Chinese / Hindi / Thai. It beats ES's cjk (0.49) on Chinese, and lands within ~0.02 of ES's hindi stemmer (0.625) and thai word-breaker (0.614) — the gap between "off-the-shelf tokenizer, no setup" and a language team's dedicated analyzer is two hundredths of an NDCG point.
• Character n-grams can edge the analyzers, but they aren't config-free. The chart carries both n-gram sizes, and the winner flips: bigrams top Chinese (0.54, above even token IDs) but trail on Hindi/Thai (0.55 / 0.62); trigrams top Hindi and Thai (0.64 / 0.67, above ES's own analyzers) but drop to 0.45 on Chinese. No single n wins all three, and picking it per script is exactly the per-language tuning this section is trying to avoid — n-grams also cost 4–12× the query latency (toggle the latency metric). A BPE tokenizer sets its own subword boundaries per script: no n to get wrong, and far fewer terms per document.
• Normalization is script-agnostic once the split is. The \w+ regex silently drops Unicode combining marks (Devanagari matras, Thai tone marks), which shreds those words into consonant skeletons — and tanks normalized BPE to 0.27 on Hindi. Splitting on marks-aware runs instead fixes it, and normalized token IDs then beat naive in every language (Hindi 0.55→0.60, Thai 0.56→0.59). The lesson isn't "don't normalize" — it's "don't pre-segment with a Latin-centric regex."
• Tokenizer coverage is decisive. r50k has no Devanagari or Thai merges, so it collapses — 0.007 on Hindi, 0.04 on Thai (the storage-post failure again). o200k, trained multilingually, handles both.

So the payoff isn't "one tokenizer beats every analyzer." It's that one multilingual BPE tokenizer, with no per-language setup, lands within a rounding error of the dedicated tool — winning outright on Chinese, trailing ES's Hindi and Thai analyzers by ~0.02, and never hitting the catastrophe a word analyzer does on an unspaced script (\w+ returns nothing for 98% of Chinese queries).

The same expansion trick generalizes across languages. Swap Porter for a language's ES analyzer as the stemmer — hit _analyze, take the stem, group corpus words by it — and on Hindi token-native BM25 climbs to 0.623, matching ES's dedicated hindi analyzer (0.625) with no stopword list: the stemmer is borrowed through _analyze, and the list is replaced by the rarity prior, whose lowest-rarity words are a 100% match to ES's hand-built Hindi one. It needs both a stemmer and word boundaries, so it's inert on Chinese and Thai — those keep winning the config-free way, through subword tokenization.

Code is the honest check. On CodeSearchNet self-match — the query is the docstring lifted from its own function — GroupCov looks dominant (0.976, subwording getUserById into get·User·By·Id). But that's near-duplicate lookup: every method lands 0.95–0.98 and the median is a perfect hit. Swap in CoSQA (real Bing questions over 20k functions) and the ranking inverts — char n-grams win, Porter helps, and token-native BPE is competitive, not king. The self-match win was the benchmark, not the method: a result that only shows up on near-duplicate lookup won't survive real queries.

Key Takeaways

• One multilingual BPE tokenizer, zero per-language config, lands within a rounding error of the dedicated tool: on Chinese it beats jieba (0.53 vs 0.40) and Elasticsearch's cjk; on Hindi and Thai it's within ~0.02 of ES's hand-built analyzers.
• The failure it avoids is the real win: \w+ word tokenization returns nothing for 98% of Chinese queries. Token-native never hits that catastrophe, in any language, for free.
• Borrow the language's stemmer as query-side expansion (via ES _analyze) and token-native reaches the dedicated Hindi analyzer (0.623 vs 0.625), replacing even the stopword list with a list-free rarity prior.
• Character n-grams can edge the analyzers but need a per-script n (bigrams win Chinese, trigrams win Hindi/Thai) and cost 4–12× the latency. A BPE tokenizer sets its own subword boundaries per script.
• Directional: single MIRACL dev sets on judged-passage pools. The English mechanism behind all this is in Token Search.