KShivendu https://www.kshivendu.dev/blog Kumar Shivendu's blog en-us mail@kshivendu.dev (Kumar Shivendu) mail@kshivendu.dev (Kumar Shivendu) Thu, 17 Sep 2026 00:00:00 GMT https://www.kshivendu.dev/blog/improving-bpe Improving BPE: Stop storing the space, store its absence https://www.kshivendu.dev/blog/improving-bpe BPE spends two vocabulary slots on every word, one with a leading space and one without. I mark the absence of a space instead, which frees ~14,000 slots and beats a properly-trained BPE baseline by 9.4% on code and 0.9% on English. Also: the baseline mistake that made my first results look 4x better than they were. Thu, 17 Sep 2026 00:00:00 GMT mail@kshivendu.dev (Kumar Shivendu) nlpcompressiontokenizationinternals