KShivendu https://www.kshivendu.dev/blog Kumar Shivendu's blog en-us hi@kshivendu.dev (Kumar Shivendu) hi@kshivendu.dev (Kumar Shivendu) Tue, 12 May 2026 00:00:00 GMT https://www.kshivendu.dev/blog/tokenization-compression Tokenization is a Compression Codec Nobody Uses That Way https://www.kshivendu.dev/blog/tokenization-compression Vector databases compress text payloads with generic codecs like LZ4, but not with token-aware schemes. BPE tokenization + entropy coding gives you 5x lossless compression using infrastructure already in every ML stack. Tue, 12 May 2026 00:00:00 GMT hi@kshivendu.dev (Kumar Shivendu) compressionvector-databasesinternalsnlp