Published on

Token-Native LLM APIs: Input and Output as Token IDs

Looking for TL;DR? Check key takeaways

The token-native storage post argued for persisting text as BPE token IDs instead of UTF-8 bytes, since embedders, rerankers, and LLMs all consume tokens, not bytes. It also flagged a limit on how far that argument reaches: reading a payload back as raw token IDs only pays off end to end "if you own the inference stack, since hosted LLM APIs take text prompts, not raw token arrays (yet)." This post is about that "yet". What would it actually take for a hosted LLM API to accept token IDs as input and return them as output, instead of forcing a detokenize-then-retokenize round trip on every call?

What Hosted APIs Accept Today

Every input path on the major hosted chat APIs is text, or text plus a small set of structured content types (images, documents, tool results). OpenAI's Chat Completions and Responses APIs take a messages array whose content is a string or a list of typed content blocks (text, image_url, and similar); none of those types is "array of token IDs". Anthropic's Messages API is the same shape: content is a string or an array of blocks (text, image, tool_result), with no token-ID block type.

There is one real precedent, and it's easy to miss because it lives on a deprecated endpoint. OpenAI's legacy Completions API (/v1/completions) has always typed its prompt field as "string, array of string, array of number, or array of array of number", where "array of number" is literally a list of token IDs from the model's own tokenizer. This isn't a proposal, it's documented, shipped behavior. The catch is where it's shipped: the Completions API only works with legacy completion models (gpt-3.5-turbo-instruct and older base models), not with any current chat model, and OpenAI has frozen the endpoint since mid-2023 rather than extending it forward. The precedent for token-ID input exists in the same company's API surface, it's just stranded on a model family nobody is pointing new traffic at.

What Hosted APIs Return Today

Output is text on every hosted chat endpoint, with token counts in usage but never the token ID sequence the model actually sampled. The closest thing to token-level detail is logprobs/top_logprobs, and it's worth being precise about what that field contains, because it looks token-adjacent but isn't:

{
  "token": " model",
  "bytes": [32, 109, 111, 100, 101, 108],
  "logprob": -0.234,
  "top_logprobs": [
    { "token": " model", "bytes": [32, 109, 111, 100, 101, 108], "logprob": -0.234 },
    { "token": " system", "bytes": [32, 115, 121, 115, 116, 101, 109], "logprob": -2.891 }
  ]
}

That's the shape OpenAI's Chat Completions API returns per position when logprobs is on. It carries the decoded token string, the raw UTF-8 bytes of that string, and a log probability, but never the integer vocabulary ID. You can reconstruct the text (the API already does that for you), and you can rank candidates by probability, but you cannot recover which row of the model's embedding table " model" actually was without re-tokenizing the string yourself and hoping your tokenizer matches the server's exactly. As of this writing, Anthropic's Messages API doesn't expose a logprobs parameter at all; a public request for OpenAI-shaped logprobs support exists as a community proposal, not a shipped feature.

Streaming makes the same point from a different angle. Both providers stream text deltas, a delta.content string per chunk on OpenAI's chat completions, an incremental text event on Anthropic's Messages API. Neither exposes the token ID that produced a given chunk mid-stream. The model samples a token ID, decodes it to text, and only the decoded text crosses the wire, on every chunk, for the entire response.

The Precedent Already Exists Server-Side

The gap isn't that token-ID-native serving is hard, it's that hosted chat APIs don't expose it. Open-source serving stacks already do, because they sit closer to the model.

vLLM's LLM class accepts prompt_token_ids directly, bypassing the tokenizer entirely, and its skip_tokenizer_init option goes further: with it set, the engine expects prompt_token_ids on the way in and returns token IDs on the way out, no text touches the request or response path at all. Text-generation-inference's details mode returns a token-level breakdown per generated position, including the integer id, alongside text and logprob, though on the input side TGI's documented routes still take a text prompt rather than pre-tokenized IDs.

So the pattern proposed below isn't speculative engineering, it's a hosted-API wrapper around something that already runs in production serving stacks. The question is just whether a provider is willing to put that surface behind a public, versioned parameter.

A Concrete Interface Proposal

The token-native storage post solved almost this exact interface problem once already, for the write and read paths of a vector database payload. The same shape carries over here with a substitution of nouns: prompt instead of payload, model tokenizer instead of collection tokenizer.

Input. A chat message's content field currently accepts a string or an array of content blocks. Add a block type that carries token IDs directly:

// Current: content is text
POST /v1/chat/completions
{
  "model": "gpt-5.1",
  "messages": [
    { "role": "user", "content": "Summarize the attached document." }
  ]
}

// Proposed: a token-ID content block, type-detected like the storage post's insert path
POST /v1/chat/completions
{
  "model": "gpt-5.1",
  "tokenizer_version": "gpt-5.1-2026-07",
  "messages": [
    { "role": "user", "content": { "type": "input_tokens", "token_ids": [3838, 42891, 262, 5301, 6190, 13] } }
  ]
}

A string is still a string, an object with type: "input_tokens" is unambiguous, so the server can dispatch on shape exactly the way token-native storage's insert endpoint tells a plain string apart from a plain array. The one addition with no analogue on the storage side is tokenizer_version: a vector database's tokenizer is a durable, collection-level choice, but a hosted API's tokenizer changes across model versions, so the request has to pin the exact tokenizer the caller tokenized against, not just the model family.

Output. Mirror the read-side idea from the storage post, where with_payload grew a per-field format string ("text" vs "tokens") instead of staying a plain boolean. A chat completion could take the same kind of parameter on the response:

POST /v1/chat/completions
{
  "model": "gpt-5.1",
  "tokenizer_version": "gpt-5.1-2026-07",
  "response_format_tokens": true,
  "messages": [ ... ]
}

{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "The document proposes...",
        "token_ids": [464, 6190, 26057, 986]
      }
    }
  ]
}

Returning both content and token_ids costs nothing extra to compute, since the server already has the ID sequence before it decodes to text, decoding is the cheap direction. A caller that wants tokens only, to skip shipping text over the wire at all, could ask for token_ids and omit content, the same way token-native storage's read path lets a caller skip text and ask for "tokens" only.

Streaming would carry the same field per chunk instead of only delta.content:

{ "choices": [{ "delta": { "content": " model", "token_ids": [2746] } }] }

Why This Would Have To Be Opt-In and Versioned

The previous post's closing limitation was that the vocabulary and frequency table become part of the on-disk format forever, so any storage layer built on token IDs has to version its encodings. The same constraint applies here, with a sharper edge, because a hosted API's tokenizer isn't a collection-level choice you control, it's the provider's, and it can change underneath you between model versions without warning if you're only looking at text.

That's why tokenizer_version in the proposal above isn't decorative. Token IDs are only meaningful relative to the exact vocabulary that produced them; the client and the API have to agree on that vocabulary or the integers decode to garbage. A token-ID mode tied loosely to "the model" rather than pinned to a specific tokenizer version would break silently on the provider's next model bump, which is a worse failure mode than today's text interface, where a prompt string is portable across every model that exists, forever, because text has no version. This is also why token-ID mode can't be portable across model families the way text is: r50k and cl100k IDs mean nothing to a Claude or Llama tokenizer, so a client sending token IDs is committing to one specific provider's one specific tokenizer version for that request, not writing a provider-agnostic prompt. The tokenizer translation post showed that decode-then-re-encode across tokenizers is lossless and cheap (121.6µs median) precisely because it always goes through text as the intermediate, portable form. A token-ID-native API can't take that shortcut for a cross-provider fallback, it would still have to decode to text on one side and re-encode on the other, meaning any system built on this mode needs a text fallback path anyway for exactly the case the earlier post covered: a degraded route, a different model family, a migration.

The Actual Payoff

For a system that has already gone token-native on storage, per the first post in this series, closing the loop here removes the last detokenize/retokenize pair in the pipeline. Today, even with token-native storage, an LLM call still forces two conversions that store nothing about the data changed shape: the retrieved payload gets detokenized to text to build the prompt, and the model's own output gets detokenized to text by the provider before it reaches you, only for you to possibly retokenize that output to store it token-natively again per the first post's argument that LLM output is born as token IDs at zero encode cost. A token-ID-native request and response removes both conversions when the storage tokenizer, the model's tokenizer, and the tokenizer version all agree: retrieve token IDs from storage, send them directly as the prompt, receive token IDs back, store those directly. No text ever gets materialized in the middle of the pipeline, only at the edges where a human actually needs to read something.

Limitations

  • This is a proposal, not a shipped feature. No major hosted chat API (OpenAI Chat Completions/Responses, Anthropic Messages) currently accepts token-ID input or returns token-ID output on its primary endpoints. Everything under "A Concrete Interface Proposal" above is this post's design, not documentation of something that exists.
  • The verified precedent is narrower than it might read. OpenAI's legacy Completions prompt field genuinely does accept "array of number" today, that's a documented, shipped API contract, not a rumor, but it's frozen to legacy completion models and has received no updates since 2023. vLLM's prompt_token_ids and skip_tokenizer_init are real, documented, and used in production, but at the serving-stack layer, not behind any hosted provider's public API. TGI's per-token id field in details mode is real on output; I did not find a documented token-ID input path for TGI's standard routes.
  • Anthropic's logprobs situation is unclear from public sources at the time of this research, beyond a community feature request; if Anthropic ships logprobs or token-level output later, the comparison in this post's "What Hosted APIs Return Today" section may already be stale.
  • Tokenizer versioning is a real adoption blocker, not a formality. A provider that ships this has to keep old tokenizer versions addressable indefinitely, or every token-ID request breaks the moment they retrain a tokenizer for a new model, which is a maintenance burden they don't currently carry for text.
  • There are plausible reasons providers might not want this, and this post may be underweighting them. Text is the layer where content-safety filtering, prompt-injection detection, and moderation classifiers run today; a raw token-ID input path is a second entry point into the model that bypasses however much of that filtering is string-pattern-based rather than embedding-based, and it's not obvious how much of it is the latter. Token IDs are also a plausible obfuscation channel: a sequence that decodes to something a text filter would catch can be constructed directly in ID space without ever producing the flagged string. Caching behavior is another open question. Prompt caching on both major providers keys off exact prefixes; whether token-ID input changes cache hit rates, cache key computation, or interacts badly with speculative decoding isn't something I have evidence for either way. None of these are arguments that token-ID input is unsafe or unworkable, they're reasons adoption could be slower or narrower than this post's proposal assumes, and I don't have a rebuttal for any of them beyond "worth testing."
  • No adoption timeline is implied. This post argues the interface is coherent and has partial precedent, not that any provider is planning to build it or that it would ship soon if they did.

Key Takeaways

  • No major hosted chat API accepts token-ID input or returns token-ID output today. OpenAI's Chat Completions/Responses and Anthropic's Messages API are text (plus structured content blocks) in, text out, on every primary endpoint.
  • OpenAI's legacy Completions API is real, verified precedent for token-ID input: its prompt field has documented support for "array of number" (token IDs), but only on legacy completion models, frozen since 2023.
  • logprobs/top_logprobs return token strings, UTF-8 bytes, and probabilities, never the vocabulary ID. You can rank and reconstruct text from them, you cannot recover the exact integer row in the embedding table without re-tokenizing yourself.
  • Streaming is text deltas only on both major providers, with no token ID exposed mid-stream.
  • vLLM's prompt_token_ids and skip_tokenizer_init prove the pattern works server-side, input and output, in production, just not behind a public hosted API.
  • The proposed interface reuses the same design language as the token-native storage post: a type-detected content block for token-ID input, a per-response format flag for token-ID output, mirroring that post's string-or-array insert path and its with_payload format-string read path.
  • A token-ID mode would have to be opt-in, versioned to an exact tokenizer version, and non-portable across model families, the same constraint the earlier posts in this series already flagged for storage and cross-tokenizer translation.
  • The real payoff is closing the loop for systems already storing tokens natively: retrieve token IDs, prompt with token IDs, receive token IDs, store token IDs, with text materializing only where a human actually needs to read it.
  • Providers may have good reasons not to expose this, mainly text-based safety filtering, token-space obfuscation risk, and unclear interactions with prompt caching, none of which this post can rule out.

Citation

If you find this useful, please cite:

@misc{kumar2026tokenllminterface,
  author       = {Kumar Shivendu},
  title        = {Token-Native LLM APIs: Input and Output as Token IDs},
  year         = {2026},
  url          = {https://www.kshivendu.dev/blog/token-llm-interface},
  note         = {Blog post}
}
Discuss on Twitter • View on GitHub
← Back to the blog