Skip to main content

CHARS_PER_TOKEN_APPROX

Constant CHARS_PER_TOKEN_APPROX 

Source
pub const CHARS_PER_TOKEN_APPROX: usize = 4;
Expand description

Character-to-token conversion factor used to translate a model’s token context size into the character budget truncate_history measures.

This is not an attempt at precise real-world tokenization — it deliberately matches the GitHub Copilot LLM Gateway extension’s own TOKEN_CONSTANTS.CHARS_PER_TOKEN = 4 (see its tokenBudget.ts), so that gglib’s advertised context window and the extension’s own char-to-token budget estimate agree on the same conversion factor. What matters is consistency between the two sides, not tokenizer accuracy.