Module truncation 

Stage 3: history truncation.

Problem

Client-side context compaction can be broken for custom OpenAI-compatible endpoints. Each tool-call result is permanently embedded in the chat history by the client, so the prompt balloons past the model’s context window after several tool-heavy turns and the model falls into repetition or logic loops.

Defence

truncate_history is a stateless pass over the request body:

1. Budget gate — if the serialized payload already fits within limit_chars the body is left completely untouched. No history is elided while there is room, so the model keeps maximum context on every turn that does not actually need trimming.

2. Oldest-first trim — only when the payload exceeds the budget are messages elided, and then only as many as necessary: unprotected role: "tool" / role: "assistant" messages whose content string exceeds TOOL_CONTENT_THRESHOLD_CHARS are replaced with TRUNCATION_PLACEHOLDER from oldest to newest, stopping as soon as the running payload estimate drops back under budget. The freshest tool outputs — the ones the model most likely still needs — are the last to be sacrificed.

3. Hard abort — if the payload still exceeds the budget after every eligible message has been trimmed (an enormous protected system prompt, say), TruncationError is returned rather than forwarding a prompt that would fail at the model. Each surface maps that to its own idiom.

4. Protected set — role: "system" messages and the last PROTECTED_TAIL_COUNT messages by index (the immediate conversational context, spanning several recent tool-call/result pairs) are never modified. Neither is tool_calls, at any role.

The budget is the model’s, and only the model’s

limit_chars is a character budget, derived from the model’s context size in tokens via CHARS_PER_TOKEN_APPROX. There is no floor: a 4096-token model gets a ~16,000-character budget and a 262,144-token model gets a ~1,000,000-character one. Callers that know the live serving context and a better chars-per-token ratio (the proxy learns one per model from observed usage frames) pass their own number; ModelContext::context_budget_chars is the answer for everyone else.

Structs

CountingWriter 🔒

An std::io::Write sink that keeps the byte count and discards the bytes.

TruncationReport

Summary of what truncate_history did to a request body.

Enums

TruncationError

The request cannot be made to fit its context budget.

Constants

CHARS_PER_TOKEN_APPROX

Character-to-token conversion factor used to translate a model’s token context size into the character budget truncate_history measures.

PROTECTED_TAIL_COUNT

Number of trailing messages (by index) always preserved from truncation regardless of role or content size.

TOOL_CONTENT_THRESHOLD_CHARS

Maximum number of characters allowed in a single unprotected role: "tool" or role: "assistant" message content string before it is eligible for replacement with TRUNCATION_PLACEHOLDER.

TRUNCATION_PLACEHOLDER

Replacement string inserted in place of truncated message content.

Functions

is_tail_protected 🔒

Returns true if the message at index (in a list of total messages) falls within the protected tail window and must not be truncated.

is_truncation_candidate 🔒

Returns true if this message’s role is eligible for content truncation.

serialized_len 🔒

Byte length of body once serialized, without allocating a copy of it.

truncate_history

Trim stale history in place so the request fits within limit_chars.