Module truncation 

Stage 3: history truncation.

Problem

Client-side context compaction can be broken for custom OpenAI-compatible endpoints. Each tool-call result is permanently embedded in the chat history by the client, so the prompt balloons past the model’s context window after several tool-heavy turns and the model falls into repetition or logic loops.

Defence

truncate_history is a stateless pass over the request body:

1. Budget gate — if the serialized payload already fits within limit_chars the body is left completely untouched. No history is elided while there is room, so the model keeps maximum context on every turn that does not actually need trimming.

2. Oldest-first trim to a low watermark — only when the payload exceeds the budget are messages elided: unprotected role: "tool" / role: "assistant" messages whose text, in either content shape, exceeds TOOL_CONTENT_THRESHOLD_CHARS are replaced with TRUNCATION_PLACEHOLDER from oldest to newest, until the estimated savings reach a quantized target aimed at LOW_WATERMARK_PCT of the budget. The freshest tool outputs — the ones the model most likely still needs — are the last to be sacrificed.

3. Hard abort — if the payload still exceeds the budget after every eligible message has been trimmed (an enormous protected system prompt, say), TruncationError is returned rather than forwarding a prompt that would fail at the model. Each surface maps that to its own idiom.

4. Protected set — role: "system" messages and the last PROTECTED_TAIL_COUNT messages by index (the immediate conversational context, spanning several recent tool-call/result pairs) are never modified. Neither is tool_calls, at any role.

Why trim past the budget, and why the target is quantized

This stage is stateless and clients resend the full raw history every turn, so an elision never persists: each request re-derives the elision set from scratch. Trimming to just under the budget therefore moves the elision frontier forward by ~one message on every turn of a long agentic session — and every move shifts the first byte at which the forwarded prompt differs from the previous turn’s, breaking llama.cpp’s common-prefix KV-cache match and forcing a near-full prompt re-prefill each turn.

Aiming lower (a classic low watermark) is not enough by itself: a minimal elision set computed against any fixed threshold still grows at the same per-turn cadence. Instead the savings target is quantized to whole multiples of the watermark margin (budget − watermark, 25% of budget). Within one margin’s worth of payload growth the target — and therefore the elision set — is identical across turns, so consecutive requests share their prompt prefix and llama-server only prefills the new tail. When growth crosses a margin boundary, several messages are elided at once and the cycle restarts: the post-trim payload lands at or below the watermark, sawtoothing within (50%, 75%] of budget, and prefix breaks happen once per ~25%-of-budget of growth instead of once per turn.

The budget is the model’s, and only the model’s

limit_chars is a character budget, derived from the model’s context size in tokens via CHARS_PER_TOKEN_APPROX. There is no floor: a 4096-token model gets a ~16,000-character budget and a 262,144-token model gets a ~1,000,000-character one. Callers that know the live serving context and a better chars-per-token ratio (the proxy learns one per model from observed usage frames) pass their own number; ModelContext::context_budget_chars is the answer for everyone else.

Structs

CountingWriter 🔒

An std::io::Write sink that keeps the byte count and discards the bytes.

TruncationReport

Summary of what truncate_history did to a request body.

Enums

TruncationError

The request cannot be made to fit its context budget.

Constants

CHARS_PER_TOKEN_APPROX

Character-to-token conversion factor used to translate a model’s token context size into the character budget truncate_history measures.

LOW_WATERMARK_PCT 🔒

Low watermark the trim aims for, as a percentage of the request budget.

PROTECTED_TAIL_COUNT 🔒

Number of trailing messages (by index) always preserved from truncation regardless of role or content size.

TOOL_CONTENT_THRESHOLD_CHARS 🔒

Maximum number of characters of text allowed in a single unprotected role: "tool" or role: "assistant" message, in either content shape, before it is eligible for replacement with TRUNCATION_PLACEHOLDER.

TRUNCATION_PLACEHOLDER 🔒

Replacement string inserted in place of truncated message content.

Functions

is_tail_protected 🔒

Returns true if the message at index (in a list of total messages) falls within the protected tail window and must not be truncated.

is_truncation_candidate 🔒

Returns true if this message’s role is eligible for content truncation.

serialized_len 🔒

Byte length of body once serialized, without allocating a copy of it.

target_savings_chars 🔒

The number of estimated characters truncation must reclaim from a payload of payload_chars under a budget of limit_chars.

truncate_history

Trim stale history in place so the request fits within limit_chars, aiming past the budget for the LOW_WATERMARK_PCT watermark once triggered.