Expand description
Prompt-cache reuse telemetry.
Records how many prompt tokens each completed request reused from
llama-server’s KV cache instead of re-processing, sourced from
usage.prompt_tokens_details.cached_tokens (see
crate::LlmStreamEvent::Usage).
The store is location-neutral — a lock-free counter with a
serde::Serialize snapshot and no dependency on any particular request
path — so more than one caller can keep its own instance. The proxy keeps
one for the traffic it forwards; a separate instance covers the in-process
agent path (council and GUI chat). The two populations are reported side by
side, never merged: a council run’s many small sub-agent calls have a reuse
profile nothing like a user’s conversation, so averaging them would make
either figure harder to read.
Deliberately raw counters. Everything exposed is something the upstream actually measured; nothing is derived, extrapolated, or turned into a “time saved” figure. Reuse counts are exact, but what that reuse saved depends on a counterfactual prefill that never ran — presenting an estimate of it as a dashboard number would invite trust it can’t earn. Consumers that want a ratio can divide two figures that are both real.
Requests whose upstream didn’t report the field are counted separately
(unreported_requests) rather than folded in as zero-reuse, so a server
that never reports can’t masquerade as a cache that never hits.
Structs§
- Cache
Metrics Store - Running totals of prompt-cache reuse.
- Cache
Usage - A point-in-time view of prompt-cache reuse since the proxy started.