Module kv_memory 

Detection of partial-KV-memory architectures from GGUF metadata.

Some model architectures do not retain the full token history in their KV memory: sliding-window attention (SWA) layers keep only a recent window, hybrid-attention models interleave SWA layers with full-attention layers, and recurrent (SSM/Mamba-family) models keep only a compressed state.

This matters for llama-server’s disk slot persistence (/slots?action=save|restore): the save path serializes only the sequence KV state and token list — not the server’s context checkpoints — and the restore path clears the slot’s checkpoint list. On a full-attention model that’s fine (the KV state alone is sufficient to resume). On a partial-memory model, resuming from position n_past requires history the SWA/recurrent layers no longer hold, which llama-server bridges with context checkpoints; with the checkpoint list empty after a disk restore, it falls back to n_past = 0 and reprocesses the entire prompt. A disk “restore” on such a model therefore costs a full re-prefill — worse than useless, since the in-RAM prompt cache (--cache-ram), which does carry checkpoints, would have resumed cheaply had the slot not been pre-filled by the restore.

Inputs come from the raw GGUF key/value map that gglib-gguf copies verbatim into crate::domain::Model::metadata (see crate::domain::estimate_kv_elems_per_token for the same pattern).

Detection is deliberately sensitive: a false positive merely forgoes the disk-cache layer (the in-RAM cache still works), while a false negative silently costs minutes of TTFT per restore. Some older GGUFs carry a sliding_window key the runtime ignores; treating them as partial is the safe direction.

Functions

kv_memory_is_partial

Whether the model’s KV memory retains only part of the token history (sliding-window, hybrid, or recurrent attention).

lookup_raw 🔒

Look up an architecture-prefixed GGUF key ({arch}.{suffix}), falling back to the bare suffix for the occasional file that omits the prefix.

lookup_u64 🔒

Numeric variant of lookup_raw.