Module kv_estimate 

KV-cache size estimation from GGUF metadata.

Estimates how many elements of KV cache a model consumes per token of context. Element counts are type-agnostic (derived purely from model architecture); converting to bytes happens separately, once the launch’s resolved K/V cache types are known (--cache-type-k/--cache-type-v may quantize K and V differently — see crate::cache_config::KvCacheType). Callers multiply the resulting bytes-per-token by a context size to size memory budgets (see crate::domain::cache_budget::compute_auto_cache_ram_mb).

Inputs come from the raw GGUF key/value map that gglib-gguf copies verbatim into crate::domain::Model::metadata, so no re-parse of the .gguf file is needed. Every key is architecture-prefixed (qwen3.block_count, llama.attention.head_count_kv, …).

This is deliberately an estimate: it models the standard transformer KV-cache layout and ignores architecture-specific extras (sliding-window layers, MLA compression, per-layer overrides). It is used only for conservative memory budgeting, never for correctness, and returns None rather than guessing when the metadata doesn’t carry what it needs.

Structs

KvElemsPerToken

Per-token K and V element counts, type-agnostic.

Functions

estimate_kv_bytes_for_context

Estimate total KV cache bytes for a given context size.

estimate_kv_elems_per_token

Estimate K and V element counts consumed per token of context.

kv_bytes_per_token

Convert per-token K/V element counts to bytes at the given cache types.

lookup 🔒

Look up an architecture-prefixed GGUF key ({arch}.{suffix}), falling back to the bare suffix for the occasional file that omits the prefix.