pub fn estimate_kv_elems_per_token<S: BuildHasher>(
metadata: &HashMap<String, String, S>,
architecture: Option<&str>,
) -> Option<KvElemsPerToken>Expand description
Estimate K and V element counts consumed per token of context.
Formula (standard transformer KV cache):
k_elems/token = block_count × head_count_kv × key_length
v_elems/token = block_count × head_count_kv × value_lengthkey_length/value_length are the per-head dimensions. When absent, both
fall back to embedding_length / head_count (the standard derivation).
head_count_kv falls back to head_count for models without grouped-query
attention.
§Arguments
metadata— raw GGUF key/value map (seecrate::domain::Model::metadata).architecture— the model’s architecture, used as the key prefix. WhenNone, falls back to thegeneral.architecturemetadata key.
§Returns
None when the metadata lacks the layer/head counts needed to compute a
meaningful figure (or carries non-numeric values) — callers should treat
that as “unknown” and substitute their own conservative allowance rather
than assuming zero.