Skip to main content

estimate_kv_elems_per_token

Function estimate_kv_elems_per_token 

Source
pub fn estimate_kv_elems_per_token<S: BuildHasher>(
    metadata: &HashMap<String, String, S>,
    architecture: Option<&str>,
) -> Option<KvElemsPerToken>
Expand description

Estimate K and V element counts consumed per token of context.

Formula (standard transformer KV cache):

k_elems/token = block_count × head_count_kv × key_length
v_elems/token = block_count × head_count_kv × value_length

key_length/value_length are the per-head dimensions. When absent, both fall back to embedding_length / head_count (the standard derivation). head_count_kv falls back to head_count for models without grouped-query attention.

§Arguments

  • metadata — raw GGUF key/value map (see crate::domain::Model::metadata).
  • architecture — the model’s architecture, used as the key prefix. When None, falls back to the general.architecture metadata key.

§Returns

None when the metadata lacks the layer/head counts needed to compute a meaningful figure (or carries non-numeric values) — callers should treat that as “unknown” and substitute their own conservative allowance rather than assuming zero.