Skip to main content

estimate_kv_bytes_for_context

Function estimate_kv_bytes_for_context 

Source
pub const fn estimate_kv_bytes_for_context(
    kv_bytes_per_token: u64,
    context_size: u64,
) -> u64
Expand description

Estimate total KV cache bytes for a given context size.

Convenience wrapper taking an already-computed bytes-per-token figure (see kv_bytes_per_token); saturating so an absurd context size can never overflow into a small (and therefore dangerously permissive) budget.