Skip to main content

fit_context

Function fit_context 

Source
pub fn fit_context(
    trained_ctx: Option<u64>,
    weights_bytes: Option<u64>,
    kv: Option<KvElemsPerToken>,
    k: KvCacheType,
    v: KvCacheType,
    budget_bytes: Option<u64>,
) -> Option<u64>
Expand description

The largest context weights_bytes can serve inside budget_bytes.

None whenever the answer cannot be computed from facts — an unknown trained context, unknown KV shape, or no memory reading. That is a refusal rather than an optimistic guess, matching SlotFootprint::new: a caller that gets None falls back down its own chain instead of launching against a number nobody stands behind.

Also None when the machine cannot fit even the smallest rung. Returning something smaller would be inventing a context this module has no basis for; the built-in default is the right thing to fall back to, and it will fail honestly if it does not fit either.

Pure. GGLIB_DISABLE_CONTEXT_FIT is read by the caller, not here — the switch belongs with the other runtime switches at the admission site, and a domain function that read the environment could not be tested without mutating process-global state.

budget_bytes must be a figure that does not move between requests: this value ends up in a resident’s identity, so a budget that drifts evicts and relaunches the model it just sized.

A live free-memory reading is therefore wrong — on Apple it is a fraction of available system RAM, and it moves with whatever else is open. So is netting out the current resident set, which was tried and removed: it moved whenever a co-resident loaded or was evicted, and made one model’s budget depend on whether another model’s KV shape was readable. What the caller supplies is total device capacity less a fixed reservation for the second resident slot.