pub fn fit_context(
trained_ctx: Option<u64>,
weights_bytes: Option<u64>,
kv: Option<KvElemsPerToken>,
k: KvCacheType,
v: KvCacheType,
budget_bytes: Option<u64>,
) -> Option<u64>Expand description
The largest context weights_bytes can serve inside budget_bytes.
None whenever the answer cannot be computed from facts — an unknown
trained context, unknown KV shape, or no memory reading. That is a refusal
rather than an optimistic guess, matching SlotFootprint::new: a caller
that gets None falls back down its own chain instead of launching against
a number nobody stands behind.
Also None when the machine cannot fit even the smallest rung. Returning
something smaller would be inventing a context this module has no basis
for; the built-in default is the right thing to fall back to, and it will
fail honestly if it does not fit either.
Pure. GGLIB_DISABLE_CONTEXT_FIT is read by the caller, not here — the
switch belongs with the other runtime switches at the admission site, and a
domain function that read the environment could not be tested without
mutating process-global state.
budget_bytes must be a figure that does not move between requests: this
value ends up in a resident’s identity, so a budget that drifts evicts and
relaunches the model it just sized.
A live free-memory reading is therefore wrong — on Apple it is a fraction of available system RAM, and it moves with whatever else is open. So is netting out the current resident set, which was tried and removed: it moved whenever a co-resident loaded or was evicted, and made one model’s budget depend on whether another model’s KV shape was readable. What the caller supplies is total device capacity less a fixed reservation for the second resident slot.