Skip to main content

resolve_sampling

Function resolve_sampling 

Source
pub fn resolve_sampling(
    body: &mut Value,
    ctx: &ModelContext,
    layers: &SamplingLayers,
)
Expand description

Resolve the sampling hierarchy into body, then pin cache_prompt.

§Force-insert, not or_insert

The client’s own parameters are extracted from body first, folded through InferenceConfig::resolve_layers alongside cli / profile / model / global, and the fully-resolved result is then written back over the top. Client parameters still win — they win by being the highest-priority layer in the fold, not by surviving an or_insert. Rewriting this as or_insert looks equivalent and silently breaks the hierarchy: every layer below the client would stop applying to any key the client happened to send.

§Client trust

layers.trust_client_sampling gates which of the client’s own fields enter that layer at all. When false (the default — see Settings::trust_client_sampling), only max_tokens survives; the rest of body’s sampling keys are read but discarded before the fold, so a client with a hardcoded temperature can no longer outrank this server’s own configuration, and every field it left unset still gap-fills from below exactly as if it had never sent that key.

A body that is not a JSON object is left alone.