Skip to main content

resolve_sampling

Function resolve_sampling 

Source
pub fn resolve_sampling(
    body: &mut Value,
    ctx: &ModelContext,
    layers: &SamplingLayers,
) -> SamplingDecision
Expand description

Resolve the sampling hierarchy into body, then pin cache_prompt.

This doc block used to sit above read_client_layer, where a split left it fused to that function’s own first line — so the entry point of the whole stage was undocumented while a private helper carried a description of something else. Restored here; read_client_layer keeps its own.

§Force-insert, not or_insert

The client’s own parameters are extracted from body first, folded through InferenceConfig::resolve_layers_with_sources alongside cli / profile / model / global, and the fully-resolved result is then written back over the top. Client parameters still win — they win by being the highest-priority layer in the fold, not by surviving an or_insert. Rewriting this as or_insert looks equivalent and silently breaks the hierarchy: every layer below the client would stop applying to any key the client happened to send.

§Client trust

layers.trust_client_sampling gates which of the client’s own fields enter that layer at all. When false (the default — see Settings::trust_client_sampling), only CLIENT_AUTHORITATIVE_KEYS survive; the rest of body’s sampling keys are read but discarded before the fold, so a client with a hardcoded temperature can no longer outrank this server’s own configuration, and every field it left unset still gap-fills from below exactly as if it had never sent that key.

That carve-out is a category, not a list of exceptions, and the two reasoning controls land on opposite sides of it: the budget is a budget and survives, the effort level is taste and does not. The reasoning is on CLIENT_AUTHORITATIVE_KEYS.

The gate covers modelled fields; sampler keys the ladder has no field for (UNMODELLED_SAMPLER_KEYS) are stripped from the untrusted body itself, because a key with no layer has nothing to be discarded from and would otherwise ride the body to llama-server ungoverned.

§What leaves the body

erase_unadopted_client_keys is the one place keys are deleted, and it deletes three things — only the first of which is about trust:

  • What the gate binned — empty when the client is trusted.
  • thinking_budget_tokens — always. It is upstream’s alias for the budget, gglib reads it and then emits the canonical key alone, and two spellings of one parameter in one body is a disagreement waiting to be resolved by somebody else’s parse order.
  • A refused reasoning_effort — on both sides of the gate, and it is the only field an issues entry removes. Every other refused value is forwarded exactly as before this PR, because upstream 400s on it and that 400 is a better answer to the client than gglib quietly rewriting the request. reasoning_effort is the exception because upstream validates it not at all: a refused "banana" left in the body is not rejected downstream, it is rendered into the prompt.

So a client’s top_k: "5" still reaches llama-server and still earns its HTTP 400, unchanged by the reasoning work. The helper carries the full argument and ADR 0007’s finding behind it.

A body that is not a JSON object is left alone.