Struct InferenceConfig 

pub struct InferenceConfig {
    pub temperature: Option<f32>,
    pub top_p: Option<f32>,
    pub top_k: Option<i32>,
    pub max_tokens: Option<u32>,
    pub repeat_penalty: Option<f32>,
    pub presence_penalty: Option<f32>,
    pub min_p: Option<f32>,
}

Inference parameters for LLM sampling.

All fields are optional to support partial configuration and fallback chains. Intended to be shared across model defaults, global settings, and request overrides.

Hierarchy Resolution

When making an inference request, parameters are resolved in this order:

1. Request-level override (user specified for this request)
2. Selected profile (Settings.inference_profiles, chosen as {model}:{profile}; absent on surfaces without profiles)
3. Per-model defaults (stored in Model.inference_defaults)
4. Global settings (stored in Settings.inference_defaults)
5. Hardcoded fallback (e.g., temperature = 0.7)

Examples

use gglib_core::domain::InferenceConfig;

// Conservative settings for code generation
let code_gen = InferenceConfig {
    temperature: Some(0.2),
    top_p: Some(0.9),
    top_k: Some(40),
    max_tokens: Some(2048),
    repeat_penalty: Some(1.1),
    presence_penalty: None,
    min_p: None,
};

// Creative writing settings
let creative = InferenceConfig {
    temperature: Some(1.2),
    top_p: Some(0.95),
    ..Default::default()
};

Fields

temperature: Option<f32>

Sampling temperature (0.0 - 2.0).

Controls randomness in token selection:

• Lower values (0.1-0.5): More deterministic, focused
• Medium values (0.7-1.0): Balanced creativity
• Higher values (1.1-2.0): More random, creative

top_p: Option<f32>

Nucleus sampling threshold (0.0 - 1.0).

Considers only the top tokens whose cumulative probability exceeds this threshold. Common values: 0.9 (default), 0.95 (more diverse)

top_k: Option<i32>

Top-K sampling limit.

Considers only the K most likely next tokens. Common values: 40 (default), 10 (focused), 100 (diverse)

max_tokens: Option<u32>

Maximum tokens to generate in response.

Hard limit on response length. Does not include input tokens.

repeat_penalty: Option<f32>

Repetition penalty (> 0.0, typically 1.0 - 1.3).

Penalizes repeated tokens to reduce repetitive output.

• 1.0: No penalty (default)
• 1.1-1.3: Moderate penalty

• 1.3: Strong penalty (may hurt coherence)

presence_penalty: Option<f32>

Presence penalty (0.0 - 2.0).

Penalizes tokens that have already appeared in the output, encouraging the model to cover new ground. Effective at preventing repetitive reasoning loops in thinking models.

• 0.0: No penalty (default; disabled)
• 1.5: Recommended for reasoning/thinking models (e.g. Qwen3.6, DeepSeek-R1)

• 2.0: Avoid; may degrade coherence

min_p: Option<f32>

Minimum-probability sampling threshold (0.0 - 1.0).

Removes tokens whose probability is below min_p × P(top token).

• 0.0: Disabled (explicit off; recommended by Qwen3.6)
• 0.05: llama.cpp built-in default when the flag is omitted

Implementations

impl InferenceConfig

pub const fn merge_with(&mut self, other: &Self)

Merge another config into this one, preferring values from other.

For each field, if other has Some(value), use it; otherwise keep self’s value. This is useful for applying fallback chains.

Example

use gglib_core::domain::InferenceConfig;

let mut request = InferenceConfig {
    temperature: Some(0.8),
    ..Default::default()
};

let model_defaults = InferenceConfig {
    temperature: Some(0.5),
    top_p: Some(0.9),
    ..Default::default()
};

request.merge_with(&model_defaults);
assert_eq!(request.temperature, Some(0.8)); // Request value wins
assert_eq!(request.top_p, Some(0.9));      // Fallback to model default

pub fn resolve_layers(layers: &[Option<&Self>], floor: &Self) -> Self

Resolve an ordered list of sampling layers (highest priority first) into a single fully-resolved config, then fill anything still unset from floor.

This is the one fold every multi-layer resolution surface goes through: resolve_with_profile wraps it for the simple request/profile/model/global shape, and crate::request_pipeline::sampling builds its own five-layer (cli/client/profile/model/global) array and calls it directly. There is exactly one place that decides what “wins” means.

Uncoupled parameters

top_p, top_k, and max_tokens gap-fill independently: each takes the first Some value found scanning the layers top to bottom.

Coupled parameters

presence_penalty, repeat_penalty and min_p are only meaningful relative to how sharp the sampling distribution is, so they travel with the temperature they were chosen for. reasoning_profile pairs temperature 1.0 with presence_penalty 1.5 deliberately; a sparse profile that sets temperature 0.2 and leaves the penalty unset must not inherit that 1.5 — that would run a recipe no layer ever intended, a penalty tuned for a broad distribution applied to a near-greedy one.

So: temperature resolves to the first layer that sets one. If some layer does, the coupled trio comes only from that same layer — never a layer beneath it — falling to floor for anything that layer itself left unset. If no layer sets a temperature at all, nothing has been tuned against anything, so the coupled trio gap-fills normally, exactly like the uncoupled parameters.

pub const fn with_hardcoded_defaults() -> Self

Create a new config with all fields set to sensible defaults.

These are the hardcoded fallback values used when no other defaults are configured.

max_tokens has no fallback

It is deliberately None. Resolution force-writes every Some field into the outgoing request, so a value here would cap every request that did not name its own — silently truncating long answers. Left unset, no max_tokens key is emitted and llama-server applies its own n_predict default of -1, generating until a stop token or the context limit.

Omitting the key is exactly equivalent to sending -1 (llama.cpp’s has_budget() treats -1 as limitless) and is the better of the two: max_tokens: -1 is invalid under the OpenAI schema, which requires a positive integer, so a strict client or intermediary proxy may reject it. Omission keeps the forwarded body OpenAI-legal.

Explicit per-request, per-profile, and per-model values are unaffected — reasoning_profile still sets its own ceiling.

pub const fn reasoning_floor() -> Self

The coupled-trio floor for models tagged reasoning.

resolve_layers falls back to a floor once it has decided which layer (if any) claims the coupled trio and that layer left a field unset. with_hardcoded_defaults’s neutral presence_penalty: 0.0 is the right floor for most models, but wrong for a reasoning-tagged one: those degrade under greedy or near-greedy decoding into repetitive reasoning loops (see reasoning_profile, which pairs presence_penalty: 1.5 with temperature: 1.0 specifically to prevent this). 1.0 keeps a real guard in place at the floor without asserting the full recipe tuned for a different temperature.

pub fn to_cli_args(&self) -> Vec<String>

Convert inference config to llama CLI arguments.

Returns a vector of argument strings suitable for passing to llama-server. Uses the same flag names as llama.cpp: --temp, --top-p, --top-k, -n, --repeat-penalty.

This is the single source of truth for CLI flag conversion, reached by every launch surface through build_server_config and ServerConfig.extra_args.

Example

use gglib_core::domain::InferenceConfig;

let config = InferenceConfig {
    temperature: Some(0.8),
    top_p: Some(0.9),
    top_k: None,
    max_tokens: Some(1024),
    repeat_penalty: None,
    presence_penalty: None,
    min_p: None,
};

let args = config.to_cli_args();
assert_eq!(args, vec!["--temp", "0.8", "--top-p", "0.9", "-n", "1024"]);

pub const fn reasoning_profile() -> Self

Return a recommended InferenceConfig profile for reasoning / thinking models.

Applied automatically at import time when the "reasoning" capability tag is detected (e.g. Qwen3.6, DeepSeek-R1, QwQ). Values follow the Qwen3.6 upstream guidance for thinking mode — general tasks and are conservative enough to work well across all thinking-capable models.

Parameter	Value	Rationale
temperature	1.0	Recommended thinking-mode baseline
top_p	0.95	Broad nucleus; standard for reasoning
top_k	20	Tighter than the 40 fallback; suppresses low-quality tokens
max_tokens	8192	Safe out-of-the-box ceiling; increase for complex tasks
repeat_penalty	1.0	No penalty; presence_penalty handles anti-repetition
presence_penalty	1.5	Prevents repetitive reasoning loops
min_p	0.0	Explicitly disabled per Qwen3.6 spec

Users can override any parameter with gglib model update <id> --<flag> or the equivalent UI control.

pub fn resolve_with_defaults( self, model: Option<&Self>, global: Option<&Self>, model_ctx: ModelSamplingContext, ) -> Self

Resolve inference parameters using the 4-level hierarchy.

Equivalent to resolve_with_profile with no profile selected — see there for the merge order. This is the entry point for surfaces that have no notion of a named profile (gglib serve, gglib chat, gglib q, the Web UI chat API).

model_ctx carries the two facts about the target model that change how resolution behaves — see ModelSamplingContext, resolve_layers and reasoning_floor.

Example

use gglib_core::domain::{InferenceConfig, ModelSamplingContext};

let request = InferenceConfig { temperature: Some(0.9), ..Default::default() };
let model   = InferenceConfig { temperature: Some(0.5), top_p: Some(0.8), ..Default::default() };

let resolved = request.resolve_with_defaults(Some(&model), None, ModelSamplingContext::default());
assert_eq!(resolved.temperature, Some(0.9)); // request wins
assert_eq!(resolved.top_p,       Some(0.8)); // model fills in
assert_eq!(resolved.top_k,       Some(40));  // hardcoded fallback

pub fn resolve_with_profile( self, profile: Option<&Self>, model: Option<&Self>, global: Option<&Self>, model_ctx: ModelSamplingContext, ) -> Self

Resolve inference parameters using the full 5-level hierarchy.

Applies fallback layers in order, with each layer filling only None fields from self — explicit values are never overwritten:

1. self — caller-supplied overrides (request params, CLI flags, etc.)
2. profile — the named profile the request selected, if any
3. model — per-model stored defaults, if user-set
4. global — global settings defaults
5. model again, if auto-detected — see below
6. the model-class floor — reasoning_floor when model_ctx.is_reasoning, otherwise with_hardcoded_defaults

This is the single source of truth for inference parameter resolution across every gglib surface that does not need its own layer set; resolve_with_defaults delegates here so there is exactly one merge order to reason about and to test. crate::request_pipeline::sampling needs a seventh layer (the client’s own request, sitting between self and profile) and calls the underlying resolve_layers directly for that reason — the merge semantics are identical either way.

Why the profile sits above the model

Selecting model:coding is an explicit act by the caller, so it has to beat the model’s stored defaults or it would appear to do nothing on any model that has them. Because profiles are sparse (see crate::domain::inference_profile), outranking the model layer costs nothing for parameters the profile does not set — those still resolve from the model, which is what keeps one global profile safe to apply across differing architectures.

Why model can rank below global

model is only ever a stand-in for Model.inference_defaults, which gets written two different ways (see DefaultsOrigin): a person tuning it deliberately, or gglib’s own import-time guess for any model tagged reasoning. Those deserve different authority. A deliberate per-model choice should keep outranking the operator’s global defaults — that is what “per-model” means. A guess nobody reviewed should not: it silently shadowed the user’s own configured global settings, which is how #685 happened. model_ctx.defaults_origin decides which rung model occupies for this call — never both at once, since only one of rungs 3 and 5 is ever populated for a given model.

Temperature-tuned parameters do not fall through

See resolve_layers for the full rule. In short: once a layer declares a temperature, lower layers may not contribute presence_penalty, repeat_penalty or min_p — those resolve from the claiming layer alone, falling to the class floor if it left them unset.

Example

use gglib_core::domain::{InferenceConfig, ModelSamplingContext};

// A sparse profile: sets temperature, says nothing about anything else.
let profile = InferenceConfig { temperature: Some(0.2), ..Default::default() };
// A thinking model's stored defaults: 1.5 is tuned for temperature 1.0.
let model = InferenceConfig {
    temperature: Some(1.0),
    presence_penalty: Some(1.5),
    top_k: Some(20),
    ..Default::default()
};
let model_ctx = ModelSamplingContext { is_reasoning: true, ..Default::default() };

let resolved = InferenceConfig::default()
    .resolve_with_profile(Some(&profile), Some(&model), None, model_ctx);

assert_eq!(resolved.temperature,      Some(0.2)); // profile beats model
assert_eq!(resolved.presence_penalty, Some(1.0)); // reasoning floor, NOT the model's 1.5
assert_eq!(resolved.top_k,            Some(20));  // untuned: still fills

pub fn from_openai_json(value: &Value) -> Self

Parse inference parameters from an OpenAI-format JSON body (snake_case keys).

Converts wire-format snake_case field names (top_p, max_tokens, repeat_penalty, etc.) to the internal camelCase representation via snake_to_camel, then deserialises using the existing serde impl. Unknown or missing fields default to None.

This is the inverse of to_openai_json_patch.

pub fn to_openai_json_patch(&self) -> Map<String, Value>

Serialise as an OpenAI-format JSON patch (snake_case keys, Some fields only).

Uses serde to produce the camelCase form, then renames each key to snake_case via camel_to_snake. Only Some fields are emitted — None values are filtered out. The returned map can be merged directly into an OpenAI-compatible request body with body_obj.insert(k, v).

This is the inverse of from_openai_json.

Trait Implementations

impl Clone for InferenceConfig

fn clone(&self) -> InferenceConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for InferenceConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for InferenceConfig

fn default() -> InferenceConfig

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for InferenceConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl PartialEq for InferenceConfig

fn eq(&self, other: &InferenceConfig) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for InferenceConfig

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl StructuralPartialEq for InferenceConfig