Struct AgenticEvalConfig 

pub struct AgenticEvalConfig {
    pub model_id: i64,
    pub task_suite: TaskSuite,
    pub weights: Option<ScoreWeights>,
    pub ctx_size: Option<u64>,
    pub seeds: Vec<u32>,
    pub include_control: bool,
    pub replicate_raw: bool,
    pub replicate_pairs: usize,
    pub control_seeds: usize,
}

Configuration for one A/B agentic eval run.

Fields

model_id: i64

Database ID of the model to evaluate.

task_suite: TaskSuite

Task suite both arms run — the same schema the tune sweep uses.

weights: Option<ScoreWeights>

Weights for each arm’s composite score.

None means “the server decides”; see TuneConfig::weights, which also explains why skip_serializing_if is required rather than cosmetic.

ctx_size: Option<u64>

Context size override (tokens). None resolves through the harness’s own chain (model server defaults → global setting → hardcoded default), which deliberately stops short of the fitted rung a real launch reaches — so a benchmark taken with nothing configured is taken at the floor. Recorded in ADR 0009’s amendment; not the serving path’s chain.

seeds: Vec<u32>

RNG seeds to repeat every task under, once each.

Why more than one

A single sample per task is not a measurement of a model, it is one draw from its output distribution. Two runs of the identical raw configuration have scored 0.728 and 0.543 on this suite — a gap wider than most of the effects the eval exists to detect. Averaging a handful of seeds is what separates a real difference from that spread.

Seeded rather than merely repeated, so a surprising number can be re-run and reproduced instead of chased. An empty list means one unseeded run per task, which is the pre-multi-seed behaviour and is kept reachable deliberately — it is the fastest smoke test.

include_control: bool

Whether to run the positive control arm. See EvalArm::Control.

replicate_raw: bool

Whether to run the A/A arm. See EvalArm::RawReplicate.

On by default, and cheap: the raw arm is the fastest of the three, so repeating it costs a fraction of what the control does and is the only thing in the report that speaks to the size of an effect rather than its direction.

replicate_pairs: usize

How many A/A pairs to run. See EvalArm::RawReplicate.

1 is the historical single-pair behaviour and the default. A single pair estimates the eval’s drift from one degree of freedom — enough to stop a delta inside its own noise being called a finding, and not enough to say how noisy the eval actually is. Every additional pair re-runs the raw arm on another derived, disjoint seed set, and the drift estimate becomes the mean pairwise gap over all replicate runs plus the primary — which is the “more pairs” the EFFECT_NOISE_RATIO doc has always named as the honest strengthening.

control_seeds: usize

How many of Self::seeds the positive control repeats, from the front. Clamped into 1..=seeds.len().

Why this is not the full seed set

Because the control is the most expensive arm in the eval by an order of magnitude, and it does not need the precision. Measured on Qwen3.5-4B: broken sampling makes the model ramble, so the control took 161 of one run’s 174 wall-clock minutes and generated 5× the tokens of the two real arms combined.

It can afford to be imprecise because of what it is asked. The two real arms are being compared to each other and need every seed they can get; the control only has to clear CONTROL_MIN_COMPOSITE_GAP, and the gap it actually opens is an order of magnitude above that threshold. Paying five seeds to resolve a 0.5 gap more precisely buys nothing the report reads.

Trait Implementations

impl Clone for AgenticEvalConfig

fn clone(&self) -> AgenticEvalConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for AgenticEvalConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for AgenticEvalConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for AgenticEvalConfig

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.