Module agentic 

Raw-vs-gglib A/B agentic evaluation: config and report types.

The eval answers one question with numbers: what does routing a small model through the gglib pipeline actually buy in agentic behaviour? It runs the same task suite the tune sweep uses — real AgentLoop, scripted BFCL-style tasks — twice against the same loaded model:

• raw: the request pipeline bypassed entirely. No sampling resolution (the server’s own defaults apply), no capability shaping, no dialect normalization, no grammar — what a client pointed straight at llama-server experiences.
• gglib: the full pipeline, exactly as the proxy runs it — per-model sampling defaults, capability-aware shaping, dialect parsing, and decode-time grammar enforcement where a task demands a tool call.

The per-axis deltas in the AgenticEvalReport are the product: the measured difference in tool-call accuracy, loop avoidance, and task completion, on this model, on this machine.

Two further arms exist to keep those deltas honest, and neither is a measurement of the pipeline:

• raw_replicate (EvalArm::RawReplicate) runs the raw arm a second time on a disjoint seed set. Nothing differs between it and the raw arm except which seeds were drawn, so whatever gap it opens is the eval’s own drift — the floor a raw-versus-gglib delta has to clear before it means anything. An A/A test.
• control (EvalArm::Control) runs the gglib pipeline with sampling deliberately broken, and must score far below it. It answers the opposite question: not is this difference real but could this apparatus have seen a difference at all.

They answer different failures and neither substitutes for the other. A control that moves 0.5 says the eval can detect a large change; it says nothing about whether it can resolve a 0.08 one, which is what the A/A arm is for.

Structs

AgenticEvalConfig

Configuration for one A/B agentic eval run.

AgenticEvalReport

The complete A/B report — the leaderboard interchange format.

AgenticTaskComparison

One task’s outcome under both arms, for drill-down.

ArmDelta

Per-axis difference, gglib − raw. Positive means gglib scored higher.

ArmScores

One arm’s aggregate scores across the task suite.

PairedEffect

The paired view of the raw-versus-gglib comparison.

Enums

ControlVerdict

What the positive control demonstrated about this run’s sensitivity.

DeltaWithheld

Why an arm-level delta is not reported.

EffectVerdict

What the A/A arm says about the size of the measured effect.

EvalArm

Which arm a task ran under.

Constants

CONTROL_MIN_COMPOSITE_GAP

The smallest composite gap the control arm must open for the apparatus to have demonstrably moved.

CONTROL_MIN_P

min_p the control arm forces. 0.0 disables the tail cut.

CONTROL_TEMPERATURE

The temperature the control arm forces.

CONTROL_TOP_K

top_k the control arm forces. 0 disables the cut entirely.

CONTROL_TOP_P

top_p the control arm forces. 1.0 keeps the whole nucleus.

DEFAULT_SEEDS

The seeds an eval uses when its config names none.

EFFECT_NOISE_RATIO

How many times the raw-versus-gglib effect must exceed the A/A drift before the report will call it more than noise.

REPLICATE_SEED_OFFSET

Offset added to each primary seed to derive the A/A arm’s seeds.

WILCOXON_MIN_PAIRS

The fewest non-tied pairs the normal-approximation Wilcoxon accepts.

Functions

as_f64 🔒

Widen a count for ratio arithmetic. Suite totals are far below the 2^53 boundary where f64 stops representing integers exactly.

control_sampling

The sampling the control arm applies, on top of a request’s seed.

default_control_seeds 🔒

default_include_control 🔒

default_replicate_pairs 🔒

default_replicate_raw 🔒

default_seeds 🔒

normal_cdf 🔒

Standard normal CDF via Abramowitz–Stegun 7.1.26 on erf, accurate to ~1.5e-7 — orders of magnitude finer than any decision read from a p.

one 🔒

per_run 🔒

A total spread over the runs that produced it. None when there were none, which keeps “no measured runs” distinct from “a measured zero”.

ratio 🔒

raw ÷ gglib, or None when either side is unmeasured or the denominator is zero — an infinite speedup is not a measurement.

replicate_seed_set

The seed set for A/A pair pair (1-based): the primary seeds offset by pair strides of REPLICATE_SEED_OFFSET.

replicate_seeds

The seed set the A/A arm runs, derived from the primary one.

wilcoxon_one_sided 🔒

One-sided Wilcoxon signed-rank p for “the deltas are positive”.