pub struct ArmScores {Show 16 fields
pub tool_accuracy: f64,
pub loop_avoidance: Option<f64>,
pub loop_eligible: usize,
pub task_completion: f64,
pub composite: f64,
pub tg_tps: Option<f64>,
pub total_completion_tokens: Option<u64>,
pub total_wall_ms: u64,
pub measured_wall_ms: u64,
pub mean_time_to_first_tool_call_ms: Option<f64>,
pub median_time_to_first_tool_call_ms: Option<f64>,
pub seeds: usize,
pub runs: usize,
pub unmeasured_runs: usize,
pub transport_retries: u32,
pub generated: GeneratedOutput,
}Expand description
One arm’s aggregate scores across the task suite.
Fields§
§tool_accuracy: f64Mean AST-style tool-call match score, 0.0–1.0.
loop_avoidance: Option<f64>Fraction of loop-eligible tasks that triggered neither the loop nor the stagnation guard.
None when no task in this arm ever reached a second tool-call batch,
so the guards had nothing to fire on: the axis was not measured, which
is distinct from a perfect 1.0. Read it together with
Self::loop_eligible, which is its denominator.
loop_eligible: usizeHow many of this arm’s tasks were loop-eligible — the sample size
behind Self::loop_avoidance.
task_completion: f64Fraction of tasks passed outright.
composite: f64Weighted composite of the axes above, over whichever of them were measured. An unmeasured loop-avoidance axis claims no weight rather than scoring zero.
tg_tps: Option<f64>Completion-token throughput (tokens per wall-clock second, pre-fill
included). None when the upstream reported no usage.
total_completion_tokens: Option<u64>Total completion tokens generated across the whole suite. None when
no task reported usage.
Reported beside the composite and never folded into it: token cost is what the quality axes cannot see, but it is also hardware- and model-specific in a way that would make a single blended score incomparable across machines.
total_wall_ms: u64Total wall-clock milliseconds across every task in the suite, unfiltered — the honest cost of running it.
Report this as “what the suite cost”; never compare two arms with it.
A run that stalled and timed out contributes its whole timeout here, so
this figure answers “how long did I wait” rather than “how fast is the
pipeline”. Self::measured_wall_ms answers the second.
measured_wall_ms: u64Wall-clock milliseconds across the runs that reached the model.
The comparable figure, and the one every ratio is taken from. It shares
its population with Self::tg_tps and
Self::mean_time_to_first_tool_call_ms, which already filtered this
way — the efficiency table used to print those beside an unfiltered
wall time, so two of its rows described different sets of runs while
looking like one table.
mean_time_to_first_tool_call_ms: Option<f64>Mean time to the first tool call, over the tasks that made one. None
when no task in the arm called a tool.
Read this beside Self::median_time_to_first_tool_call_ms, never
alone. The population is not unimodal. On 2026-08-29 one arm reached
its first call in about a second on most tasks and after roughly 950
seconds on five of them; the mean of that is ~94s, which describes
neither group and no individual run. The mean is kept because a large
gap between it and the median is itself the finding.
median_time_to_first_tool_call_ms: Option<f64>Median time to the first tool call, over the tasks that made one.
The typical run, which the mean stops describing the moment a handful of runs generate for a quarter of an hour. Reported alongside rather than instead of the mean: the median alone would have hidden those five runs as effectively as the mean misrepresented them, and the pair is what makes the spread visible.
seeds: usizeHow many seeds every task was repeated under.
The sample size behind every mean above, and the thing that makes them comparable across runs. A composite from one seed and a composite from five are not the same measurement, and a report that renders them identically invites exactly the mistake this eval exists to prevent.
1 on a legacy row, which is what it was.
runs: usizeTotal task runs behind these scores — tasks × seeds.
unmeasured_runs: usizeHow many of those runs never reached the model, and therefore contributed a zero that measures nothing.
See TuneTaskResult::unmeasured. An arm where this equals
Self::runs is not a low score — it is an empty column, and the eval
refuses to report one rather than rendering it as an arm that did
badly. Anything between 1 and runs contaminates every mean above by
an amount this number is the only record of.
transport_retries: u32How many attempts this arm threw away to transport failures and retried.
Distinct from Self::unmeasured_runs in both direction and meaning: a
retry that worked leaves this non-zero and unmeasured_runs at zero, so
an arm can be fully measured and still have been fighting the upstream
the whole way. A report that showed only the survivors would call that
arm clean.
Summed over runs, so one run retried twice and two runs retried once
both read 2. See TuneTaskResult::transport_retries.
generated: GeneratedOutputWhat this arm generated, summed over its measured runs.
Aggregation differs per field, and deliberately: character counts,
llm_calls and system_warnings are sums over the arm, while
GeneratedOutput::max_tool_calls_in_batch is the arm-wide maximum
— a single runaway batch is the thing worth seeing, and a mean would
dissolve it into 63 ordinary runs.
Restricted to measured runs for the reason Self::measured_wall_ms
gives: a run that never reached the model generated nothing, and folding
its zeros in would understate the arm’s output exactly where the arm was
least healthy.
Implementations§
Source§impl ArmScores
impl ArmScores
Sourcepub const fn measured_runs(&self) -> usize
pub const fn measured_runs(&self) -> usize
Runs that reached the model — the denominator every comparable figure on this arm is taken over.
Saturating, because a legacy row can carry an unmeasured_runs it never
had a runs for, and a wrapped denominator would be far worse than a
zero one.
Sourcepub const fn is_empty_column(&self) -> bool
pub const fn is_empty_column(&self) -> bool
Whether no run in this arm reached the model.
The state that must never render as a score. An arm in it has a composite, a tool accuracy and a task completion, all arithmetically correct and all meaningless — computed over runs that produced no response to score.
Sourcepub const fn is_partly_unmeasured(&self) -> bool
pub const fn is_partly_unmeasured(&self) -> bool
Whether some but not all of this arm’s runs reached the model.
Distinct from Self::is_empty_column because it wants a different
action: the arm has real observations mixed with empty ones, so its
means are contaminated by a knowable amount rather than vacant.