Struct GeneratedOutput 

pub struct GeneratedOutput {
    pub reasoning_chars: u64,
    pub answer_chars: u64,
    pub llm_calls: usize,
    pub max_tool_calls_in_batch: usize,
    pub system_warnings: u32,
}

The shape of what a run generated, as opposed to how much.

Why this exists

Until this struct, the eval counted output and threw it away: 7 of the 9 AgentEvent variants — TextDelta and ReasoningDelta among them — fell through the benchmark’s event loop untouched. A run was therefore knowable only as a token total and a wall time.

That is not enough to read a run. On 2026-08-29 five runs generated ~32,900 completion tokens apiece against ~510 for the same task without the pipeline, took ~950s, and passed. Nothing recorded anywhere could say whether that was a small reasoning model thinking at length or a generation fault, and the two call for opposite responses. This struct is the difference between those two readings.

Every field is taken from events the loop already emitted, so nothing here changes what the eval sends, executes or scores.

Fields

reasoning_chars: u64

Characters the model emitted as reasoning (chain-of-thought).

This is only meaningful when the upstream separates reasoning

It counts AgentEvent::ReasoningDelta, which exists only when llama-server was launched with --reasoning-format deepseek and so splits thinking into its own reasoning_content SSE field. Without that flag a reasoning model’s thinking arrives inline as <think>…</think>, the normalizer strips the tags, and every one of those characters is counted as Self::answer_chars instead.

So reasoning_chars: 0 beside a large answer_chars is ambiguous: it means either the model did not think, or it thought and nobody could tell. Resolve it by checking whether the model carries the reasoning capability tag, not by assuming.

answer_chars: u64

Characters the model emitted as ordinary answer text, summed across every turn — not just the final one.

Counted from AgentEvent::TextDelta rather than from FinalAnswer, which carries the same text already accumulated and would double it.

llm_calls: usize

How many requests the run actually sent to the model.

Distinct from TuneTaskResult::iterations, which counts only tool-executing turns — a run that ends by answering in text made one more request than it reports iterations. Dividing tokens by iterations therefore overstates per-request generation, by 50% on a two-iteration run, which is exactly the arithmetic a reader performs when asking whether a token cap was in force.

Derived from the event stream (one per IterationComplete, plus one for a FinalAnswer), so a run a guard aborted mid-turn under-counts by the aborting request. Read it as a floor on those runs.

max_tool_calls_in_batch: usize

The largest single batch of tool calls any one turn executed.

The fingerprint of a constrained-decoding runaway. gglib’s generated grammar admits root ::= sp call (sp call)* sp — unbounded repetition — so a model that never emits an end-of-generation token can keep producing syntactically valid calls until it hits a token cap or the context limit. Scoring cannot reveal this: extra unrequested calls cost nothing, so a batch of hundreds containing the right call still scores 1.0 and the task still reads as passed.

It saturates at 64

The collector drops tool-call fragments past MAX_TOOL_CALL_INDEX (64), so this is a floor, and a reading of exactly 64 means “at least 64” — the true batch went into CollectedResponse::tool_calls_truncated and from there into a SystemWarning message this eval keeps no text of. Measured 2026-08-29: a reading of 64 was kept=64 dropped=542 — 606 calls in one response, for a task whose expected output is one call. An earlier run logged dropped=1237. Read a 64 as “consult the daemon log”, never as a batch size.

system_warnings: u32

How many recoverable conditions the loop reported during this run.

Counts AgentEvent::SystemWarning, whose main source is the loop recovering from a model that requested more parallel tool calls than the configured limit. That recovery costs a whole extra request and was previously invisible to the eval: the warning was emitted, discarded, and the run reported as though nothing had happened.

Warnings, not incidents. One over-wide batch raises two — the collector’s slot limit and then the parallel-tool limit — so this over-states how many times the model ran away, by up to a factor of two. It is a “something went wrong here” flag; the log holds the account.

Trait Implementations

impl Clone for GeneratedOutput

fn clone(&self) -> GeneratedOutput

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for GeneratedOutput

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for GeneratedOutput

fn default() -> GeneratedOutput

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for GeneratedOutput

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for GeneratedOutput

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.