Module benchmark 

benchmark

Benchmark domain types.

Two benchmark modes today (a third, tuning, is being added incrementally):

• Compare (compare): send the same prompt to N models sequentially; capture live streamed text and real-world timing data from llama-server’s timings response field.
• Perf (perf): run llama-bench for raw prompt-processing (pp) and token-generation (tg) throughput in tokens/sec.
• Tune (tune): sweep sampling parameters for one model against an agentic tool-calling task suite, scoring each candidate for tool-call accuracy and loop/stagnation avoidance to find the best-scoring settings.

All timing fields are Option<f64> because llama-server may omit the timings object (e.g. older builds, stream errors). Missing timing data is gracefully represented as None — never causes a panic or parse error.

Modules

Module	Contents
run	BenchmarkRun, BenchmarkRunType, BenchmarkRunStatus
summary	ModelBenchmarkSummary — denormalised per-model aggregate
compare	CompareConfig, ModelCompareResult
perf	PerfConfig, ModelPerfResult
tune	TuneConfig, task-suite schema, scoring result types
events	BenchmarkEvent (SSE units), BenchmarkModelResult

Modules

Module	LOC	Complexity	Coverage
agentic.rs
agentic_tests.rs
compare.rs
events.rs
perf.rs
run.rs
summary.rs
tune/

Re-exports

pub use agentic::AgenticEvalConfig;

pub use agentic::AgenticEvalReport;

pub use agentic::AgenticTaskComparison;

pub use agentic::ArmDelta;

pub use agentic::ArmScores;

pub use agentic::CONTROL_MIN_COMPOSITE_GAP;

pub use agentic::CONTROL_MIN_P;

pub use agentic::CONTROL_TEMPERATURE;

pub use agentic::CONTROL_TOP_K;

pub use agentic::CONTROL_TOP_P;

pub use agentic::ControlVerdict;

pub use agentic::DEFAULT_SEEDS;

pub use agentic::DeltaWithheld;

pub use agentic::EFFECT_NOISE_RATIO;

pub use agentic::EffectVerdict;

pub use agentic::EvalArm;

pub use agentic::REPLICATE_SEED_OFFSET;

pub use agentic::control_sampling;

pub use agentic::replicate_seeds;

pub use run::BenchmarkRun;

pub use run::BenchmarkRunStatus;

pub use run::BenchmarkRunType;

pub use tune::CandidateSource;

pub use tune::GeneratedOutput;

pub use tune::ScoreWeights;

pub use tune::SweepSpec;

pub use tune::TaskCategory;

pub use tune::TaskSuite;

pub use tune::TuneCandidateResult;

pub use tune::TuneConfig;

pub use tune::TuneTask;

pub use tune::TuneTaskResult;

Modules

agentic

Raw-vs-gglib A/B agentic evaluation: config and report types.

compare 🔒

Compare-mode benchmark types: configuration and per-model results.

events 🔒

SSE / channel event enum shared by all benchmark run types.

perf 🔒

Perf-mode benchmark types: llama-bench configuration and results.

run

Benchmark run metadata: type, lifecycle status, and the run record itself.

summary 🔒

Denormalised per-model benchmark summary.

tune

tune

Structs

CompareConfig

Configuration for a compare benchmark run.

ModelBenchmarkSummary

Denormalised benchmark summary for a single model.

ModelCompareResult

Result of running a single model through a compare (inference) benchmark.

ModelPerfResult

Result of running llama-bench on a single model.

PerfConfig

Configuration for a performance (llama-bench) run.

Enums

BenchmarkEvent

Typed event emitted over the mpsc channel (and serialised as SSE to the browser) during a benchmark run.

BenchmarkModelResult

Wraps either a compare or perf result for BenchmarkEvent::ModelComplete.