pub enum EvalArm {
Raw,
Gglib,
RawReplicate,
Control,
}Expand description
Which arm a task ran under.
Variants§
Raw
Pipeline bypassed — bare llama-server behaviour.
Gglib
The full gglib request/response pipeline.
RawReplicate
A/A control. The raw arm again, on a disjoint seed set.
Nothing about the request differs from Self::Raw — same bypass, same
tasks, same machine, same loaded model — so any gap between the two is
the eval measuring itself. That gap is the floor a raw-versus-gglib
delta has to clear, and without it a small delta has two readings that
the report cannot separate: the pipeline helped a little, or five seeds
is not enough seeds.
It answers a strictly different question from Self::Control. The
control establishes that a large change registers; this establishes
how large a change has to be before it registers as anything but drift.
A run carrying only the control can say “the apparatus works” about an
effect it has no ability to resolve.
Control
Positive control. The gglib pipeline with the temperature forced to
CONTROL_TEMPERATURE, which should sample visibly worse.
It exists to answer a question the other two arms cannot: can this apparatus detect a sampling change at all? A raw-versus-gglib run showing no difference has two explanations — the pipeline does not help, or the harness cannot see — and nothing in that run distinguishes them.
This arm is a deliberate, large, known-bad change. If it does not score below the gglib arm, the apparatus failed to move under a difference that should be impossible to miss, and no other number in the report can be believed. That is the same discipline ADR 0004 applies to its instruments: a comparison in which nothing could have varied, reporting that nothing varied, is not evidence.