Expand description
Agentic tool-calling task schema for tune runs.
A TuneTask is a single scripted scenario: a prompt, a set of tools
advertised to the model, and an expected outcome. Tasks are evaluated
through the real agent loop (not a toy harness) so the recorded tool
calls reflect exactly what the model would do in production.
§Categories
Modeled after the Berkeley Function Calling Leaderboard (BFCL)
methodology: single-call and parallel-call correctness, multi-turn
(stateful) tool use, and — importantly for avoiding loops — irrelevance
detection (can the model correctly abstain from calling a tool when none
applies). A fifth category, TaskCategory::LongContext, goes beyond
BFCL: it pre-fills the conversation with a long simulated history before
user_prompt, testing whether context degradation over a long session
(attention fixating on stale context) causes the model to loop or
stagnate on a task it would otherwise handle cleanly from a cold start.
Structs§
- Expected
Call - One expected tool call within a task’s
ExpectedOutcome::ToolCalls. - Tune
Task - A single scripted agentic scenario evaluated during a tune run.
Enums§
- Expected
Outcome - What a task expects the agent loop to do.
- Task
Category - Category of an agentic tool-calling scenario, following the BFCL split
(plus
LongContext, which is gglib-specific). - Task
Suite - The set of tasks a tune run evaluates each candidate against.