Agentic AI: Annotating Tool-Use Trajectories and Function Calls

Why Agentic AI Needs Its Own Annotation Approach

Standard LLM training data annotation evaluates a single input-output pair: a prompt and a response. An agentic AI task is a sequence of input-output pairs connected by state: the agent observes context, selects a tool or action, receives a result, updates its understanding, and selects the next action — repeating until the task is complete or it reaches a step limit. A single agentic task may involve 5–50 discrete tool calls across 3–8 tools.

The GAIA benchmark (Mialon et al., 2023, Meta AI Research) evaluated frontier LLM agents on real-world tasks requiring tool use and multi-step reasoning. Even the best-performing models at the time of evaluation — GPT-4 with tool access — achieved only 15% accuracy on Level 3 tasks requiring 5+ reasoning and tool-use steps. The primary failure modes were not knowledge gaps but execution errors: incorrect tool selection (calling a calculator tool when a search was needed), malformed function call parameters (passing a string where an integer was required), and failure to interpret tool output correctly and update the reasoning state.

These failure modes cannot be addressed by prompt engineering or knowledge fine-tuning — they require training on annotated trajectories where step-level errors are labelled and process reward models can distinguish correct from incorrect intermediate actions. The annotation methodology for agentic training data must therefore operate at the step level, not just the trajectory level.

The Tool-Use Trajectory Annotation Schema

A trajectory annotation schema captures the structure needed to train both supervised fine-tuning (SFT) and process reward models (PRMs) from the same annotated data. The schema operates at two levels: step level and trajectory level.

Step-Level Labels

For each step in a trajectory, annotators capture: the observation (the context presented to the agent at this step), the agent's reasoning trace (chain-of-thought or scratchpad, if visible), the tool or function called, the call parameters in structured form (JSON), the tool output or API response, and a set of binary and categorical step-level labels.

Step-level labels include: tool selection correct (Y/N), parameters correct (Y/N — with free-text specification of the parameter error if N), output interpretation correct (Y/N), reasoning sound (Y/N — does the stated reasoning justify the action taken), unnecessary step (Y/N — does this step add no value and could the task have progressed without it), and safety concern flagged (Y/N). The combination of tool-correct and parameters-correct labels distinguishes the two most common tool-use failure modes: the agent knows which tool to use but formats the call incorrectly, versus the agent selects the wrong tool entirely.

Trajectory-Level Labels

Trajectory-level labels evaluate the complete execution sequence: task completed successfully (Y/N), final answer correct (Y/N — evaluated against a gold standard answer where available), efficiency rating (steps used vs minimum required, scored 1–5), error recovery observed (Y/N — did the agent recover from a tool error or API failure during the trajectory), and overall quality rating (1–5).

Trajectory-level labels are used for outcome reward modelling and for filtering which trajectories are used for SFT (typically using only trajectories rated task-completed-successfully = Y and quality ≥ 3). Step-level labels enable process reward modelling across all trajectories, including unsuccessful ones — a trajectory where the agent makes three correct tool calls and then fails on the fourth provides step-level positive examples for the first three steps and a negative example for the fourth.

Annotator Requirements and Domain Expertise

Tool-use trajectory annotation cannot be performed by general-purpose annotators without domain expertise. The step-level correctness labels — particularly parameter correctness and output interpretation correctness — require the annotator to understand what a correct tool call looks like and what the tool output means in context. This is domain-dependent: a financial agent's tool calls require a financial expert to evaluate, a medical coding agent's tool calls require a clinical coder, a legal research agent's tool calls require a paralegal or solicitor.

Technical literacy is a separate requirement from domain expertise. Annotators must be able to read JSON function call parameters without misinterpreting data types (a date string "2026-09-15" versus an integer timestamp 1757865600), evaluate null values and error responses from APIs (a 404 response is not the same as a valid empty result), and distinguish structural correctness of a parameter (the JSON is valid) from semantic correctness (the parameter value is what the task requires). Annotator training programs for trajectory annotation should include a technical literacy module with worked examples of correctly and incorrectly formatted tool calls before domain content begins.

Our custom annotation service matches domain-expert annotators to agentic AI trajectories based on task domain — finance, legal, medical, engineering, customer service — with technical literacy screening before assignment.

Case Study: Enterprise SaaS — Task Completion 52% to 83% with Trajectory Annotation

An enterprise SaaS company building an internal AI agent for procurement workflows — the agent handles supplier search, price comparison, purchase order generation, and approval workflow routing — deployed an initial GPT-4-based agent using prompt engineering and zero-shot tool calling. After six weeks in production with 200 pilot users, the team collected task completion metrics and identified persistent failure patterns.

Before: The agent's overall task completion rate was 52.3% across the 14-step procurement workflow. The most common failures were: incorrect supplier search query formulation (the agent passed unstructured natural language queries to a structured API expecting field-value parameters, failing in 38% of supplier search steps), purchase order generation errors (wrong date format in PO templates, failing in 24% of PO generation steps), and approval routing errors (incorrect role mapping based on cost thresholds, failing in 31% of approval routing steps). Users were abandoning agent-initiated procurement tasks and reverting to manual workflows.

The annotation project collected 3,200 procurement workflow trajectories — 2,100 from human demonstrators (procurement specialists completing tasks using the tool set with the agent interface), and 1,100 from the deployed agent (successful and failed trajectories, with per-step correctness labels). Step-level annotation was performed by eight annotators with procurement and finance backgrounds, calibrated on 200 gold-standard trajectories with expert-consensus labels. Inter-annotator agreement reached κ = 0.78 for tool selection labels and κ = 0.72 for parameter correctness labels after three calibration rounds.

The annotated trajectories were used for supervised fine-tuning on a base LLM and for training a process reward model on the step-level labels. The process reward model was used to re-rank candidate tool calls at inference time — selecting the tool call that the PRM rated highest rather than the one the base model generated first.

After: Task completion rate improved from 52.3% to 83.1% — a 31-point improvement. Supplier search step success improved from 62% to 91%. Purchase order generation step success improved from 76% to 95%. Approval routing step success improved from 69% to 88%. User abandonment of agent-initiated tasks dropped from 47% to 12%. The annotation project cost AUD $58,000 for 3,200 trajectories with step-level labels and PRM training set preparation. The procurement team estimated the agent's productivity contribution at AUD $420,000 per year in staff time recaptured from manual procurement workflows, based on average task completion time improvement from 47 minutes (manual) to 8 minutes (agent-assisted).

Process Reward Modelling vs Outcome Reward Modelling

Outcome reward modelling (ORM) trains a model to evaluate the quality of a complete trajectory based on whether it achieved the task goal. ORM is straightforward to annotate — the annotator needs only a trajectory-level success label — but provides sparse feedback that is difficult to use for RL training of long-horizon tasks. A 20-step trajectory that fails on step 18 receives the same failure signal as one that fails on step 1, obscuring the 17 correct steps.

Process reward modelling (PRM) trains a model to evaluate the quality of individual steps within a trajectory. Let et al. (2023, OpenAI) showed that PRM-guided search significantly outperformed ORM-guided search on mathematical reasoning tasks — the PRM could identify the first incorrect reasoning step and redirect the search before the error propagated. The same principle applies to tool-use trajectories: a PRM trained on step-level correctness labels can identify incorrect tool calls before they cause cascading failures in subsequent steps.

The practical trade-off is annotation cost. Step-level trajectory annotation costs 4–8× more per trajectory than trajectory-level outcome annotation, because annotators must evaluate each step rather than only the final outcome. Teams building narrow-domain agents with deterministic tool outputs (APIs with well-defined success/failure responses) can often use automated validators for parameter correctness and reserve annotator time for reasoning soundness and edge cases. Teams building general-purpose agents with ambiguous tool outputs require full step-level human annotation.

Collecting Trajectory Data: Human Demonstration vs Agent Rollouts

Trajectory annotation data can be sourced from two primary methods: human demonstration (domain experts completing tasks using the agent's tool set) and agent rollouts (the deployed or prototype agent completing tasks, with trajectories annotated post-hoc). Each has different quality characteristics and cost structures.

Human demonstration trajectories are high-quality by construction — a domain expert completing a procurement workflow correctly provides a near-perfect positive example. The annotation effort is primarily verification (confirming the demonstrator made no errors) rather than step-level correctness evaluation. Human demonstration is expensive to collect at scale: a domain expert may complete 15–30 trajectories per day on complex tasks, versus an automated agent that can generate thousands of trajectories per day. Human demonstration is most cost-effective for the first 500–2,000 trajectories that seed the SFT fine-tuning.

Agent rollouts — trajectories from the agent itself — are cheaper to collect at scale and cover failure modes that human demonstrations do not produce. The annotation burden is higher (full step-level correctness evaluation rather than verification), but the resulting labelled failures are the most valuable training signal for process reward models. A mix of 60–70% human demonstration and 30–40% agent rollout trajectories typically produces the best SFT and PRM training data for narrow-domain agents.

Related reading: our guide to RLHF vs DPO preference data covers the preference learning stage that follows SFT fine-tuning on trajectory data, and our post on LLM evaluation rubric design addresses how to build the quality rubrics that trajectory annotation schemas build on.

Safety and Risk Annotation for Agentic Systems

Agentic AI systems that take real-world actions — sending emails, making API calls, executing financial transactions, modifying files — carry safety risks that standard LLM annotation does not address. A trajectory annotation schema for production agentic systems must include safety labels at both the step and trajectory level.

Step-level safety labels flag: irreversible actions taken without confirmation (sending a payment, deleting a file, submitting a form), actions taken outside the agent's authorised scope (accessing a database table the agent was not configured to query), actions that expose sensitive information to an external tool (passing customer PII to an API that was not designated to receive it), and prompt injection attempts in tool outputs (a tool response containing instructions that attempt to redirect the agent's subsequent behaviour).

Prompt injection in tool outputs — where an external API or web page returns text that contains instructions attempting to redirect the agent — is an emerging safety concern specific to agentic systems. Annotators reviewing agent rollout trajectories should flag tool outputs that contain instruction-like text (imperative sentences, phrases like "ignore previous instructions", "your new task is", or "system:") for safety review, regardless of whether the agent responded to the injection attempt.

For custom agentic annotation with safety labelling built into the schema, see our custom annotation service and our guide to red-teaming and LLM safety data.

Related Reading

• RLHF vs DPO: Which Preference Data Should You Collect?
• Human Evaluation of LLMs: Building Rubrics Annotators Can Apply
• Red-Teaming and Safety Data for LLMs: An Annotation Playbook
• Custom Annotation Services

Frequently Asked Questions