Red-Teaming and Safety Data for LLMs: An Annotation Playbook

Why Red-Teaming Cannot Be Automated

Automated safety testing — jailbreak pattern libraries, keyword filters, automated adversarial prompt generation — identifies known attack vectors that the model has already been trained to resist. It systematically misses the novel, contextually embedded attacks that actual misuse looks like in production.

A 2024 study by Perez et al. at Anthropic found that human red teamers consistently discovered safety failures that automated methods missed, particularly in multi-turn conversation attacks where no single message was harmful but the sequence elicited policy-violating outputs. The study found that the most effective automated methods (gradient-based adversarial prompt optimisation) identified roughly 60% of the safety failures found by human red teamers within the same testing budget — and that the 40% gap corresponded precisely to the failure modes most likely to appear in real-world deployment.

This is not an argument against automation. Automated testing is valuable for coverage at scale — screening millions of examples against known attack patterns efficiently. The argument is that human red-teaming fills the gap that automation cannot close, particularly for novel attack vectors, culturally specific harm, domain-specific misuse, and multi-turn manipulation sequences. A complete safety programme uses both.

Building a Red-Teaming Taxonomy

Effective red-teaming is systematic, not opportunistic. Without a taxonomy, red teamers produce adversarial prompts clustered around a few intuitive attack types — typically the most obvious jailbreaks — while leaving large portions of the model's risk surface untested. A red-teaming taxonomy structures the adversarial prompt generation task so coverage is deliberate.

A red-teaming taxonomy has two dimensions: the attack strategy (how the adversarial prompt is constructed) and the target harm category (what kind of harmful output it attempts to elicit). Each combination of attack strategy and harm category defines a distinct red-teaming task.

For each attack strategy, red teamers write prompts targeting each harm category in the taxonomy. The matrix of attack strategies × harm categories defines the minimum required coverage. Most production red-teaming programmes set a target of 20–50 prompts per cell in this matrix for the highest-risk combinations, and 5–10 prompts per cell for lower-risk categories.

Harm Taxonomy: Designing Categories Annotators Can Apply

A safety harm taxonomy serves two purposes: it structures red-teaming coverage, and it provides the classification framework for safety labelling. Both purposes require the taxonomy to be operationally precise — categories defined in terms of observable output properties, not abstract harm concepts.

A production harm taxonomy for a general-purpose LLM should include the following top-level categories, each decomposed into subcategories:

For domain-specific models, add domain harm categories. A healthcare AI taxonomy adds: clinical advice without appropriate caveats, diagnosis framing for conditions the model cannot reliably identify, and treatment recommendations that contradict clinical guidelines. A financial AI taxonomy adds: investment advice without risk disclosure, market manipulation facilitation, and targeted advice for high-risk financial products. Our custom annotation service designs harm taxonomies tuned to each deployment context as part of the safety data collection process.

Annotator Requirements: Who Should Do This Work

Red-teaming annotation is fundamentally different from standard data labelling. It requires capabilities that general crowdworkers — the annotator pools used for most labelling tasks — do not reliably possess.

Three capabilities are essential:

For domain-specific models, include domain experts in the red-teaming team. A healthcare LLM should be red-teamed by annotators with clinical training who understand what constitutes inappropriate medical advice. A legal AI should include annotators with legal background who can identify bias or erroneous guidance that a non-expert would miss. General red teamers without domain knowledge will miss the domain-specific failure modes that matter most for compliance and liability.

Safety Labelling: IAA and Calibration

Safety labelling — classifying model outputs against the harm taxonomy — requires the same calibration rigour as any annotation task, with higher IAA targets because the decisions are higher stakes.

For binary safe/unsafe labels, target Cohen's kappa ≥ 0.80 before production annotation begins. This is a higher bar than the kappa ≥ 0.60 minimum for general annotation tasks, reflecting the policy consequences of safety misclassifications. A false negative — classifying a harmful response as safe — directly contributes to model behaviour that causes harm. A false positive — classifying a safe response as harmful — degrades model helpfulness.

When binary IAA falls below 0.70, the systematic approach is to analyse the disagreement cases rather than simply re-training annotators. Disagreements cluster on specific content patterns that reveal taxonomy gaps:

• Ambiguous severity thresholds: annotators disagree on whether a borderline output crosses into a harm category. The taxonomy definition for that category needs a severity threshold statement.
• Context-dependent harm: an output that is safe in one context (a nurse asking about medication overdose thresholds) is harmful in another (no professional context established). The taxonomy needs context-sensitivity rules.
• Category boundary confusion: annotators disagree on which harm category applies, not whether harm occurred. Taxonomy category definitions have overlapping examples that need clarification.
• Intent vs output harm: the model attempted to comply with a harmful request but failed (provided incorrect synthesis instructions). Whether an unsuccessful harmful attempt is itself a safety violation needs a policy decision and explicit guidance.

Resolving these disagreement patterns requires taxonomy revision, not annotator replacement. Once the taxonomy definition gap is closed, IAA typically improves substantially on similar cases.

Case Study: HealthTech AI Safety Programme, 4 Months to Production Clearance

In 2026, an Australian digital health company was deploying a patient-facing LLM assistant integrated into a chronic disease management platform. The model answered patient questions about medications, symptoms, lifestyle, and care plans. Regulatory review required the company to demonstrate structured safety evaluation before patient-facing deployment.

The safety challenge was domain-specific: beyond general LLM safety concerns, the model needed to handle medical questions without providing inappropriate clinical advice — a category where the boundary between "helpful health information" and "inappropriate medical advice" is clinically and legally complex. A general-purpose harm taxonomy was insufficient.

The safety annotation programme designed and delivered by AI Taggers:

• Custom harm taxonomy: 6 general harm categories and 4 medical-specific categories (inappropriate diagnosis framing, contraindicated treatment suggestions, medication dosing beyond patient context, emergency escalation failure). Each category defined with clinical examples reviewed by a GP and a clinical pharmacist.
• Red-teaming team: 6 red teamers — 3 with general safety backgrounds and 3 with clinical backgrounds (a nurse, a pharmacist, and a medical student). Over 3 weeks, the team produced 2,840 adversarial prompts across 8 attack strategies × 10 harm categories.
• Red-team findings: the model failed on 347 of 2,840 adversarial prompts (12.2% failure rate). Failure modes concentrated in multi-turn escalation attacks (38% of failures), authority claim attacks from users claiming to be healthcare professionals (27%), and indirect elicitation of medication information (21%).
• Safety labelling: 4 safety labellers calibrated on a 30-item clinical gold set, reaching binary kappa of 0.83 before production. Labelled 4,200 model outputs (red-team failures plus a random sample of production-like queries).
• Safety fine-tuning data: the 347 red-team failures and 180 borderline cases were used to produce 527 refused + explanation training pairs, added to the model's safety fine-tuning dataset.

Post-retraining evaluation on a new 2,000-adversarial-prompt red-team batch: failure rate dropped from 12.2% to 2.8% — a 77% reduction. Authority claim attack failures (the highest-liability category for a patient-facing model) dropped from 27% of failures to 4%. The model passed regulatory clinical safety review and entered patient-facing deployment.

The full safety annotation investment was AUD 94,000, covering taxonomy design, red-teaming, safety labelling, and safety fine-tuning data delivery. The company estimated that a clinical incident from an unsafe model response would carry a regulatory fine and remediation cost of AUD 800,000–1,200,000 under Australian Digital Health Agency guidelines — making the safety data investment a 9–13x risk-adjusted return before considering reputational cost.

Ongoing Safety Monitoring: Beyond the Pre-Launch Programme

A pre-launch safety programme addresses the risk surface that existed when the model was tested. Production deployment changes that surface: users discover novel attack vectors, the model encounters query distributions not represented in the red-team set, and model updates or system prompt changes introduce new failure modes.

Ongoing safety monitoring requires two components:

Production safety sampling: A sample of production conversations — typically 0.1–1% depending on volume — is reviewed by safety labellers against the harm taxonomy. Safety incidents discovered in production sampling are escalated immediately and, if they represent systematic model failures rather than one-off edge cases, trigger a targeted red-teaming sprint to characterise the failure mode and generate additional safety fine-tuning data.

Red-team refresh cycles: As models are updated and as publicly known jailbreaks evolve, the red-team dataset needs periodic refresh. A quarterly red-teaming cycle targeting new attack vectors and any failure modes identified through production sampling maintains coverage of the evolving risk surface. Our custom annotation service supports ongoing safety monitoring programmes alongside pre-launch safety data collection.

Multilingual Safety: The Coverage Gap Most Teams Miss

Safety fine-tuning datasets are overwhelmingly English-language. For LLMs deployed in multilingual contexts — or even in ostensibly English-language products where users code-switch or submit queries in other languages — English-only safety training produces models with substantially weaker safety properties in non-English languages.

This is a documented failure mode, not a theoretical risk. A 2023 study by Yong et al. found that GPT-4 could be induced to produce harmful content at significantly higher rates in Zulu, Thai, and Scots Gaelic than in English, even though the harmful content produced was in English — the model's safety training did not generalise across language boundaries as robustly as its capabilities did.

For models deployed in Arabic, Chinese, Hindi, or any non-English primary language, multilingual red-teaming by native speakers is essential. Language switching attacks — embedding harmful requests in a non-primary language — should be part of every red-teaming taxonomy regardless of the model's stated deployment language, because users will attempt this attack vector in production regardless of what the deployment language is specified to be.

Frequently Asked Questions