Human Evaluation of LLMs: Building Rubrics Annotators Can Apply

Why Automated Metrics Are Not Enough

BLEU, ROUGE, and perplexity were designed for narrow generation tasks with clear reference outputs — machine translation, text summarisation against a single gold reference, language modelling on a fixed corpus. Applied to open-ended LLM outputs, they produce misleading quality signals. A model that produces a technically fluent but factually incorrect response to a medical question will score well on perplexity and ROUGE against a held-out reference answer if the language patterns are similar. A model that produces a correct, helpful response in a slightly different phrasing will score worse.

A 2023 study by Karpinska et al. found that BLEU scores correlated with human judgement at only r = 0.23 across 370 translation outputs — weaker than the correlation produced by randomly shuffling scores. For tasks that involve instruction-following, multi-step reasoning, domain expertise, or nuanced communication, automated metrics are close to noise.

LLM-as-judge approaches (using GPT-4 or Claude to evaluate outputs from another model) reduce the cost of evaluation at scale and correlate better with human judgement than lexical metrics. However, they inherit the judging model's biases — preferences for longer responses, own-model outputs, and responses that match the training style of the judge. For model release decisions, human evaluation remains the gold standard. LLM-as-judge is best used as a high-throughput screening layer before human evaluation, not as a replacement.

Rubric Design: Criteria That Annotators Can Apply

The most common failure mode in LLM human evaluation is a rubric that is too abstract to apply consistently. Criterion definitions like "helpful" or "high quality" produce high variance between annotators because each annotator interprets the criterion differently. The result is inter-annotator agreement scores so low that the evaluation data cannot reliably distinguish between model versions.

Effective rubric design requires decomposing each evaluation dimension into behaviourally anchored rating scale (BARS) definitions — concrete descriptions of what a 1, 3, and 5 rating looks like on each criterion. Below is an example decomposition for a helpfulness criterion on a customer service AI:

Note what this rubric does: it describes observable, specific behaviours rather than abstract qualities. "Anticipates the obvious follow-up question" is something an annotator can check. "High quality" is not. The definitions also explicitly address the most common edge case — a response that is partially correct — and tell the annotator exactly what makes it a 3 rather than a 1 or a 5.

Each criterion needs its own BARS definitions. For a general-purpose LLM evaluation, the core criteria are: accuracy (factual correctness), completeness (full coverage of the prompt's requirements), coherence (logical structure and internal consistency), helpfulness (practical utility for the user's goal), and safety (absence of harmful content). Domain-specific models require additional criteria: citation accuracy for research AI, code correctness for coding AI, medical correctness and clinical appropriateness for healthcare AI.

Calibration: Getting Annotators to a Shared Standard

Rubric design alone is insufficient. Annotators reading the same rubric will still apply it differently until they have calibrated on a shared set of examples. Calibration is the process of closing that gap before production annotation begins.

A practical calibration workflow involves three phases:

This calibration process is the same rigorous approach applied in our data QA and validation service for annotation quality across all project types. The underlying principle is universal: you cannot produce reliable quality signals without a calibrated annotation workforce.

Inter-Annotator Agreement: What Scores to Target

Cohen's kappa is the standard IAA metric for LLM evaluation because it adjusts for chance agreement — unlike raw percent agreement, which can be deceptively high when annotators happen to use similar rating distributions. For Likert-scale criteria (1–5 ratings), use weighted kappa, which gives partial credit for near-miss disagreements (rating 3 vs 4 is penalised less than rating 1 vs 5).

IAA benchmarks for LLM evaluation tasks:

When IAA falls below target, the correct response is rubric revision, not annotator replacement. Low IAA is diagnostic information: it tells you which criteria are ambiguous in the rubric, not which annotators are performing poorly. A criterion with kappa 0.38 almost always reveals a definitional gap — the rubric does not adequately distinguish the middle rating from the top and bottom ratings, or two criteria are partially overlapping and annotators are resolving the overlap differently.

Landis and Koch (1977), the most-cited reference for kappa interpretation benchmarks, established that kappa 0.61–0.80 represents "substantial agreement" and kappa above 0.80 is "almost perfect." These benchmarks remain the standard reference for LLM evaluation teams even though the original study addressed medical diagnostic agreement, not NLP annotation. For LLM evaluation specifically, kappa below 0.40 renders the evaluation data unreliable for model comparison purposes.

How Many Items Do You Need?

Evaluation dataset size is a statistical power problem. The number of items required depends on the magnitude of the quality difference you need to detect and the confidence level required. For model release decisions — where you need to be confident a new model version is genuinely better than the current production model — consider:

• To detect a 10-percentage-point quality difference (e.g. 75% vs 85% positive ratings) at 95% confidence: approximately 200–250 items.
• To detect a 5-percentage-point difference at 95% confidence: approximately 700–800 items.
• To detect a 3-percentage-point difference at 95% confidence: approximately 1,800–2,000 items.

Most production LLM teams targeting reliable model comparison use 800–1,200 items, which provides reasonable power to detect 4–5 percentage point quality differences — the typical magnitude of improvement from a fine-tuning iteration. Fewer items are acceptable for broad model selection decisions (choosing between architecturally different models with large quality gaps); more items are required for fine-grained comparisons between model versions trained on similar data.

For each evaluation item, use a minimum of 3 annotators per item to allow majority-vote resolution on disagreements. High-stakes evaluations (safety, medical, legal) should use 5 annotators per item and apply adjudication protocols rather than simple majority vote when annotators disagree.

Case Study: Customer Service LLM Evaluation, 0 to Production in 6 Weeks

In early 2026, an Australian telecommunications company was preparing to deploy an LLM-powered customer service assistant handling billing, network, and device support queries. The team had fine-tuned a 7B model on 14,000 synthetic instruction examples but had no structured evaluation framework — quality assessment had been done via informal employee testing with no rubric.

The evaluation challenge: the model needed to handle 47 distinct query types across three support categories, many requiring specific product knowledge. Automated metrics were useless for quality signals on open-ended support responses. Internal employee testing had produced no useful IAA measurement, with post-hoc analysis showing evaluator ratings varied by ±1.8 points on a 5-point scale for identical responses — effectively random.

The evaluation programme designed and run by AI Taggers:

• Rubric design: Five criteria (accuracy, completeness, helpfulness, appropriateness, efficiency) with BARS definitions for each of the three primary support categories — billing, network, and device. Each category had distinct criterion definitions reflecting the different resolution pathways.
• Annotator team: 8 trained annotators with prior telecommunications experience, calibrated over 3 days using a 30-item gold set prepared by senior customer support staff.
• Calibration result: Weighted kappa ranged from 0.67 (completeness) to 0.78 (accuracy) after one calibration round. One annotator required a second calibration batch on the helpfulness criterion.
• Evaluation scale: 1,100 model responses rated by 3 annotators each, covering all 47 query types with proportional sampling based on actual query volume distribution.

Results: The baseline model scored an average of 3.41/5 on helpfulness and 3.18/5 on accuracy across all query types. Evaluation data identified four query categories with accuracy below 2.5/5 — all in billing dispute resolution, where the model was generating plausible-sounding but policy-incorrect resolutions. The team retrained the model with 800 additional billing-dispute examples (human-authored, reviewed by billing operations staff), targeting these specific failure categories.

Post-retraining evaluation on a new 1,100-item batch: helpfulness improved to 4.02/5 (+17.9%) and accuracy to 3.89/5 (+22.3%). Billing dispute accuracy specifically improved from 2.3/5 to 4.1/5. The model entered production with a structured ongoing evaluation protocol — 200-item monthly evaluation batches run by the same calibrated team. The evaluation investment was AUD 38,000; the team estimated it prevented approximately AUD 340,000 in customer escalation costs by identifying billing resolution errors before deployment.

Absolute Rating vs Pairwise Comparison

Two evaluation paradigms dominate production LLM quality measurement, and the right choice depends on the evaluation goal.

Absolute rating asks annotators to rate a single response on each rubric criterion, typically on a 1–5 Likert scale. It is better suited for: ongoing quality monitoring (tracking model quality over time on the same scale), identifying specific failure categories (low accuracy ratings on a subset of query types is actionable), and evaluating models in isolation without a comparator. The limitation is that absolute ratings are harder to calibrate — annotators' internal scales drift, and what a 3 means to annotator A may differ from what it means to annotator B even after calibration.

Pairwise comparison asks annotators to choose which of two responses (from different models or model versions) is better overall or better on a specific criterion. It is better suited for: head-to-head model comparisons, RLHF preference data collection (where the format directly aligns with training objectives), and evaluation tasks where the quality gap is small. Pairwise comparison produces higher IAA because the binary judgment is less ambiguous than placing a response on a 5-point scale.

The practical recommendation: use pairwise comparison for model selection decisions (choosing which model version to deploy) and absolute rating for ongoing quality monitoring. Many teams run both in parallel — pairwise comparison for the release decision, then convert to absolute rating for the post-deployment monitoring protocol where a comparator model is not available. Our data QA and validation service designs evaluation programmes using both paradigms depending on the team's specific monitoring requirements.

Common Evaluation Mistakes and How to Avoid Them

Frequently Asked Questions