How to Choose an LLM Evaluation Vendor: 12 Questions to Ask

Why Vendor Choice Matters More Than You Think

The appeal of automated evaluation — running GPT-4 or a smaller judge model against your outputs — is understandable. It is fast, cheap, and requires no vendor management. The problem is systematic: LLM judges are trained on the same kind of data they are being asked to assess, which means they share the same blind spots.

Research by Zheng et al. (2023) in "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" found that GPT-4 achieves roughly 80% agreement with human expert preferences on general instruction-following benchmarks. That agreement falls substantially for domain-specific tasks — medical diagnosis support, legal reasoning, culturally nuanced dialogue — where the evaluating model lacks the expertise to distinguish a good response from a plausible-sounding bad one.

A separate body of research, including the Stanford HELM study (Liang et al., 2022), consistently shows that human raters with domain expertise surface failure modes that automated evaluators systematically miss: factual errors that a specialist can see and a generalist cannot, tone failures invisible to an English-trained judge, and safety gaps that only appear in specific dialects or cultural contexts.

This is why the choice of LLM evaluation vendor is not administrative — it shapes what your team learns about your model at every release. A vendor with poor rubric design produces noisy scores. A vendor with thin multilingual coverage misses dialect-specific failures. A vendor without IAA reporting leaves you unable to distinguish model quality from measurement noise.

Questions 1–4: Rater Quality and Vetting

Question 1: How do you qualify raters for this task type?

This is the most revealing question you can ask. A strong answer describes a multi-stage qualification process: an initial language or domain knowledge screening, a calibration exercise on task-representative items, and a minimum accuracy threshold on gold-standard items before any rater touches production work. A weak answer — "we screen for education level and English fluency" — tells you that rater qualification is not adapted to your task.

For domain-expert evaluations, qualification must go further: a licensed GP evaluating clinical AI outputs requires clinical credentialling verification, not a general knowledge test. Ask the vendor how they verify professional credentials.

Question 2: What is your rater onboarding pass rate?

A pass rate of 100% means the qualification bar is low or non-existent. A pass rate of 30–50% on a well-designed calibration exercise indicates that the vendor is genuinely filtering for quality. Ask for this statistic directly and treat evasion as a yellow flag.

Question 3: How do you monitor rater quality during the project?

Rater quality degrades over time — a phenomenon called rater drift — as annotators become fatigued or develop idiosyncratic interpretations of the rubric. Vendors who embed gold-standard items throughout the project (not just at onboarding) and continuously monitor each rater's accuracy can catch and remediate drift before it contaminates a significant portion of the batch. Ask specifically whether gold-standard items are present in production batches and at what density.

Question 4: Can you provide IAA statistics from a comparable past project?

Inter-annotator agreement (IAA) statistics — typically Cohen's kappa or Krippendorff's alpha — are the only objective measure of rater consistency. Vendors who run structured human evaluation will have these on file. Vendors who do not produce them either do not measure IAA or know their scores are low. For preference-pair tasks, look for kappa above 0.60; for binary safety classifications, above 0.70.

Questions 5–7: Rubric Design and Calibration

Question 5: Do you design rubrics collaboratively or do we supply them?

The rubric — the set of definitions, examples, and decision rules that raters apply to each evaluation task — is the primary determinant of data quality. Vendors who accept a client-supplied rubric without challenge are a risk: a poorly designed rubric produces internally consistent but meaningless data. Strong evaluation vendors co-design the rubric, stress-test it against edge cases, and iterate until inter-rater calibration scores meet the project threshold.

Question 6: How many calibration rounds do you run before production begins?

At minimum, two calibration rounds are required for a new task: a first round to surface ambiguities in the rubric, a revision, and a second round to verify that the revision resolved them. For complex domain-expert tasks — clinical quality assessment, legal reasoning evaluation — three or more rounds are standard. Vendors who move to production after a single calibration exercise are cutting a corner that will show up in your IAA scores.

Question 7: How are rubric ambiguities resolved during the project?

Real evaluation projects encounter edge cases the rubric did not anticipate. A strong vendor has a documented process: raters flag unclear items, a senior reviewer adjudicates with a written rationale, and the rubric is updated with the resolved case as an example. This process keeps rubric drift — where raters gradually interpret the rubric differently — from degrading data quality over the course of a long project.

Questions 8–9: Language, Domain, and Confidentiality

Question 8: What does your language coverage actually look like for my target market?

"We support 50 languages" is a marketing claim that tells you nothing about actual evaluation quality. The critical distinction is between translation-based coverage — where an English rubric is translated and applied by bilingual workers — and native-speaker coverage, where raters assess in their first language using a rubric designed for that language. LLM evaluation via translation produces systematically worse data because pragmatic and cultural nuance is lost in translation.

For Arabic specifically: Gulf (Khaleeji) dialect evaluation requires raters who can distinguish compliments from sarcasm in Emirati, Kuwaiti, and Saudi registers — a distinction that even fluent MSA speakers often miss. Ask the vendor how many active raters they have for your specific dialect or language variant, not just the top-level language.

Question 9: What confidentiality controls apply to raters handling sensitive prompts?

LLM evaluation often involves prompts that contain commercially sensitive information, proprietary system prompts, or queries that are not safe for general audiences. Ask the vendor: do raters sign task-specific NDAs? Are prompts flagged for sensitivity before distribution? Are there controls preventing raters from saving or sharing prompt content? Is the evaluation platform air-gapped from consumer internet access?

For regulated industries — healthcare, finance, government — also ask whether the vendor's data handling practices are compatible with your data residency requirements. Australian organisations working with health data should verify that all rater access occurs within the required jurisdiction.

Questions 10–12: QA Delivery and Release Cadence

Question 10: What does a standard data delivery include?

Minimum delivery for a production LLM evaluation batch: per-item scores with rater IDs and rationales, IAA statistics broken down by task category, gold-standard item pass rates, and a summary of any excluded items with reasoning. Advanced deliveries also include rater confidence metadata, calibration session summaries, and flags on items where raters disagreed significantly — the disagreement items are often the most informative for model improvement.

Question 11: How do you handle quality disputes?

Quality disputes — where your team believes a batch of scored outputs contains systematic errors — require a clear escalation process. Ask the vendor: who is the escalation contact? What is the agreed response time? How is a disputed batch re-evaluated? Is rework included in the project cost or billed separately? Vendors who cannot answer these questions clearly are telling you that dispute resolution is not part of their standard operating model.

Question 12: Can you support a release-by-release evaluation cadence?

The most valuable use of human evaluation is not a one-off assessment but a continuous programme aligned to your model's development cycle. Each release brings regression risk — improvements in one capability can degrade others — and the only reliable way to catch regressions is structured human evaluation at each release gate.

A vendor who can support a regular cadence — whether weekly, fortnightly, or per-release — becomes a strategic partner in your model development, not a one-time supplier. Ask specifically: do you maintain a standing rater pool for ongoing clients? Can you guarantee turnaround within our release cycle? How do you handle urgent re-evaluation when a critical issue is found post-release?

Our LLM evaluation services are designed around this cadence model, with standing rater pools for clients who evaluate at every release and a structured SLA for turnaround times.

Green Flags and Red Flags in Vendor Proposals

After sending the 12 questions above, read the vendor's written responses against these patterns:

How to Structure the Pilot

After shortlisting 2–3 vendors, run a structured pilot before committing production volume. A well-designed pilot uses 500–1,000 evaluation items, with 10–15% embedded gold-standard items whose correct assessments you have determined independently. This lets you measure: rater accuracy against gold standard, IAA scores on the non-gold items, and qualitative assessment quality on a 50-item expert review.

For multilingual pilots, also have a native-speaking expert who was not involved in the evaluation review a random 50-item sample. A meaningful disagreement rate on this independent audit — above 10% for safety tasks, above 15% for quality tasks — is a warning sign that the vendor's language coverage is thinner than claimed.

The pilot is not a courtesy; it is the most important step in vendor selection. The quality difference between vendors is invisible in their proposals and completely visible in their data. See our related post on human annotation for LLM evaluation for detail on how to design evaluation tasks that produce reliable signal. For teams also considering RLHF data alongside evaluation, our RLHF vendor comparison guide applies many of the same criteria to preference data providers.

Frequently Asked Questions