Multilingual LLM Evaluation: Why Native-Speaker Raters Change the Results

Why Your English Evaluation Results Do Not Transfer to Other Languages

Most LLM evaluation programmes are designed in English and extended to other languages by translation. The rubric is translated. The gold-standard items are translated. Sometimes even the raters are bilingual workers who assess outputs by mentally translating them into English first. The result is an evaluation that measures English-language quality standards applied to non-English outputs — which is not what your users experience.

The scale of the problem becomes visible when you look at pre-training data composition. Analysis of Common Crawl — the primary source for most large pre-training corpora — consistently shows that English and five other high-resource languages (German, French, Chinese, Spanish, Russian) account for well over 90% of usable text. Every model trained on this data has an English-centric set of defaults: English-style directness, English-appropriate politeness levels, English assumptions about information structure.

These defaults are invisible in English evaluation because the evaluation system shares them. They become visible only when a native speaker of Arabic, Turkish, Indonesian, or Korean assesses the same outputs.

What the Research Shows About Non-English Evaluation Quality

The MEGA benchmark (Ahuja et al., 2023 — "MEGA: Multilingual Evaluation of Generative AI") tested state-of-the-art models including GPT-4 on 70 datasets across 80+ languages. The study found consistent performance degradation on non-English tasks, with the gap widening substantially for lower-resource languages. Critically, the study also found that automated evaluation metrics showed weaker correlation with human judgement in non-English settings than in English ones.

A separate result from Zheng et al. (2023) is instructive about the LLM-as-judge problem specifically. Their research on GPT-4 as an evaluator (published as "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena") showed approximately 80% agreement between GPT-4 judgements and human expert preferences on English instruction-following benchmarks. That 80% figure is often cited as validation for automated evaluation — but it applies to English general-capability tasks, which is where GPT-4's training data is most concentrated. For culturally and linguistically specific tasks, the picture is different.

The practical implication for teams deploying multilingual models is that automated evaluation scores may tell you your Arabic or Turkish capability is at 85% of your English baseline, when native-speaker evaluation tells you it is at 65% — with specific, actionable failure categories your automated scores never captured.

Four Failure Categories Native Speakers Catch That Automated Judges Miss

1. Pragmatic failures

Pragmatics is the layer of language that governs appropriate communication in context: how direct is too direct, how formal is insufficiently formal, when an indirect refusal is expected versus when a direct one is appropriate. These norms differ significantly across languages and cultures, and they are almost entirely invisible to automated evaluators.

A customer support assistant responding to an Emirati Arabic-speaking customer with the directness level appropriate for Australian English is not producing a grammatical error — it is producing a pragmatic failure. The response will be rated highly by an LLM judge and poorly by any Gulf Arabic native speaker, because the register signals that the responding party does not understand the cultural context.

2. Dialect-specific register mismatches

Arabic is the clearest example, but the principle applies to any language with significant dialect variation. A model that responds to Gulf (Khaleeji) Arabic queries in Modern Standard Arabic (MSA) is producing technically correct outputs that feel wrong to the user — similar to responding to an Australian asking for "servo directions" with formal written English about "petroleum retail facilities." The content may be accurate but the register signals unfamiliarity with the speaker's actual language context.

Automated judges cannot detect this mismatch because they are not calibrated to the social expectations of a Khaleeji, Egyptian, or Levantine Arabic speaker. Only a native speaker of the relevant dialect can assess whether a response sounds natural in that context.

3. Cultural misattributions

Models trained predominantly on English data absorb English-language assumptions about how institutions work, how social relationships are structured, and what is considered common knowledge. When these assumptions are projected into non-English outputs, they produce responses that are factually wrong for the target market — even though they would be correct in an English-language context.

A model advising an Indonesian user about data privacy rights by referencing GDPR-style frameworks is not producing a hallucination — it is producing a cultural misattribution. The response may be well-intentioned and plausible-sounding to an English speaker, and completely wrong for the user's actual regulatory context. A native Indonesian rater, familiar with OJK regulations and Peraturan Pemerintah on data protection, will catch this immediately.

4. Tone failures in safety-sensitive contexts

Safety evaluation is particularly vulnerable to non-native assessment errors. Expressions of distress, frustration, or intent vary significantly across languages and cultures. Saudi Arabic speakers frequently use somatic language ("my heart is heavy", "my chest tightens") to express emotional distress — expressions that a non-native speaker might not recognise as safety-relevant. Japanese speakers tend to express strong negative emotion indirectly in ways that English-trained safety classifiers systematically underweight.

For safety-critical applications — mental health support, crisis services, vulnerable-user interfaces — the cost of these misses is not a slightly degraded user experience. Our multilingual LLM evaluation services include native-speaker safety raters specifically trained to recognise these culture-specific signals.

Case Study: Evaluating a Multilingual Support Assistant Across Four Language Markets

An anonymised project for a technology company deploying a customer support assistant across Gulf Arabic, Turkish, Indonesian, and English markets illustrates how much evaluation results shift when native-speaker raters replace automated assessment.

The initial evaluation used an LLM judge (GPT-4) across all four languages, scoring helpfulness, accuracy, and appropriateness on a 5-point scale. Scores across all four languages clustered in a narrow band — Gulf Arabic at 78% of the English baseline, Turkish at 81%, Indonesian at 76%. The team was comfortable with these results and planned to move to production rollout.

A native-speaker pilot evaluation — 400 items per language, evaluated by four native raters per language with IAA verification — produced substantially different results. Gulf Arabic dropped to 58% of the English baseline once pragmatic failures and dialect register mismatches were included in the scoring. Turkish dropped to 71%, largely due to formality level errors in customer service contexts. Indonesian dropped to 62%, primarily due to cultural misattributions about regulatory and institutional context.

More importantly, the native-speaker evaluation produced actionable failure categories: specific prompt types where the register failures were most frequent, cultural context areas requiring targeted fine-tuning, and a ranked list of issues by frequency and business impact. The LLM judge evaluation had produced scores. The native-speaker evaluation produced a remediation plan.

After a targeted fine-tuning cycle addressing the top failure categories, a follow-up native-speaker evaluation raised Gulf Arabic to 76%, Turkish to 84%, and Indonesian to 79% of the English baseline — a material improvement, and one that would have been invisible to the automated evaluation system that had approved the initial build.

How to Design a Native-Speaker Evaluation Programme

Building a multilingual evaluation programme that produces reliable signal requires different design decisions than extending an English programme to new languages. The most consequential differences are in rubric design, rater recruitment, and IAA measurement.

Design rubrics natively, not by translation

A rubric designed in English and translated into Arabic, Turkish, or Indonesian will encode English-language quality standards. The calibration sessions will generate discussion about whether a translation of the rubric is accurate, not about what quality actually looks like in the target language. This produces worse data than starting from scratch with a native-speaker rubric designer.

The rubric design process for each language should involve at least one native speaker with both linguistic expertise and familiarity with the domain. For customer support assistants, this means someone who has used customer support in that language market. For medical assistants, a native-speaking clinician or healthcare worker.

Recruit for dialect, not just language

"Arabic speakers" is not a rater pool. Gulf Arabic, Egyptian Arabic, Levantine Arabic, and Modern Standard Arabic speakers evaluate outputs differently because they have different expectations. If your product is for the Saudi market, your raters need to be Gulf Arabic native speakers — preferably Najdi and Hejazi speakers, given the regional register differences within KSA.

The same principle applies to Chinese (Mandarin vs Cantonese, Simplified vs Traditional), Spanish (Latin American vs Iberian registers), and Portuguese (Brazilian vs European). For support from dialect-specific native speaker annotation across 40+ language variants, ensure your evaluation partner can specify rater dialect profiles rather than just top-level language.

Measure IAA per language, not just overall

Inter-annotator agreement statistics should be broken down by language in every evaluation delivery. A project where Arabic IAA is 0.45 and English IAA is 0.78 is not a project with 0.65 average IAA — it is a project where the English evaluation is reliable and the Arabic evaluation is not. Aggregate IAA figures obscure the language-level variation that matters for deciding whether a given market is ready for production.

Our guide on how to choose an LLM evaluation vendor covers IAA requirements and the 12 questions that separate production-quality evaluation partners from general annotation platforms.

Integrating Multilingual Evaluation Into Your Release Cycle

The most effective multilingual evaluation programmes are integrated into the standard release process — not run as one-off exercises before a major launch. Each release brings regression risk: a change that improves Arabic customer service dialogue may simultaneously degrade Turkish formality handling, and this regression is invisible until a native-speaker evaluation catches it.

For teams releasing on a weekly or fortnightly cadence, a standing rater pool is essential — a group of vetted, calibrated native speakers per language who can turn around evaluation batches within your release window. Building this pool from scratch for each evaluation round adds two to four weeks of onboarding overhead each time.

For the data collection side of multilingual AI — building the training and fine-tuning datasets that improve non-English performance — see our overview of multilingual annotation and localisation services. For RLHF preference data specifically, our RLHF vendor comparison guide applies similar criteria to preference data partners for non-English markets.

The evaluation and data sides are also connected: a well-run multilingual evaluation programme identifies the specific failure categories your training data is missing. The most efficient path to a better non-English model is a native-speaker evaluation that tells you exactly what to fix, followed by targeted data collection that addresses those categories.

Frequently Asked Questions