Evaluating Arabic RAG Systems: The Annotation Behind Faithful Retrieval

What Makes Arabic RAG Evaluation Distinct From English RAG Evaluation

Retrieval-Augmented Generation (RAG) systems work by retrieving relevant passages from a knowledge base, then using those passages as context to generate a response. The evaluation challenge is measuring faithfulness: does the generated response accurately reflect what the retrieved passages say, without introducing unsupported claims?

In English, faithfulness evaluation is primarily a semantic task — do the claims in the response match the claims in the retrieved documents? In Arabic, faithfulness evaluation has two additional dimensions that English evaluation frameworks do not address.

The first is register mismatch. Arabic knowledge bases — legal texts, medical literature, government documents, news archives — are predominantly in Modern Standard Arabic (MSA) or Classical Arabic. Users of Arabic AI products — government service chatbots, legal research tools, medical information systems — write queries in their spoken dialect: Gulf/Khaleeji, Egyptian, Levantine. When a RAG system retrieves an MSA passage and a generation model paraphrases it into Khaleeji Arabic for the Gulf user, the paraphrase involves genuine translation — not just wording changes but register choices that can shift meaning.

A Gulf Arabic phrase that paraphrases an MSA legal concept may collapse a distinction that matters in the original text. An Egyptian Arabic paraphrase of a medical term may use a colloquial synonym that implies a different severity level. These are faithfulness failures — the response is grounded in the retrieved document but not faithful to its meaning — and they are invisible to automated metrics that operate at the surface token level.

The second is diacritical ambiguity. Written Arabic typically omits vowel marks (tashkeel). The same consonantal string can represent different words depending on how it is vowelised. In formal Arabic texts, diacritical context resolves most ambiguities through grammatical agreement. In a RAG generation model's handling of the retrieved passage, ambiguous terms may be interpreted differently than intended by the source author. Human annotators who are native readers of both the source language register and the target dialect can detect these shifts; automated metrics cannot.

The Five Annotation Tasks in Arabic RAG Evaluation

A production-grade Arabic RAG evaluation dataset requires five distinct annotation tasks applied to each (query, retrieved documents, generated response) triple:

Task 1: Retrieval relevance. Does the retrieved passage actually address the query? Annotators read the query (typically in dialect) and the retrieved document (typically in MSA), and assess whether the document contains information that could answer the query. This requires Arabic-bilingual annotators comfortable with both the query dialect and the source register. A Khaleeji Arabic speaker reading an MSA legal text must assess relevance across the dialect gap.

Task 2: Faithfulness grounding. Is each factual claim in the generated response traceable to at least one retrieved document? Annotators highlight each claim in the response and identify its source passage. Claims with no source passage are flagged as hallucinations. Claims where the source passage says something subtly different are flagged as faithfulness failures.

Task 3: Hallucination detection. Does the response introduce factual claims not supported by any retrieved document? This is the most consequential failure mode for high-stakes Arabic RAG applications — a legal chatbot that adds a claim about Sharia law not present in any retrieved document, or a medical information system that adds a symptom not mentioned in the retrieved clinical text.

Task 4: Dialect coherence. Is the response register appropriate for the query dialect? A Khaleeji Arabic user who asked in Gulf dialect should receive a response in Gulf register — not in MSA or in a different dialect. Dialect coherence annotation requires native speakers of the target dialect who can assess whether the response register is appropriate, not just whether it is grammatically Arabic.

Task 5: Answer completeness. Does the response fully address the query given the retrieved evidence? Partial answers — where the retrieved documents contain the information needed for a complete answer but the response only addresses part of the query — are a common failure mode in RAG systems and require human judgement to detect.

Case Study: Arabic RAG Evaluation for a GCC Legal AI Platform

A Kuwait-based legal technology company built a RAG system for GCC corporate lawyers — a tool for querying Kuwaiti and Saudi commercial law, Islamic finance regulations, and GCC free zone frameworks. The knowledge base contained 840,000 pages of Arabic legal text, predominantly in formal legal Arabic with significant Classical Arabic sections in historical Sharia references.

Initial automated evaluation using RAGAS showed faithfulness scores of 0.81 and answer relevance of 0.77 — performance that appeared acceptable for a beta release. However, testing with actual lawyers revealed that the system was producing responses that were technically grounded in retrieved documents but legally misleading — paraphrasing formal legal Arabic into Gulf dialect in ways that collapsed distinctions between contract terms.

The company commissioned a human evaluation corpus: 1,200 annotated triples covering Kuwaiti commercial law, Saudi SAMA regulations, and Islamic finance contracts, annotated by a team combining legal Arabic specialists (for faithfulness and hallucination) and Khaleeji-native annotators (for dialect coherence). Results:

• Human faithfulness score: 0.61 — vs. RAGAS-automated score of 0.81. The automated metric was overstating faithfulness by 20 percentage points.
• Hallucination rate detected by human annotators: 14.3% of responses contained at least one claim not traceable to any retrieved document — a legal liability issue of material significance.
• Dialect coherence failures: 31% of responses were assessed as register-inappropriate by Khaleeji native speakers — too formal or using MSA legal register where colloquial clarification was expected.
• After targeted fixes to retrieval weighting and generation prompting (informed by the annotation findings): human faithfulness score improved to 0.84, hallucination rate dropped to 3.2%, dialect coherence improved to 89% appropriate responses.
• Total annotation project cost: AUD $44,000. The company's legal team estimated that the pre-fix hallucination rate would have created AUD $600,000+ in legal liability exposure if the system had launched to clients.

The core lesson: RAGAS-level automated metrics are useful for regression testing but not sufficient for Arabic legal RAG systems where faithfulness failures have direct professional and legal consequences. Human annotation establishes the real baseline; automated metrics can then be calibrated against it. Our Arabic NLP annotation service provides this kind of dialect-native, domain-expert evaluation annotation across GCC legal, medical, and government contexts.

Why Automated Arabic RAG Metrics Fall Short

RAGAS is currently the most-used open framework for RAG evaluation, and it provides genuine value for English-language systems. For Arabic, its limitations are structural rather than implementation failures.

RAGAS uses an LLM as a judge to assess faithfulness and relevance. The LLM judge is typically a multilingual model — GPT-4 Turbo, Gemini 1.5, or a similar frontier system. The problem is that these models' Arabic capability is predominantly MSA-trained: they have strong performance on formal Arabic evaluation benchmarks but significantly weaker performance on dialect-specific semantic assessment.

A 2024 analysis of LLM-as-judge performance on Arabic tasks (Arabench evaluation, Abdelali et al.) found that frontier LLM judges achieved 74–82% agreement with human annotators on MSA Arabic tasks but only 48–61% agreement on dialectal Arabic tasks — well below the 80% agreement threshold typically required for automated metrics to substitute for human evaluation. At 48–61% agreement, the LLM judge is barely better than chance for dialect-specific faithfulness assessment.

The practical implication: teams that use RAGAS to evaluate Arabic RAG systems serving dialect-speaking users are measuring something different from what their users experience. A high RAGAS score on an Arabic RAG system does not indicate that the system is faithful to dialect-native users — it indicates that the system appears faithful to an LLM judge with limited dialect competence.

The solution is not to discard automated metrics but to calibrate them. Build a human-annotated gold standard evaluation set, measure automated metric agreement with human labels, and use the calibrated metric for iteration. This is the same approach used in the broader annotation quality measurement literature and produces reliable evaluation infrastructure rather than false confidence.

Building the Arabic RAG Evaluation Corpus: Practical Design Decisions

The design of an Arabic RAG evaluation corpus involves five practical decisions that significantly affect both cost and validity:

Query sourcing. Evaluation queries should reflect the distribution of real user queries, not ideally phrased benchmark questions. For a Gulf legal AI system, queries collected from actual lawyer usage logs (with PDPL-compliant anonymisation) are far more valuable than researcher-authored queries. The specific dialectal phrasings, abbreviations, and query structures that real users employ are the test that matters — and they differ substantially from how researchers think users would phrase queries.

Knowledge base coverage sampling. Evaluation triples must sample across the knowledge base — not just the documents that the system retrieves well. Coverage gaps in the evaluation set will produce inflated system scores. For a 840,000-page legal knowledge base, this requires stratified sampling across document types (legislation, case law, Sharia references, regulatory guidance), time periods, and subject areas.

Annotator dialect matching. The dialect of the annotator assessing dialect coherence must match the target user dialect — not just "native Arabic speaker". For a GCC legal AI product, Khaleeji-native annotators are required. For an Egyptian government services product, Egyptian Arabic-native annotators. Dialect mismatch in the evaluation corpus produces invalid coherence scores.

Hallucination adjudication protocol. Hallucination annotation is inherently subjective at the boundary — a claim that is heavily implied by the retrieved document but not explicitly stated. The evaluation corpus needs a documented adjudication protocol for boundary cases: what does "traceable to retrieved documents" mean specifically, and how are disputed cases resolved? Without this, inter-annotator agreement on hallucination will be below an acceptable threshold (typically 0.70+ kappa for safety-relevant tasks).

Domain expert vs. dialect native division of labour. For knowledge-intensive domains, not every annotator can perform every task. A practical division: domain experts (legal specialists, medical specialists) perform Tasks 1–3 (relevance, faithfulness, hallucination) where subject-matter knowledge is critical; dialect natives perform Tasks 4–5 (dialect coherence, completeness) where register sensitivity is critical. This two-tier approach is more efficient than requiring all annotators to have both domain expertise and dialect competence — a rare combination that significantly increases cost and constrains pool size.

PDPL Compliance for Arabic RAG Evaluation Data

Arabic RAG evaluation datasets derived from real user interactions are subject to PDPL (Saudi Personal Data Protection Law) and UAE PDPL depending on jurisdiction. The query side of the evaluation corpus — real user queries from production logs — is personal data under PDPL because it can reveal the user's identity through content and context.

The standard compliance approach for Saudi-deployed systems: anonymise queries to SDAIA standards (remove named individuals, organisation names, case reference numbers, and any other identifying details) before the evaluation corpus is transferred to an annotation vendor. Maintain a data processing agreement (DPA) that specifies PDPL-compliant handling. For UAE systems, apply equivalent controls under Federal Decree-Law No. 45 of 2021.

For systems where query anonymisation is impractical (legal AI where the query is inseparable from case specifics, or medical AI where the query is inseparable from patient context), the alternative is synthetic query generation: have domain-expert annotators write representative queries that reflect the distribution of real user queries without containing personal data. This approach is slightly less ecologically valid but fully PDPL-compliant without requiring anonymisation infrastructure.

See our detailed post on PDPL vs GDPR for annotation vendors for a full comparison of how the two frameworks apply to AI training and evaluation data.

Arabic RAG Evaluation Across Domains: Different Requirements per Vertical

The annotation requirements for Arabic RAG evaluation vary substantially by deployment domain. Three domains illustrate the range:

Legal AI (GCC). Faithfulness is the primary concern: a legal AI response that paraphrases a statutory provision incorrectly creates liability. Annotators require legal Arabic literacy — ability to parse formal Arabic legal text, understand Sharia legal terminology, and identify when a dialect paraphrase changes the legal meaning. This is the highest-cost annotation category, typically AUD $18–$35 per triple depending on document complexity.

Medical AI (MENA). Hallucination is the primary concern: a medical information system that adds a drug interaction not present in the retrieved clinical literature creates patient safety risk. Annotators require medical domain knowledge — clinical literacy sufficient to evaluate whether retrieved passages support claims about symptoms, treatments, and drug interactions. Medical Arabic RAG evaluation also requires HIPAA-equivalent thinking for UAE deployments and understanding of the Saudi FDA (SFDA) regulatory framework for Saudi deployments.

Government services AI (KSA/UAE). Completeness and dialect coherence are primary concerns: citizens using government services AI need complete answers in their natural register. A response that is factually grounded but formally registers beyond the typical Gulf citizen's reading comfort effectively withholds the information. This domain requires both Khaleeji-native annotators and familiarity with government services terminology — a combination available within our Arabic NLP annotation service annotator pool.

Connecting RAG Evaluation to the Broader Arabic AI Quality Stack

Arabic RAG evaluation annotation does not stand alone — it connects to the broader Arabic AI quality infrastructure that makes production MENA AI systems reliable. The evaluation findings from a RAG evaluation corpus typically reveal needs for improvement in three upstream areas:

Arabic instruction-tuning data. If RAG evaluation reveals consistent dialect coherence failures — the generation model cannot produce natural Gulf Arabic responses to Gulf queries — the fix is usually dialect-appropriate fine-tuning data, not retrieval changes. The hallmark of this failure is a system that retrieves correctly and avoids hallucination but produces responses that feel foreign to dialect-native users.

Arabic safety annotation. If RAG evaluation reveals that the system produces harmful responses when retrieved documents contain edge-case content — inflammatory legal language, sensitive medical information, politically sensitive government documents — the fix involves safety annotation and alignment work. The retrieved content is not the problem; the generation model's handling of it is. Our post on red-teaming Arabic LLMs covers this in detail.

Arabic dataset construction. If RAG evaluation reveals that retrieval is consistently failing — relevant documents exist in the knowledge base but are not being retrieved — the fix often involves improving the Arabic retrieval model's training data: better Arabic embedding training data with dialectal query-document pairs that reflect the actual query-document register gap in the deployment. See our post on building Arabic NLP datasets for the upstream data requirements.

Frequently Asked Questions