Arabic Instruction-Tuning Data: How to Build SFT Sets That Don't Sound Translated

What Arabic Instruction-Tuning Data Actually Is

Supervised fine-tuning (SFT) on instruction-response pairs is the standard method for turning a pretrained base language model into a useful assistant. You provide thousands of examples of (instruction, ideal response) pairs, fine-tune the model to produce those responses, and you get a model that generalises the instruction-following pattern to new inputs.

For Arabic LLMs serving Gulf users, the instruction-response pairs need to be in Gulf Arabic — specifically in the dialect register the target users actually write in. Gulf users in customer service contexts predominantly write in Khaleeji dialect, often mixed with English for technical terms and brand names. Government services contexts tend toward MSA for official transactions but Khaleeji for clarification questions. Healthcare chatbot inputs skew Khaleeji for symptoms and MSA for documentation.

The problem most teams encounter is resourcing: native-authored Arabic instruction data is expensive and slow to produce. The temptation is to translate English SFT datasets (OpenHermes, Alpaca, Dolly) into Arabic using MT systems, validate superficially, and proceed. This approach consistently underdelivers for Gulf products.

Research published via the ArabicMMLU evaluation track in 2025 found that models trained on English-to-Arabic translated instruction data scored 15–24 percentage points lower on dialect instruction-following benchmarks compared to models trained on natively authored Arabic SFT data of equivalent volume (ArabicMMLU Technical Companion, 2025). This is not a marginal performance difference — it is the gap between a product that works and one that frustrates Gulf users at scale.

The Translationese Problem in Arabic SFT

Translationese is the characteristic unnaturalness of machine-translated or human-translated text. In Arabic, translated SFT data exhibits specific failure patterns that break instruction-following for Gulf users:

• Dialect register collapse: MT systems translate to formal MSA regardless of source register. A casual English request becomes a formal Arabic response — the model learns to respond in MSA to all Arabic inputs, including dialect messages.
• Cultural norm import: English SFT data encodes English pragmatic norms — direct question framing, hedging with "please" and "thank you", low-context communication. Gulf Arabic communication is often more high-context and relationship-oriented. Translated data trains the model on English norms dressed in Arabic words.
• False entity preservation: MT systems often transliterate English proper nouns rather than using Arabic-native equivalents. Gulf users encounter responses with English brand names in Arabic script rather than the commonly used Arabic terms.
• Code-switching blindness: English SFT data has no code-switching. Gulf users routinely mix Arabic and English. A model trained on pure-Arabic (translated) SFT cannot replicate the natural code-switching pattern Gulf users expect in responses.

The cumulative effect is a model that is measurably less useful to Gulf users — not because it lacks knowledge, but because it communicates in a register that feels foreign and bureaucratic. This is a well-documented failure mode of translated training data that compounds particularly severely for Arabic given the gap between MSA and spoken dialects.

What Native-Authored Arabic SFT Data Looks Like

High-quality Arabic instruction-tuning data for Gulf applications has four structural requirements:

• Dialect-native authorship: Instructions and responses are written directly in the target dialect by native speakers — not translated. Gulf/Khaleeji instructions are written by Khaleeji native speakers; MSA formal instructions by MSA-proficient writers; code-switching pairs by bilingual annotators.
• Domain coverage: Instructions cover the domains the deployed model will encounter. For a Saudi banking chatbot, this means Islamic finance queries, account management, regulatory disclosures in MSA, and casual Khaleeji support questions. Generic SFT data from non-domain sources degrades performance on domain-specific instruction types.
• Response quality calibration: Native-speaker reviewers assess whether each response is genuinely the ideal answer to the instruction — not just grammatically correct, but contextually appropriate, culturally aligned, and factually accurate for the Saudi or Gulf context.
• Schema consistency: Instruction-response pairs follow a consistent format (single-turn vs multi-turn, system prompt presence or absence) that matches the deployment context. Inconsistent format in SFT data confuses the fine-tuning process.

The Arabic NLP annotation workflows that produce reliable SFT data involve dialect routing (tasks assigned by dialect match), native-speaker QA review, and an iterative calibration process where annotators align on edge cases before bulk authorship begins. This is substantially more complex than a standard labelling task and requires annotators who are both linguistically expert and familiar with the product domain.

Case Study: Abu Dhabi Government Services Chatbot — Before and After Native SFT

An Abu Dhabi government agency launched a citizen services chatbot in mid-2024 using a multilingual base model fine-tuned on 35,000 English instruction-response pairs translated to Arabic via GPT-4 translation. The chatbot handled residency inquiries, permit applications, and fee payment queries.

Post-launch metrics after 60 days revealed significant underperformance: task completion rate (citizen successfully completing the intended transaction without human escalation) was 38%. User satisfaction surveys showed 71% of Emirati and Khaleeji users cited "language feels too formal / doesn't match how we speak" as the primary complaint. The model was defaulting to MSA even for casual Khaleeji dialect inputs, and struggling with code-switching queries that mixed Arabic and English — common for technical permit terminology.

The agency engaged AI Taggers to produce a replacement SFT dataset of 18,000 natively authored instruction-response pairs. The dataset was structured as:

• 55% Gulf/Khaleeji Arabic (Emirati and broader Gulf sub-varieties) for citizen service queries
• 30% MSA formal register for legal and regulatory response contexts
• 15% Gulf code-switching (Arabic-English) for technical permit and infrastructure terminology

All pairs were authored directly by native Emirati and Gulf Arabic speakers — no translation. Domain specialists reviewed every response in the residency, permits, and payments categories for factual accuracy and regulatory alignment. A PDPL-equivalent data handling framework was applied under UAE PDPL (Federal Decree-Law No. 45 of 2021).

After fine-tuning on the native SFT dataset — same base model, same deployment infrastructure — results over the following 90 days:

• Task completion rate: 38% → 73% (+92%)
• Human escalation rate: 62% → 27% of interactions
• "Language feels unnatural" complaints in user surveys: 71% → 11%
• Code-switching query resolution (Arabic-English mixed inputs): 24% → 69%

The domain knowledge and regulatory accuracy were present in the original model. The failure was not knowledge — it was register. Native SFT data fixed the register, and task completion followed.

How to Scale Arabic SFT Without Losing Quality

Scaling native-authored Arabic SFT from 5,000 to 50,000 pairs introduces calibration drift — annotators diverge on edge cases as volume increases. The production disciplines that prevent this:

• Seed dataset calibration: Before bulk authorship, a seed set of 200–500 pairs is written, reviewed, and adjudicated by senior annotators and the client team. This seed set becomes the quality reference throughout the project.
• Stratified batching: Work is batched by dialect and domain. Khaleeji formal queries, Khaleeji casual queries, MSA regulatory responses, and code-switching pairs are authored in separate batches by matched annotator groups — preventing dialect contamination.
• Rolling inter-annotator agreement (IAA): A proportion of instructions are independently authored by two annotators and responses compared. Divergence above threshold triggers calibration refresh and guideline updates.
• Domain expert review: For regulated domains (finance, healthcare, government), domain subject-matter experts review a sample of responses per batch for factual accuracy and regulatory compliance, separate from linguistic QA.

These workflows mirror the production-grade approaches described in the annotation guidelines guide — specifically the principle that edge case taxonomy and examples-per-class minimums must be established before bulk annotation, not discovered during it.

Vision 2030 and the Saudi SFT Data Opportunity

Saudi Arabia's Vision 2030 AI strategy is creating substantial demand for Arabic SFT data across government digitalisation programmes, Arabic foundation model development (SDAIA, Aramco, NEOM), and private sector AI adoption. The National AI Strategy explicitly targets Arabic language AI as a sovereignty priority — which translates directly into demand for large-scale, dialect-native instruction-tuning datasets.

Several distinctive requirements shape Saudi Arabic SFT:

• Najdi and Hijazi sub-dialect coverage: Saudi Arabic has two major sub-dialect clusters — Najdi (Riyadh and central regions) and Hijazi (Jeddah, Mecca, Medina coast). SFT data for products targeting both regions should include both, with annotators matched to sub-dialect.
• Islamic reference accuracy: Saudi-deployed models handling religious, legal, or financial queries must respond accurately within Islamic jurisprudence frameworks. SFT responses in these domains require annotators with Islamic studies knowledge alongside linguistic competence.
• PDPL compliance from day one: All SFT data handling for Saudi projects — including storage, transfer, and annotator access — must be PDPL-compliant before the project begins, not retrofitted after.

Our Arabic NLP annotation service covers all major Gulf and Saudi sub-dialect requirements with PDPL-compliant data workflows designed for Vision 2030-aligned projects.

Frequently Asked Questions