Arabic Multimodal Data: Annotating Vision-Language Pairs for MENA AI

Why Arabic Multimodal Datasets Cannot Be Translated From English

The dominant approach to building multilingual vision-language training data has been to translate English datasets — COCO Captions, Conceptual Captions, VQAv2 — into target languages using machine translation. For Arabic, this approach fails in two compounding ways: the language is wrong and the images are wrong.

Machine-translated Arabic carries what linguists call translationese — syntactic structures, vocabulary choices, and register levels that reflect the source English sentence rather than natural Arabic expression. A native Khaleeji speaker writing a caption for a busy market image would use vocabulary, sentence rhythm, and cultural reference that differs fundamentally from a translation of the English phrase "A crowded market with vendors selling various goods." The translated version is grammatically valid Arabic; it is not how Arabs describe what they see.

The image problem is equally serious. English image datasets — COCO, ImageNet, Conceptual Captions — draw heavily from Western social media and photography. They contain very few images of traditional Gulf architecture, Islamic prayer spaces, Arab family settings, MENA street environments, or the clothing, food, and cultural objects that appear in everyday Saudi, Emirati, or Egyptian visual experience. A vision-language model trained on translated COCO data learns to describe Arabic text accurately but in a visual world that does not resemble MENA.

According to the CAMEL-Bench evaluation published by MBZUAI in 2024, leading vision-language models including GPT-4V, LLaVA-1.6, and Gemini Pro scored 18–34 percentage points lower on MENA-contextualised visual question answering tasks than on equivalent English tasks. The primary failure modes were: cultural visual concept misidentification (describing a kandura as "a white robe" rather than using culturally accurate vocabulary), incorrect Arabic dialect output when prompted in Gulf Arabic, and spatial reasoning errors on Arabic-script signage.

The Five Annotation Tasks in an Arabic Multimodal Dataset

Arabic vision-language datasets are typically built around five core annotation tasks. Each requires different annotator skills, and dialect matters differently across them.

1. Image captioning in Arabic

A native Arabic speaker views an image and writes a descriptive caption in their natural dialect. For Gulf AI products, this means Khaleeji-speaking annotators from KSA, UAE, Kuwait, or Bahrain. For Egyptian or pan-Arab products, Egyptian Arabic annotators provide captions in a widely understood dialect. The instruction to the annotator must specify: target dialect, desired formality level (colloquial versus formal), caption length constraints, and how to handle mixed-script text in the image (Arabic-English shop signs, for example).

Annotation guidelines must address culturally loaded visual categories explicitly. A photograph of a woman in niqab, a traditional suq, a mosque interior, or a Saudi government building carries cultural context that annotators bring automatically from lived experience — but which must be codified in guidelines to produce consistent labelling across an annotator pool.

2. Visual question answering (VQA) in Arabic

VQA annotation involves one annotator generating questions about an image in Arabic and a second (or the same) annotator providing ground-truth answers. The challenge for Arabic VQA is that questions involving culturally specific objects — the type of food in a dish, the occasion depicted in a social scene, the denomination of currency shown — require annotators with MENA cultural knowledge, not just language competence. An Egyptian annotator can identify Egyptian currency accurately; they should not be assigning answers about Saudi riyal notes.

3. Arabic OCR ground-truth annotation

Images containing Arabic text — street signs, product labels, menus, documents photographed in context — require transcription annotation for training OCR-capable vision models. Arabic OCR annotation must handle: right-to-left text direction, diacritics in formal or educational signage, Arabic numerals versus Eastern Arabic numeral variants, and mixed-script environments where Arabic and Latin text appear in the same image area.

4. Image-instruction pairs for Arabic instruction-following

For fine-tuning instruction-following VLMs on Arabic tasks, annotators create (image, instruction, expected response) triplets. Example: an image of a Saudi government form, with the instruction in Khaleeji Arabic "summarise the purpose of this document," and a ground-truth response that demonstrates accurate document comprehension. This task requires high-competence annotators — typically with professional or university-level Arabic and domain familiarity.

5. Cultural relevance verification

A quality gate task specific to MENA multimodal datasets: native annotators review images flagged for cultural content and assess whether they are appropriate for the target deployment context. Images that are culturally inappropriate for Saudi or UAE markets, that depict alcohol or gambling, or that show gender-mixed settings inconsistent with KSA product requirements, are tagged for exclusion. This task requires annotators from the target cultural market, not generalised Arabic speakers.

Case Study: GCC Retail AI — Product Visual Search in Arabic

In late 2025, a GCC e-commerce retailer operating across Saudi Arabia, UAE, and Kuwait needed to deploy an Arabic-language visual search feature — allowing customers to photograph a product and receive Arabic-language descriptions, category tags, and recommendations. Their existing pipeline used machine-translated COCO captions fine-tuned on a general-purpose multilingual VLM.

Performance on a 1,000-image internal benchmark of Gulf retail products — including traditional clothing (thobes, abayas, bisht), Arabic perfume (oud, bakhoor) packaging, and regional food products — showed:

• Category correct classification: 52% (baseline model on Gulf product imagery)
• Arabic description naturalness rating by Gulf Arabic speakers: 2.6 / 5.0
• Product search precision@5: 0.41 (relevant products in top 5 results)
• Cultural appropriateness errors (misidentified garment type, wrong occasion label): 34% of fashion product descriptions

The team partnered with AI Taggers to build a purpose-built Gulf Arabic retail multimodal dataset. The annotation process involved:

• 62,000 image-caption pairs: Khaleeji-speaking annotators from KSA and UAE wrote product captions in natural Gulf Arabic
• 18,000 VQA triplets: questions and answers about product category, occasion appropriateness, material description, and styling
• Cultural verification pass: 100% of fashion imagery reviewed by a Saudi cultural consultant for appropriateness tags
• PDPL compliance: all product images cleared for commercial use with documented consent chains; no identifiable people in the annotation corpus

Fine-tuning the VLM on this dataset with LoRA (80,000 training steps, cosine LR schedule) produced the following results on the same benchmark:

• Category classification: 52% → 81% (+29 percentage points)
• Arabic description naturalness: 2.6 → 4.3 / 5.0
• Product search precision@5: 0.41 → 0.73
• Cultural appropriateness errors: 34% → 6%

The naturalness improvement was the most commercially significant. Customers in the target market described the baseline output as "like a translation" — technically correct but clearly not written by someone who speaks Gulf Arabic. After fine-tuning, the same panel described descriptions as "sounds like a Saudi store employee." Conversion rate on visual search results increased 23% in the A/B test rollout.

How Many Pairs Do You Need? Benchmark Reference Points

Dataset size requirements for Arabic vision-language fine-tuning depend on the adaptation scope and the base model's existing Arabic capability. As a reference framework based on published research and production projects:

The KAUST Arabic Visual Understanding study (2024) found that 25,000 culturally verified Arabic image-text pairs produced larger gains on MENA VQA tasks than 200,000 machine-translated English pairs. Quality — specifically, cultural accuracy and dialect authenticity — dominates over volume in the range most production teams are working at. Above approximately 500,000 pairs, volume begins to matter more than individual pair quality, but at that scale the annotation budget requires industrial planning.

Dialect Routing for Arabic Multimodal Annotation

The most common operational failure in Arabic multimodal annotation projects is assigning Gulf Arabic annotation tasks to Egyptian Arabic annotators — or MSA to any dialect speaker. Dialect routing is the process of matching annotation tasks to annotators who are native speakers of the relevant variety. For multimodal datasets, dialect routing applies to:

• Caption tasks: captions written in the target deployment dialect, routed to annotators from that dialect region
• VQA question generation: questions phrased in the dialect a target user would use to ask about the image
• Cultural verification: appropriateness review by annotators from the specific national context (KSA ≠ UAE ≠ Egypt)
• QA review: senior reviewers must be native speakers of the same dialect, not just "Arabic speakers"

Automated dialect routing using tools like CAMeL Tools dialect identification is possible for text inputs but cannot be applied to assign annotators for image tasks — the image has no dialect. For multimodal datasets, routing must be based on the target market metadata attached to each image batch, not on analysis of the image content itself.

This operational detail has a direct quality consequence. In a comparative IAA study (Baly et al., 2024), Arabic image captioning inter-annotator agreement dropped from κ=0.74 to κ=0.51 when dialect-mismatched annotators were used — a degradation comparable to using non-native speakers entirely. The degradation was concentrated in culturally specific visual categories (traditional dress, religious objects, MENA-specific food) rather than generic object descriptions.

PDPL and Image Data: What Arabic Multimodal Datasets Must Comply With

Saudi Arabia's Personal Data Protection Law (PDPL), enforced since September 2023, classifies biometric data — including photographs of identifiable people — as sensitive personal data requiring explicit written consent. Arabic multimodal datasets that contain images of recognisable individuals captured in KSA or depicting KSA residents require:

• Documented consent from each identifiable person in source images
• Data residency in KSA or a PDPL-compliant cross-border transfer agreement with SDAIA
• Access logs showing which annotators viewed which images
• De-identification of images not requiring identified persons for the annotation task
• Data deletion capability on request under PDPL's right of erasure

In practice, most GCC retail and commercial multimodal datasets avoid PDPL complications by using product photography without people, or by using images with documented model releases. Datasets depicting public spaces require more careful legal review under PDPL Article 4 and SDAIA's guidance on biometric data processing.

The UAE's Federal Personal Data Protection Law (PDPL 2021, effective 2023) carries similar requirements for Emirati personal data. GCC multimodal dataset projects that span multiple national markets require country-by-country compliance review — the PDPL equivalents are not harmonised across GCC states.

Connecting Arabic Multimodal Data to Arabic NLP Annotation

Multimodal Arabic datasets do not exist in isolation from text-only Arabic NLP work. Vision-language models for Arabic markets depend on the same underlying Arabic language understanding that powers NER, sentiment analysis, intent classification, and document processing. Teams that have already invested in high-quality Arabic NLP annotation pipelines have a significant advantage when extending to multimodal work: the dialect routing protocols, annotator qualification frameworks, and PDPL-compliant data handling they have established for text tasks transfer directly to multimodal annotation.

The annotator pools are also partially transferable. An annotator with proven Khaleeji Arabic NER capability is well-placed for Gulf Arabic image captioning after task-specific orientation. The incremental cost of extending an existing Arabic NLP annotation programme to multimodal tasks is lower than starting a multimodal programme from scratch, because the dialect vetting, consent handling, and QA infrastructure already exists.

For teams currently building Arabic text datasets and planning future multimodal expansion, it is worth establishing cultural image curation protocols at the same time as text annotation — sourcing or commissioning MENA-contextual imagery concurrently with text data collection prevents the bottleneck that arises when the text dataset is ready but the visual corpus is not.

Frequently Asked Questions