The Open Arabic LLM Leaderboard (OALL) Explained for AI Teams

What Is OALL and Why Does the Arabic AI Community Need It?

The Open Arabic LLM Leaderboard (OALL) was created to address a specific gap: the absence of a standardised, multi-dimensional evaluation framework for Arabic language models that could function as an independent quality signal for the research community, AI companies, and government AI programmes across the Arab world.

Prior to OALL, evaluating Arabic LLMs required running multiple disparate benchmarks separately: ArabicMMLU for multitask understanding, ACVA for cultural alignment, AlGhafa for NLP capability, and various ad hoc dialect tests. OALL aggregates these into a single leaderboard with consistent evaluation methodology, making it possible to compare models across all dimensions simultaneously and track progress over time.

The leaderboard is maintained by researchers at multiple Arab universities and AI labs, with contributions from the Hugging Face Arabic community. It uses standardised evaluation protocols — same prompts, same few-shot settings, same scoring methodology — across all submitted models, making it the most reliable comparative signal currently available for Arabic LLM capability.

OALL's Benchmark Components: What Each One Measures

Understanding what each OALL component measures is essential for using it as an annotation strategy tool. A low score on one component points to a different type of training data gap than a low score on another.

Who Leads on OALL in 2026 — and Why

The OALL leaderboard in mid-2026 reveals a pattern consistent with the ArabicMMLU-specific leaderboard: no single model leads on all four components simultaneously.

The ACVA column is the most instructive. Frontier multilingual models score 8–12 points lower on ACVA than on ArabicMMLU — their cultural alignment in Arabic is weaker than their factual Arabic comprehension. This is a direct consequence of training data: a model trained on web-crawled Arabic text plus translated English SFT data will produce culturally misaligned outputs even if it scores well on academic-style question answering. The fix is authentic, native-speaker-annotated Arabic instruction-tuning data that embeds Gulf and MENA cultural norms into model behaviour — exactly what our Arabic NLP annotation service is designed to produce.

Case Study: UAE Conversational AI Team — ACVA Score from 59% to 74%

A Dubai-based conversational AI company building a customer-service assistant for the UAE healthcare sector discovered a critical OALL weakness during a pre-launch evaluation round. Their model — a fine-tuned variant of a strong open-weight base model — was performing acceptably on ArabicMMLU (67.4% overall) but failing badly on ACVA (59.1%), particularly on questions relating to Islamic medical ethics, patient communication norms in Gulf healthcare contexts, and gender-appropriate communication styles in a UAE hospital setting.

The issue was identifiable in hindsight: their fine-tuning data was almost entirely translated English healthcare dialogue. The English source data used clinical communication styles, directness levels, and gender interaction norms appropriate for Western healthcare — none of which transferred correctly to Emirati and broader Gulf Arabic healthcare contexts.

AI Taggers designed a two-part annotation programme over ten weeks:

1. 11,000 records of UAE healthcare dialogue SFT data — annotated by native Emirati and Gulf-Arabic annotators with familiarity with UAE healthcare communication norms. Tasks included patient greeting protocols in Gulf Arabic, culturally appropriate phrasing for sensitive health topics, Islamic ethical framing for clinical recommendations, and gender-appropriate register selection.
2. 7,500 records of Khaleeji conversational instruction-tuning data — covering UAE and broader GCC registers for service sector dialogue, with special attention to indirect communication styles and formulaic politeness expressions absent from MSA corpora.

Post-intervention OALL results after fine-tuning on this dataset:

The 15.2-point ACVA improvement was driven almost entirely by the culturally-annotated healthcare dialogue data. In production piloting, patient satisfaction scores for the AI assistant rose from 3.1/5 to 4.3/5 (measured by post-consultation survey) — with the most common free-text complaint in the pre-intervention period being that the assistant "spoke like a textbook" rather than like a respectful Gulf Arabic interlocutor. The cultural alignment data fixed this in a single fine-tuning round, without any architecture changes.

Using OALL as an Annotation Strategy Tool

For Arabic AI teams, OALL serves three practical functions beyond competitive benchmarking:

1. Pre-annotation diagnostic

Before commissioning any annotation, run your current model (or base model) against the full OALL suite. Generate a breakdown by benchmark component and by subject area within ArabicMMLU. This takes approximately four hours on a standard GPU and produces a ranked list of your model's weakest areas — which is your annotation priority list.

A model scoring 65% on ArabicMMLU but 55% on ACVA needs culturally-grounded Arabic SFT data before anything else. A model scoring 70% on ArabicMMLU but 48% on Khaleeji dialect tasks needs Gulf-native annotators producing dialectal dialogue data. These are different annotation products with different annotator requirements, and OALL tells you which one to prioritise.

2. Post-annotation quality gate

Re-run OALL after each annotation and fine-tuning round. Improvements in the targeted benchmark areas confirm that the annotation data is working. A round that improves ArabicMMLU but regresses ACVA signals that the SFT data was culturally neutral (MSA-dominant) when it needed to be culturally embedded. OALL's multi-benchmark structure catches these trade-offs that single-benchmark evaluation misses.

3. Comparative positioning for procurement

Gulf government AI procurement is increasingly requiring OALL benchmark submissions alongside vendor proposals. A model that can demonstrate 70%+ on ACVA alongside strong ArabicMMLU performance signals that it was trained on authentic Arabic data — not translated English content — which is a meaningful differentiator for sensitive government and healthcare applications.

For a deeper look at how Saudi Arabia's AI ambitions are driving demand for Arabic training data at scale, see our analysis of Vision 2030 and the sovereign Arabic LLM push. For teams building on the ArabicMMLU component specifically, our post on ArabicMMLU top models and what they reveal about training data covers the subject-area breakdown in detail. See also our Arabic data labeling service for end-to-end annotation pipelines targeting specific OALL benchmark gaps.

The OALL Data Gap the Leaderboard Reveals

Every model currently on the OALL leaderboard has the same weakness: ACVA scores trail ArabicMMLU scores by 8–15 points. This is a reliable signal that the Arabic AI training data ecosystem is undersupplied with authentic, native-speaker-annotated cultural data relative to MSA factual content.

According to analysis by the OALL maintainers, the primary cause is that the Arabic NLP data commons is built predominantly on MSA news corpora, legal texts, and translated English datasets — all of which are culturally thin relative to the rich oral and social culture of Gulf and MENA communities. Filling this gap requires native-speaker annotators with genuine cultural competence, not just language fluency. A fluent MSA speaker from an Egyptian academic background will not reliably annotate Emirati social norms or Saudi Islamic jurisprudence questions correctly without domain expertise.

This is why the ACVA gap persists even for frontier multilingual models with enormous total training budgets: scale cannot substitute for cultural authenticity when the underlying data is culturally thin. As we explain in our guide to Arabic text annotation tools and best practices, the people question — which annotators, with what native competence — matters far more than the platform or tooling question.

Frequently Asked Questions