Levantine Arabic Content Moderation: What Models Get Wrong Without Native Annotators

Why MSA-Trained Content Moderation Fails on Levantine Arabic

Content moderation AI is trained on what humans label as harmful — and what humans label depends on their cultural and linguistic competence. MSA-trained Arabic moderation classifiers learn from annotators working with formal, explicit Arabic text. Levantine Arabic social expression operates on entirely different norms.

Shami Arabic — the dialect cluster spoken across Syria, Lebanon, Palestine, and Jordan — expresses conflict indirectly. A Lebanese user who writes "يسلم إيدك يا أخي" (literally: "may your hands be blessed, brother") after a lengthy complaint may be deploying the most cutting sarcasm in Lebanese social vocabulary. An MSA classifier reads the religious blessing and marks the post as non-threatening. It is, in fact, a passive-aggressive escalation that native Levantine readers recognise immediately.

Research on Arabic social media NLP published in proceedings of EACL 2023 and ACL 2024 consistently reports that Arabic content moderation classifiers fine-tuned on MSA and Gulf Arabic data produce false positive rates of 35–48% on Levantine Arabic text and true positive rates for genuine Levantine hate speech that are 28–40 percentage points lower than on the dialect they were originally trained for. The models are not malfunctioning — they are performing correctly on the language variety they were built for. The problem is that Levantine is a different language variety with different pragmatic conventions.

For platforms operating in Lebanon, Syria, Jordan, or serving Levantine diaspora communities, this translates directly to account suspensions of users who were engaging in culturally-normal banter, and to genuinely harmful content remaining live because it was encoded in sarcastic register that the classifier scored as benign.

The Levantine Moderation Signals MSA Models Miss

Sarcasm as the primary vehicle for aggression

Levantine Arabic, particularly Lebanese and Syrian social media, uses sarcasm more systematically than almost any other Arabic dialect variety. Genuine hostility is frequently delivered through exaggerated politeness, ironic religious blessings, or mock-formal register. The phrase "والله حيلك تعبني" ("by God, your strength has exhausted me") functions as an insult in Levantine banter but appears, to an MSA classifier trained on overt insult vocabulary, as an expression of admiration or fatigue.

Sarcasm detection in Levantine Arabic requires contextual cues that extend beyond sentence boundaries — the prior conversation turn, the relationship between accounts, and the topic under discussion all modulate interpretation. Native annotators with cultural fluency process these cues automatically. Non-native reviewers, and classifiers trained on explicit insult vocabulary, cannot.

Banter versus harassment: the insider knowledge gap

Lebanese and Syrian social media communities use strong language in affectionate banter between friends. Terms that would constitute harassment between strangers are routine expressions of solidarity between people who follow each other. Platform moderation classifiers have no signal for the relationship between accounts — they evaluate text in isolation.

Native Levantine annotators can identify when vocabulary that resembles harassment is being used in a banter register and provide the correct non-violation label. Without this cultural competence in annotation, the training data itself is mislabelled — teaching the classifier to either suppress all strong language (producing massive false positives in normal Levantine social interaction) or ignore it entirely (missing genuine harassment).

Political and sectarian content in the Levantine context

The Levantine region has specific political and sectarian vocabularies that carry incitement potential that is not present in the same terms when used in other contexts. References to Lebanese confessional groups, Syrian political factions, or Palestinian territorial disputes require annotators who understand the regional history to distinguish political commentary from incitement to sectarian violence.

MSA-trained classifiers built on Egyptian or Gulf data have no representation of these Levantine-specific political contexts. A term that is routine political shorthand in one context can be a sectarian slur in another. Only annotators embedded in Levantine political culture can make these distinctions at the accuracy level platform moderation requires.

French Code-Switching and Multi-Script Harmful Content

Lebanese Arabic mixes French and Arabic within sentences at higher rates than any other MENA variety. A 2022 study of Lebanese Twitter/X posts found that 38% of posts in Lebanese Arabic contained at least one French word or phrase, and that French words appeared disproportionately in emotionally charged content — both positive and negative.

This has a direct implication for content moderation: harassment, threats, and hate speech on Lebanese platforms frequently carry the most explicit harmful content in the French component while the surrounding Arabic provides plausible deniability. An Arabic-only moderation classifier evaluating the Arabic components of a mixed post will miss violations encoded in the French segments.

Effective Lebanese Arabic content moderation annotation requires bilingual Arabic-French annotators who can evaluate the full communicative intent of mixed-language posts as a unified expression, not as two separate linguistic inputs. This is a materially different skill requirement from MSA annotation and from French-language moderation annotation separately.

For our Levantine Arabic NLP annotation work, we route Lebanese platform moderation tasks to bilingual Lebanese annotators who are native Arabic speakers with strong French language competence — the combination required to evaluate Franco-Arabic content accurately.

Case Study: Lebanese Social Media Platform Content Moderation

A Lebanese digital media platform with 2.4 million monthly active users across Lebanon, Syria, and the Gulf Lebanese diaspora brought us a content moderation problem that had generated significant user trust issues. Their automated moderation system — a multilingual BERT-based classifier fine-tuned on a pan-Arabic moderation dataset weighted toward Egyptian and MSA content — was producing unacceptably high account suspension rates among Lebanese users while simultaneously failing to catch genuinely harmful sectarian content.

The baseline metrics before intervention told the story clearly. The classifier's false positive rate on Lebanese Arabic content was 41.7% — meaning over four in ten posts flagged as policy-violating were, on native-speaker review, not violations at all. The false negative rate for genuine Levantine hate speech and targeted harassment was 47.3% — nearly half of real violations were passing through undetected. The combined effect was a moderation system that was simultaneously over-restricting normal Lebanese social interaction and under-detecting the content it was built to catch.

The annotation programme covered 28,000 Lebanese Arabic social posts across four content categories — general discourse, political commentary, sectarian content, and Franco-Arabic mixed posts. Native Levantine annotators from Lebanon (Beirut, Tripoli, and south Lebanon sub-dialects represented) and Syria reviewed each post, with three-way agreement checking and adjudication by a senior Lebanese linguist for borderline cases.

The false positive rate dropped from 41.7% to 13.9% — a 66.7% reduction in mistaken account suspensions. Hate speech recall improved from 52.8% to 83.7%. Critically, Franco-Arabic mixed-post violation detection improved from 31.4% to 76.1%, capturing the harmful content that had been consistently evading the Arabic-only classifier. Human review escalation rate fell from 68.2% to 29.4%, reducing moderator workload by approximately AUD $780,000 annually against an annotation investment of AUD $94,000.

Annotation Workflow for Levantine Content Moderation

Effective Levantine Arabic content moderation annotation requires workflow design that accounts for the dialect's specific challenges. Standard content moderation annotation pipelines — designed for explicit, direct-language content — do not transfer without modification.

The annotation team must include sub-dialect representation. Lebanese, Syrian, Jordanian, and Palestinian Arabic differ in sarcasm conventions, taboo vocabulary, and political sensitivity. A Lebanese annotator will catch Lebanese-specific sarcasm patterns that a Syrian annotator may read differently, and vice versa. For pan-Levantine platform moderation, routing by post origin or detected sub-dialect is the most accurate approach.

Three-way annotation with adjudication is the minimum standard for moderation tasks. Content moderation has higher stakes than most NLP annotation — errors directly affect user account status and platform legal exposure. Inter-annotator agreement (Cohen's kappa) of 0.75 or above is the production threshold for binary harmful/not-harmful classification on Levantine content; fine-grained category classification (hate speech subtype, severity) requires 0.70 or above across categories.

Our Arabic NLP annotation team uses a tiered review structure for Levantine moderation work: first-pass annotation by native dialect annotators, second-pass by a senior Levantine linguist for borderline cases, and monthly calibration sessions to maintain annotation consistency as platform vocabulary evolves. This matters particularly for political content, where new terms and codes emerge rapidly in the Levantine digital context.

Compliance: Regional Data Protection for Moderation Training Data

Social media content used for moderation training is personal data under Lebanese Law No. 81 on Protection of Personal Data (2018), Jordan's Personal Data Protection Law (2023), and the broader Arab League data protection framework. Voice-of-user social content used in annotation must be treated as personal data regardless of whether it was originally public.

For Lebanese clients, Law No. 81 requires explicit consent for collecting and processing personal data for new purposes beyond the original platform use. Using user posts as annotation training data is a new purpose — consent obligations apply unless the data is fully pseudonymised before annotation. Syrian clients face additional complexity due to the transitional status of data governance frameworks following political change in late 2024.

Jordan's 2023 PDPL is the most structured of the Levantine frameworks and most similar in obligation structure to GDPR. Jordanian platform operators working with annotation vendors outside Jordan must assess cross-border data transfer adequacy before sharing user content for annotation. The practical minimum for all Levantine jurisdictions is pseudonymisation of user identifiers, data minimisation to only the text content needed for annotation, and deletion of source content after annotated outputs are validated and ingested into training pipelines.

Cost and Volume Benchmarks for Levantine Moderation Annotation

Levantine Arabic content moderation annotation is priced per post or per annotation decision. Binary harmful/not-harmful classification on standard Arabic social posts (under 280 characters) runs AUD $0.09–$0.22 per item with three-way agreement checking at production volumes of 10,000 items or more. Fine-grained classification adding hate speech category, severity score, and policy rule citation costs AUD $0.28–$0.58 per item.

Franco-Arabic mixed posts requiring bilingual review carry a 30–40% premium over standard Arabic review, reflecting the additional annotator skill required. Long-form content — comment threads, posts with media context — is typically priced per decision rather than per character, with rates between AUD $0.35–$0.85 per decision depending on thread length.

A pilot dataset of 4,000–6,000 Levantine Arabic posts — balanced across sub-dialects, content categories, and violation/non-violation examples — is the standard starting point before committing to full moderation training data production. The pilot establishes realistic inter-annotator agreement baselines for the specific platform's content mix and identifies edge cases that require guideline refinement before scale.

Related reading on our Levantine Arabic annotation work:

• Levantine Arabic Sentiment Analysis: What Models Get Wrong Without Native Annotators
• Levantine Arabic Chatbot Intent Annotation: What Models Get Wrong Without Native Annotators
• Levantine Arabic Named Entity Recognition: What Models Get Wrong Without Native Annotators
• Arabic Sentiment Analysis: The Complete Guide for MENA AI Teams

Frequently Asked Questions