Yemeni Arabic Chatbot Intent Annotation: What Models Get Wrong Without Native Annotators

Why Yemeni Arabic Breaks Standard Arabic Intent Classifiers

Standard Arabic chatbot NLU models are trained on MSA, Egyptian Arabic, or Gulf Arabic conversational data. Yemeni Arabic diverges from all three in ways that systematically corrupt intent classification. The divergence is not just lexical — it is pragmatic, morphological, and register-level.

At the morphological level, Yemeni Arabic — particularly San'ani — preserves Classical Arabic phonological and grammatical features that most other dialects have simplified. The negation circumfix ‘ما...ش’ used in San'ani speech produces sentences that MSA parsers segment incorrectly, misidentifying the scope of negation and flipping the polarity of the intent. A negated request (“I don't want to cancel, I want to modify”) becomes a cancellation intent in the classifier output because the negation scope is missed.

At the pragmatic level, Yemeni conversational norms structure chatbot interaction differently from MSA or Egyptian norms. Yemeni users routinely open with multi-sentence honorific preambles before stating the functional request. The first sentence of a chatbot message may be pure social framing — and it will be the highest-confidence sentence in the classifier's intent scoring — while the actual request appears in the second or third sentence with lower lexical salience to a non-native model.

Published evaluations at the WANLP 2022 workshop and ACL 2023 shared tasks confirm the scale of the problem: intent classifiers fine-tuned on MSA and Egyptian Arabic data show accuracy degradation of 38–51% on Yemeni dialect test sets, with the largest drops in complaint intent, service modification requests, and disambiguation utterances (Habash et al., 2023; Salameh et al., 2022). Arabic NLU benchmarks widely used for fine-tuning — ATIS-Arabic, SQuAD-Arabic — contain fewer than 1% Yemeni dialect examples, giving models no Yemeni calibration signal whatsoever.

Five Intent Patterns That Break MSA Models on Yemeni Chatbot Text

1. Tribal honorific openers that bury the request

The most pervasive pattern in Yemeni chatbot interaction is the honorific opener. Yemeni users frequently begin messages with tribal politeness formulae — ‘يا أهل الكرم’ (people of generosity), ‘حفظكم الله’ (may God preserve you), or sub-dialect-specific salutations — before stating the functional request. These openers are not preamble noise; they are socially obligatory framing devices that establish the relationship context before a request is made.

An MSA intent classifier reads the opener as the primary intent signal and classifies the message as ‘greeting’ or ‘positive_expression’, ignoring the functional transfer, query, or complaint that follows. Native Yemeni annotators recognise these constructions immediately and classify by the substantive intent in the trailing sentences. Annotation guidelines must list the 15–20 most common Yemeni honorific openers and explicitly instruct annotators to classify by the substantive utterance content, not the framing preamble.

2. Indirect complaint through honour-register constructions

Yemeni social norms around tribal honour produce highly indirect complaint expression in written digital communication. A customer reporting a failed transaction through a Yemeni-facing fintech chatbot is unlikely to write ‘أريد أن أشكو’ (I want to complain). Instead, they may write ‘ما كان هذا من أمثالكم’ (this was not befitting of people like you) — a strong complaint expressed through a tribal honour register that MSA classifiers consistently categorise as neutral social statement or positive expectation.

For fintech and customer service chatbots, misclassifying complaint as neutral means failed escalation — the complaint never reaches a human agent queue. In the case study below, this was the primary driver of customer dissatisfaction metrics before the annotation rework.

3. San'ani negation scope errors producing intent inversion

San'ani Arabic uses a negation circumfix pattern that wraps the verb and often the object of the sentence in a negation scope structure absent from MSA. When MSA parsers encounter this construction, they typically identify only the pre-verbal negation particle and miss the post-verbal negation suffix, producing an incorrect positive parse of the sentence. In intent classification, this means a ‘do_not_cancel’ utterance gets classified as ‘cancel_account’ and a ‘not_a_complaint’ clarification gets classified as ‘complaint’.

This pattern is particularly damaging in service chatbots because it produces false escalations — the system escalates a non-issue to a human agent while missing the actual complaint. Annotation guidelines for Yemeni intent tasks must include an explicit negation scope section with San'ani examples and correct annotation decisions for each construction.

4. Hadrami diaspora code-switching for financial intent

Gulf-based Yemeni diaspora users — particularly Hadrami speakers in Saudi Arabia and UAE — frequently code-switch between Hadrami Arabic and Gulf Arabic in financial chatbot interactions. They may use Gulf Arabic terminology for financial products (‘حوالة’ for remittance, ‘كرت’ for card) while framing the request in Hadrami syntactic structures. Standard Gulf Arabic intent models have the vocabulary but miss the Hadrami framing; standard Yemeni models have the syntax but lack the Gulf financial lexicon.

Annotating Hadrami diaspora chatbot data correctly requires annotators with experience in both Hadrami Arabic and Gulf-region code-switching patterns — a combination that crowdsourced platforms cannot reliably recruit for. Our Yemeni Arabic NLP annotation service maintains annotators with Hadrami homeland and Gulf-diaspora experience specifically for this use case.

5. Conflict-era vocabulary shifting intent semantics

Since 2014, a layer of conflict-related vocabulary has entered everyday Yemeni Arabic digital communication. In chatbot contexts, conflict-era terms can appear in what would otherwise be standard service enquiries — a user asking about a ‘transfer’ may use conflict-associated vocabulary that shifts the semantic register of the request. MSA classifiers with no exposure to this vocabulary produce inconsistent intent scoring on sentences containing it, sometimes misclassifying routine financial intents as out-of-scope or escalation-worthy content.

Annotation guidelines for Yemeni chatbot intent tasks must include a conflict-vocabulary supplement that maps the most common conflict-era terms to their functional semantic context in chatbot interactions — distinguishing when they modify a financial intent versus when they appear as incidental background vocabulary.

Sub-Dialect Coverage Requirements for Yemeni Intent Annotation

Yemeni Arabic spans four primary sub-dialect groups with distinct honorific conventions, negation patterns, and vocabulary. For chatbot intent annotation, these differences are operationally significant — assigning San'ani highland text to a Hadrami annotator produces IAA scores 18–25% below intra-dialect annotation on ambiguous intent utterances.

San'ani (Central Highlands, Sana'a region) is the dominant register in Yemeni government digital services, educational platforms, and domestic e-commerce chatbots. It uses the most indirect honorific opener constructions and the most complex negation patterns. Hadrami (Hadramawt and Gulf diaspora) is the most commercially significant for fintech products serving the Gulf diaspora, with distinct remittance and financial service vocabulary. The Gulf-diaspora register of Hadrami additionally incorporates Gulf Arabic financial lexicon that requires bilingual Hadrami-Gulf annotators. Ta'iz–Ibb (Southwestern Highlands) is the most populous Yemeni sub-dialect region and is prominent in migrant-worker chatbot interactions in Saudi Arabia. Adeni adds port-city vocabulary with English and South Asian loanwords relevant to logistics and trade chatbots.

The Yemeni diaspora in Saudi Arabia alone numbers over one million people, with a further 500,000 in UAE (IOM Yemen Diaspora Report, 2024). For Gulf-facing fintech platforms and remittance services, this is a commercially significant user population that requires dedicated sub-dialect coverage — not generic ‘Arabic’ NLU.

Case Study: Yemeni Diaspora Remittance Chatbot — 44% to 87% Intent Accuracy

A Gulf-based digital remittance platform targeting the Yemeni diaspora in Saudi Arabia and UAE operated a WhatsApp-integrated Arabic chatbot handling transfer initiation, balance enquiry, fee dispute, and account management intents. The chatbot used an intent classifier fine-tuned on MSA and Egyptian Arabic conversational data — standard for Arabic NLU at the time of deployment.

Before: The intent classifier achieved 44.1% overall accuracy on held-out Yemeni-dialect chatbot logs. Complaint intent recall was 29.3% — meaning 70.7% of complaint utterances were classified as neutral enquiry or greeting, never triggering escalation. Transfer initiation intents embedded in honorific-opener messages were missed at a rate of 63%, routing to a ‘greeting’ fallback instead of the transfer flow. The platform's customer satisfaction score for the chatbot channel was 2.1 out of 5, with the most common negative feedback phrase translating as “it doesn't understand me.”

The annotation project produced 21,400 labelled utterances across 18 intent classes, covering San'ani, Hadrami (homeland and Gulf-diaspora register), and Ta'iz–Ibb source text. The annotation team of twelve native Yemeni annotators — five San'ani-native, five Hadrami-native (three with Gulf-diaspora experience), and two Ta'iz-native — worked with annotation guidelines developed in collaboration with a senior Yemeni linguist. The honorific-opener protocol was tested on a 2,000-utterance calibration set before full-scale production. Final inter-annotator agreement kappa across the dataset was 0.84; disagreements were adjudicated by the senior linguist.

After fine-tuning on the annotated dataset: Overall intent accuracy improved from 44.1% to 87.3%. Complaint intent recall improved from 29.3% to 83.6%. Transfer initiation intents in honorific-opener messages were correctly classified at 91.2%. The platform's chatbot customer satisfaction score improved from 2.1 to 4.3 out of 5. Escalation to human agents for genuine complaints improved from 29.3% to 83.6%; false escalations of neutral enquiries dropped by 74%. The annotation project cost AUD $52,000. Estimated twelve-month revenue impact from improved chatbot conversion and complaint resolution: AUD $1.1M, driven primarily by reduced churn in the Hadrami diaspora user segment and improved first-contact resolution on transfer-initiation failures.

Annotation Protocol for Yemeni Arabic Chatbot Intent Projects

Yemeni Arabic intent annotation requires a structured protocol that differs from generic Arabic NLU annotation in four key respects.

Honorific-opener handling protocol. Before the annotation project begins, compile a list of the 15–20 most common Yemeni Arabic honorific opener patterns from a sample of source chatbot logs. Add an explicit guideline section instructing annotators to identify these openers, skip them as framing devices, and classify by the substantive intent in the following sentences. Annotators should mark which sentences they used for classification so QA reviewers can spot-check adherence.

Sub-dialect routing before assignment. Source text should be pre-screened for sub-dialect markers — at minimum, San'ani versus Hadrami — before annotator assignment. This can be done manually by a senior Yemeni linguist on a sample, then extended using lightweight dialect detection on the full dataset. Cross-dialect assignment on ambiguous honorific or negation constructions produces IAA drops of 18–25% that compound across a large annotation project.

Intent boundary calibration pilot. Yemeni Arabic complaint and service modification intents are particularly ambiguous at the boundary — honour-register complaint can look like a polite enquiry even to experienced annotators from a different Yemeni sub-dialect. A 1,500–2,000 utterance adjudicated calibration pilot covering all target intents and sub-dialects should precede full-scale production. IAA should be measured per intent class and per sub-dialect, not just overall, to surface class-level and dialect-level annotation inconsistencies.

Conflict-vocabulary supplement. Annotation guidelines should include a supplement covering the 25–30 most common conflict-era Yemeni Arabic terms that appear in the source chatbot domain — mapped to their functional intent context and flagged for human adjudication when they appear as the primary semantic content of an utterance. This supplement should be compiled by native annotators from the relevant sub-dialect communities and reviewed before production begins.

For multi-intent and slot-filling tasks — common in financial chatbots where a single utterance may contain a transfer intent, a beneficiary entity, and a fee dispute — see our guide on end-to-end Arabic data labelling pipelines for structured entity and intent co-annotation workflows.

Tooling and Infrastructure for Yemeni Arabic Intent Annotation

Standard Arabic annotation platforms — Label Studio, Doccano, Prodigy — support Arabic text input and RTL rendering, but none provides Yemeni-specific features out of the box. The infrastructure requirements for Yemeni chatbot intent annotation are primarily operational rather than tool-specific.

Intent taxonomy configuration should present the full taxonomy with definitions visible to annotators during labelling — Yemeni Arabic intent boundary ambiguities are frequent enough that annotators need the taxonomy reference on-screen rather than memorised. For multi-intent tasks, the annotation tool must support overlapping label spans; Prodigy's token classification interface or Label Studio's nested classification views work well for this.

QA infrastructure should flag for manual review any utterance where annotators from different sub-dialects disagreed — these are the documents most likely to contain honour-register or negation-scope constructions that need expert adjudication. Automated IAA monitoring per sub-dialect should be set up from the first annotation batch so that systematic calibration issues are caught early rather than discovered at project end.

Our Arabic NLP annotation service handles the full tooling, sub-dialect routing, QA, and adjudication stack — teams do not need to configure this infrastructure independently.

Related Reading

• Yemeni Arabic Sentiment Analysis: What Models Get Wrong Without Native Annotators
• Gulf (Khaleeji) Arabic Chatbot Intent: What Models Get Wrong
• Arabic Sentiment Analysis: The Complete Guide for MENA AI Teams
• Arabic NLP Annotation Service

Frequently Asked Questions