Persian (Farsi) NLP Data Annotation: What Makes It Hard and How to Get It Right

Why Generic NLP Annotation Pipelines Fail on Persian

Persian is an Indo-Iranian language written in the Perso-Arabic script — a right-to-left abjad (consonant alphabet) that shares its script with Arabic but differs in character set, phonology, and grammar. Teams that annotate Arabic and then attempt to extend those workflows to Persian consistently encounter three failure modes that Arabic experience does not prepare them for.

The first failure mode is character normalisation. Persian and Arabic share most Unicode characters but have dialect-specific variants: the Arabic letter ي (U+064A) and the Persian letter ی (U+06CC) look visually identical in most fonts but are different Unicode code points. Iranian Persian keyboard layouts produce U+06CC; Arabic input methods produce U+064A. A dataset containing both variants silently doubles the vocabulary seen by a tokeniser, fragmenting embeddings and reducing model performance. The same issue affects ك (Arabic) vs ک (Persian, U+06A9), and several other character pairs.

The Iranian AI market is significant despite international sanctions: Iran ranks consistently in the global top 15 for AI research publications per capita (Scopus, 2024), with particular strength in Persian NLP through groups at Sharif University, Amirkabir University, and IPM. Annotation demand from diaspora-serving applications — particularly in North America and Europe — has grown substantially as Persian-speaking communities engage with digital services in their native language.

The Four Structural Challenges of Persian NLP Annotation

1. The ezafe construction: the unwritten syntactic connector

The ezafe (اضافه, meaning "addition") is the most distinctive feature of Persian syntax and the most dangerous one for annotation. It is a linking element — typically the vowel -e or -ye after vowel-final words — that connects a noun to a following modifier (adjective, genitive, or relative clause). Crucially, the ezafe is spoken but almost never written in standard modern Persian orthography.

Consider the phrase ketâb-e doktor-e Ahmadi (Doctor Ahmadi's book): in written Persian it appears as three separate orthographic tokens with no connecting markers. A NER annotator who does not know Persian cannot determine whether doktor-e Ahmadi is a possessive chain (Doctor Ahmadi) or two separate entities (a doctor and someone named Ahmadi). Native speakers resolve this through syntax and context; non-native annotators default to heuristics that produce 20–35% error rates on possessive NER chains in Persian text.

For dependency parsing annotation, the ezafe creates head-attachment ambiguities across multiple tokens. Published Persian dependency parsing benchmarks (Rasooli et al., 2013; updated in Sadeghipour et al., 2020 on the PerDT treebank) show that native-speaker annotators achieve attachment score IAA of 0.91 on ezafe chains versus 0.67 for intermediate learners on the same corpus — a gap of 24 percentage points driven almost entirely by ezafe head resolution.

2. Absent short vowels and orthographic ambiguity

Like Arabic, Persian orthography omits short vowels in everyday text. Unlike Arabic, Persian has fewer diacritic options for adding them when disambiguation is needed, and the practice of adding vowels (tashdid and fatha equivalents) is less common in modern Persian publishing than in, for example, Arabic religious or educational text.

The practical effect for NER annotation is that many Persian word forms are graphically ambiguous. The written form بار (bâr) can mean "load," "time" (as in "one time"), or "fruit." Without short vowel markers, only context determines the reading. For NER tasks, the word مهر (mehr) can be a person name, a month name (corresponding to September–October in the Iranian calendar), or the noun meaning "love." Annotators must resolve these from context; non-native annotators consistently over-annotate the ambiguous forms as named entities when they should be common nouns.

3. Morphologically rich verb system and argument structure

Persian verbs are formed by combining a prefix, a present or past root, tense/aspect/mood suffixes, and agreement suffixes that encode person and number. The result is a highly productive verb system where a single lemma generates dozens of surface forms. The verb didan (to see) generates mi-binam (I see), mi-bini (you see), nami-binad (he/she does not see), dide budam (I had seen), dide basham (if I have seen) — each a different orthographic form.

For intent and sentiment annotation, Persian light verb constructions add complexity. Persian forms many verb phrases with a light verb (kardan = to do, shodan = to become, dâdan = to give) combined with a noun or adjective. The phrase telefon kardan (to telephone) is a light verb construction where the semantic content is in the noun telefon, not in the verb kardan. Intent classification models trained on Persian text must annotate the full light verb phrase as the intent-bearing unit — annotators who parse at the verb alone will produce annotations that miss the semantic content.

4. Dialect variation and code-switching

Modern Persian exists in three main written varieties: Iranian Farsi, Afghan Dari, and Tajik (written in Cyrillic since the Soviet era). For annotation projects, Farsi and Dari share the Perso-Arabic script and are largely mutually intelligible, but differ in vocabulary (Dari retains more classical Persian words that Iranian Farsi has replaced with loanwords), some grammatical constructions, and orthographic conventions.

Iranian Persian also has significant code-switching with English in urban, educated, and tech-sector contexts. The phrase software-e mâ crash kard (our software crashed) is natural in Tehran tech culture — mixing Persian morphology with English technical vocabulary. Annotation guidelines must specify how to label English words that have received Persian inflectional suffixes, as these hybrid tokens appear consistently in conversational AI and customer service annotation corpora.

The Persian NLP Tooling Ecosystem

Persian NLP has a well-developed open-source tooling ecosystem centred on Iranian university research and diaspora-based open-source projects. The critical tools for annotation pipelines are:

Case Study: Iranian Fintech Platform — Customer Service Intent Recovery

In mid-2025, a fintech platform serving the Iranian diaspora in Europe and North America needed 60,000 annotated customer service messages for an intent classification model. Messages were written in informal Iranian Farsi, with heavy English code-switching in the technology and financial vocabulary domains.

The initial annotation run used a multilingual crowdsourcing platform. After 14,000 records, an internal Persian NLP engineer reviewed a validation sample and found:

• Intent accuracy of 68.3% on a 400-record gold standard evaluated by native Farsi speakers — against a 85% target
• 27% of light verb construction spans (e.g., پرداخت کردن, "to pay") were labelled on the verb alone, missing the semantically critical noun
• Arabic/Persian Unicode variants had not been normalised, causing 11.4% of records to have split-vocabulary tokenisation errors that no pre-annotation model could resolve correctly
• Code-switched English tokens with Persian suffixes (transfer کردم, "I transferred") were being assigned the "unknown" intent class at a 31% rate

The team rebuilt the annotation pipeline with native Iranian Farsi-speaking annotators and proper Persian pre-processing:

• PARSIVAR normalisation on all 60,000 messages to resolve Arabic/Persian Unicode character variants
• Hazm tokenisation with half-space (U+200C) preservation, followed by light verb compound detection to group light verb constructions as single annotation units
• Native Tehran Farsi-speaking annotators, with a calibration set of 150 code-switching sentences designed to test correct intent classification of hybrid Persian-English constructions
• Double annotation on 12% of records with kappa measurement per intent class
• Intent taxonomy reviewed and adjusted by the client's native Farsi-speaking product manager

Results on the re-annotated corpus:

The native-speaker annotation cost was 2.4× higher per record. The 14,000 crowd-annotated records were structurally compromised and could not be salvaged — they were discarded, representing a sunk cost that was comparable to the total premium for the remaining 46,000 records annotated with native speakers. The downstream intent classification model, trained on native-speaker data, achieved 87.2% accuracy on the production holdout set, enabling the platform to route customer service queries automatically with a false-routing rate below 3%.

Persian Annotation Guidelines: What Generic Templates Miss

Persian annotation guidelines adapted from English or Arabic templates routinely omit the language-specific instructions that prevent the most common errors. Critical inclusions are:

• Ezafe chain annotation rules: Define explicitly whether NER entity spans should cover the full ezafe-connected phrase or only the head noun. For person names in possessive chains (Doctor Ahmadi's report), the standard for most NER schemas is to annotate the full possessive phrase as a single person entity, which requires annotators to understand the invisible ezafe links. Illustrate with 10+ examples including multi-word ezafe chains.
• Light verb compound rules: Specify that light verb constructions (noun + kardan/shodan/dâdan) must be annotated as a unit, not at the verb alone. Provide a reference list of the 50 most common Persian light verb compounds in the relevant domain (customer service, medical, legal) so annotators can recognise them without translating each one.
• Unicode normalisation confirmation: Require annotators to confirm they are using a Persian (not Arabic) keyboard input method or that the source text has been pre-normalised, to prevent Arabic/Persian character variant inconsistencies entering the annotation output.
• Half-space (U+200C) handling: Specify the project's policy on half-space characters, which Persian uses to separate morphemes within words (e.g., می‌روم for "I go" with a half-space before the verb stem). Some annotation schemas count half-space-separated segments as one token; others treat them as two. This must be decided and documented before annotation begins.
• Code-switching classification: Specify whether English-origin words with Persian inflectional suffixes should be annotated as Persian tokens, English tokens, or a special code-switched class. The most practical policy for conversational AI annotation is to annotate by semantic content regardless of origin language — so transfer کردم annotated as a single payment-intent expression.

The Persian AI Market and Annotation Demand

Persian is spoken by approximately 110–130 million people globally when all three major varieties (Iranian Farsi, Afghan Dari, Tajik) and diaspora populations are included. Despite international sanctions limiting Iran's access to major commercial AI platforms, Iran's domestic AI research output is substantial — the country produced more AI research publications per capita than Australia in 2023 (Scopus, 2024), with strong output from Sharif University of Technology, Amirkabir University, and Isfahan University of Technology.

The international annotation demand for Persian comes from multiple directions: humanitarian and development organisations working in Afghanistan (Dari requirement), diaspora-serving fintech and e-government applications in North America, Germany, and Scandinavia (Iranian Farsi requirement), and multilingual AI product teams adding Persian to their language support rosters. The Afghan context adds urgency — post-2021, international development organisations have accelerated Dari NLP development for remote service delivery and digital literacy programmes.

Our multilingual annotation services include native Farsi and Dari-speaking annotators for NER, sentiment, intent, and morphological tasks, with the PARSIVAR and Hazm pre-processing pipeline included as standard. For teams building Arabic and Persian capabilities simultaneously, our Arabic data labeling services and Persian annotation share the same RTL QA protocols and can be coordinated in a single managed workflow. For AI teams requiring broad language coverage, our native-speaker annotation network covers Persian alongside 120+ languages with consistent quality standards.

Related Reading

If you are building Persian NLP annotation alongside other language requirements, these posts cover annotation challenges and strategies for related contexts:

• Where Do Arabic NLP Datasets Come From — and How Do You Build Your Own? — Parallel challenges of Arabic dataset sourcing for teams working across both Arabic and Persian
• Turkish NLP Data Annotation: What Makes It Hard and How to Get It Right — Agglutinative morphology and native-speaker annotation for a related Middle Eastern language context
• How Does Multilingual Annotation and Localization Work for Global AI? — Multi-language annotation workflows when Persian is one of several target languages

Frequently Asked Questions