Hebrew LLM Training Data: SFT, Preference and Evaluation Data

Why Hebrew Needs Its Own LLM Training Data

Modern Hebrew has approximately 9 million native and near-native speakers, the vast majority in Israel. That makes it a relatively small language by global standards, but Israeli AI investment is disproportionately large — Israel's AI startup ecosystem is one of the most active per capita in the world, with significant activity in defence, fintech, medtech, and enterprise software.

Despite this, Hebrew is significantly underrepresented in the training data of major foundation models. Web-crawled corpora such as Common Crawl contain roughly 0.5–1% Hebrew content by token volume. When those base models are fine-tuned for Israeli-market products using machine-translated English data, the results underperform native-trained models in ways that Hebrew speakers immediately notice.

AlephBERT (Bar et al., 2021), the dedicated Hebrew BERT model, demonstrated that a monolingual Hebrew model trained on natively-produced web text significantly outperforms multilingual BERT on Hebrew NLP tasks including named entity recognition, morphological disambiguation, and dependency parsing. The same principle applies to instruction-tuned models: natively-written SFT data outperforms translated data on Hebrew task benchmarks.

The Three Structural Challenges for Hebrew LLM Data

1. Unvocalised script and morphological ambiguity

Standard Modern Hebrew text is written without niqqud — the vowel diacritical marks that appear in children's books, liturgical texts, and poetry. For a proficient adult reader, this is not a problem: context resolves ambiguity. For a language model processing text character by character, the same written string can correspond to multiple different Hebrew words with different pronunciations and meanings.

The Hebrew NLP research community has quantified this: a given unvocalised word form in Hebrew news text has on average 2.7 possible morphological analyses (Adler and Elhadad, 2006). For SFT data, this means annotators writing responses must be sensitive to context — their outputs must be unambiguous in the reading context provided, not technically correct but potentially misread. A QA step that tests whether a second native reader parses the response as intended catches these issues before training.

2. Root-and-pattern morphology across binyanim

Hebrew verbs are conjugated through seven binyanim — conjugation patterns (pa'al, nif'al, pi'el, pu'al, hitpa'el, hif'il, huf'al) that encode voice, intensity, and reflexivity. A single three-letter root generates different verbs across different binyanim, and selecting the wrong one is a systematic error in SFT data from non-native or semi-native contributors.

This is not simply an issue of grammatical correctness — it is a fluency signal. An LLM that uses Pi'el forms in contexts that require Pa'al sounds stilted to native speakers even when the meaning is technically conveyed. Annotators writing SFT demonstrations need to be screened on binyan selection accuracy, not just general Hebrew fluency.

3. Register: formal vs. colloquial Modern Hebrew

Modern Hebrew has two active registers: formal written Hebrew (closer to Classical Hebrew in vocabulary and syntax) and colloquial spoken Hebrew (with significant lexical borrowings from Yiddish, Arabic, Russian, and English). Many Israeli technology companies want their AI assistants to communicate in the colloquial register — but this is underrepresented in formal web text and may not appear in base model training data.

Before collecting SFT data, specify the target register explicitly. A legal document assistant should use formal Hebrew; a consumer chatbot for a retail brand might target colloquial Hebrew. Mixing the two registers in training data confuses alignment and produces a model that inconsistently switches between styles.

Supervised Fine-Tuning Data for Hebrew Models

Hebrew SFT data collection follows the same high-level process as any LLM fine-tuning project: define a task domain, create a contributor prompt set, screen and train native Hebrew contributors, collect responses, and QA.

For Hebrew, contributor screening should include a morphological challenge — not a grammar test, but a writing task that produces binyan and verb-form selections in ambiguous contexts. Standard fluency tests pass contributors who speak Hebrew natively but may still have inconsistent written register or non-standard morphological preferences. Screened contributors who write consistently correct formal or colloquial Hebrew are a smaller, higher-quality pool.

Our Hebrew data annotation and SFT service maintains a pool of screened native Hebrew contributors across formal, colloquial, legal, medical, and technical domains. Projects typically begin with a scoped pilot of 200–500 instruction-response pairs to validate contributor quality and task specifications before full-scale collection.

RLHF and Preference Data for Hebrew Models

Hebrew preference data collection presents the same structural challenges as SFT data, with an additional layer: native-speaker reviewers must be calibrated on the specific quality dimensions that matter for the target product, including register-appropriateness.

A critical issue in Hebrew preference collection is inter-annotator agreement on register. When two native Hebrew speakers have different intuitions about whether a given colloquial form is appropriate for a customer service context, their preferences will conflict without shared training on the target register. Preference rubrics for Hebrew should make register a first-class dimension — not implicit in “naturalness” — and reviewer calibration should include register-specific example pairs.

For the broader principles of preference data design that apply to Hebrew and other languages, see our post on RLHF data collection for production LLMs. The Hebrew-specific register and morphology issues layer on top of those general design principles.

Hebrew LLM Evaluation: What Benchmarks Miss

Public Hebrew benchmarks — the AlephBERT evaluation tasks, XNLI Hebrew, and the Hebrew Universal Dependencies evaluation suite — measure general language understanding at the token and sentence level. They do not measure whether a model produces register-appropriate responses, whether its binyan selection is consistent with native speaker preferences, or whether its outputs are useful for the specific Israeli product domain.

For production Hebrew AI, the most important evaluation asset is a custom benchmark of 200–500 domain-specific Hebrew prompts with gold-standard responses, human-verified by native Hebrew speakers. This benchmark should include at least three types of adversarial cases: (1) prompts where the correct response requires formal register and the model is likely to produce colloquial, (2) prompts where morphological ambiguity in the question could be resolved in two ways, and (3) prompts where the correct verb binyan is non-default for the root used.

For multilingual teams who need evaluation across Hebrew and other languages, our post on why native-speaker raters change multilingual LLM results covers the evaluation design principles that apply broadly. For Hebrew-specific annotation services, see our full Hebrew data annotation service page.

How to Scope a Hebrew LLM Training Data Project

Key decisions before any Hebrew LLM data collection: (1) task domain, (2) target register (formal or colloquial), (3) whether niqqud is required for the specific use case, and (4) whether you need SFT only, SFT plus preference data, or the full alignment pipeline including evaluation.

Volume guidance: for fine-tuning a 7B–13B parameter base model on a narrowly defined Hebrew task, 5,000–15,000 high-quality SFT pairs is typically sufficient. Broader Hebrew instruction-following capability generally requires 30,000–80,000 pairs. Preference data volume is usually 30–50% of the SFT volume for the same domain coverage.

Quality control for Hebrew should include a minimum 10% expert review pass specifically checking binyan selection accuracy, register consistency, and that unvocalised forms are unambiguous in context. This review pass is the primary quality gate — automated Hebrew grammar tools do not reliably catch register errors or contextual morphological ambiguity.

For teams building multilingual AI that spans Hebrew alongside other low-resource languages, our multilingual annotation and localisation service provides native-speaker coverage across 120+ languages under a single vendor relationship. For understanding why translated English training data fails Hebrew models, see our guide on why translated training data fails.

Frequently Asked Questions