Turkish LLM Training Data: SFT, RLHF and Evaluation for Turkish Models

Why Turkish Is Demanding for LLM Training Data

Turkish is spoken by approximately 90 million people globally, making it one of the world's 20 most-spoken languages by native speakers. Yet its representation in the training corpora of major foundation models is disproportionately low. Web-crawled datasets such as Common Crawl contain roughly 1–2% Turkish content by token volume, compared to over 45% English content — a gap that compounds at every stage of model training.

That data gap matters most when teams move from pre-training to post-training. Supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) require carefully crafted, domain-relevant data. When teams try to bridge the gap by machine-translating English SFT datasets into Turkish, they produce what linguists call translationese — text that is grammatical but unnatural, structured around English word order rather than Turkish suffix sequences.

Turkish builds meaning through agglutination: the word evlerinizden means “from your houses” and encodes what takes four English words in a single string. A model trained on translated English SFT data learns to answer in English-derived syntax dressed in Turkish vocabulary. Native speakers of Turkish immediately recognise this as unnatural — and it translates directly into lower task success rates in production.

Supervised Fine-Tuning Data for Turkish Models

SFT data for Turkish LLMs consists of instruction-response pairs in which a human writer demonstrates how the model should respond to a given prompt. The quality of this data — not just its volume — is the primary lever on downstream task performance.

A well-specified Turkish SFT project defines three things before collection begins: the task domain (customer service, legal, medical, e-commerce), the target register (formal siz-based address vs. informal sen-based), and the morphological correctness bar. Turkish has two parallel politeness registers that are not interchangeable — a customer service bot that mixes registers sounds amateurish to a native speaker.

Contributors should be screened on a morphologically complex writing task, not just fluency. The -mış evidentiality suffix (indicating reported or inferred information rather than direct observation), the subjunctive and conditional suffix stacking, and correct vowel harmony across long derivational chains are all points where non-native or semi-native contributors make consistent errors that standard fluency tests miss.

For production-grade Turkish data annotation and SFT collection, teams typically target 10,000–50,000 instruction-response pairs for initial fine-tuning, with ongoing collection batches to address coverage gaps identified during evaluation.

The Morphology Problem in Turkish SFT Data

Turkish morphology is the single biggest quality risk in LLM training data collection. The language attaches suffixes in strict sequences governed by vowel harmony — back vowels attract back-vowel suffixes, front vowels attract front-vowel suffixes — and violations are immediately perceptible to native speakers even when they are invisible to automated validators.

BERTurk, the Turkish BERT model published by Schweter (2020), demonstrated that a monolingual Turkish model trained on natively-produced text outperforms mBERT on Turkish NER by approximately 8–12 F1 points. This is consistent with the broader multilingual NLP finding that low-resource languages benefit disproportionately from native-speaker-produced training data versus multilingual models trained on translated or web-scraped text.

For SFT data, the implication is straightforward: quality control must include a native-speaker review step that specifically targets suffix-chain correctness and vowel harmony, not just semantic accuracy. An automated grammar checker will not catch these errors — they require human review.

RLHF and Preference Data for Turkish Models

Preference data for Turkish RLHF or DPO alignment involves presenting native-speaker reviewers with pairs of model responses to the same Turkish prompt and asking them to choose the better one, or rank them, according to a defined rubric.

Turkish preference collection has a specific challenge that English collection does not: the same content can be expressed in many different morphological forms, all grammatically correct but varying in formality, register, and naturalness. A preference rubric that evaluates helpfulness and accuracy without explicitly rating register-appropriateness will produce inconsistent preferences when different reviewers have different intuitions about which register is correct for the target use case.

Well-designed Turkish preference rubrics separate four dimensions: (1) factual accuracy, (2) morphological correctness, (3) register-appropriateness for the stated context, and (4) idiomatic naturalness. Reviewers need calibration training on dimensions 2–4 specifically, because these are the dimensions that differ from English and where inter-annotator agreement is lowest without training.

For more on preference data design, see our post on RLHF data collection for production LLMs. The Turkish-specific considerations layer on top of those general principles.

Turkish LLM Evaluation Datasets

Evaluation is where many Turkish LLM projects discover that their data collection did not achieve what they expected. Public Turkish benchmarks include the Turkish portion of XNLI (cross-lingual NLI), TQuAD (Turkish question answering on Wikipedia), and the Turkish subset of XTREME-R. These are useful baselines but they test general language understanding, not task performance in a specific domain.

For production Turkish AI, the critical evaluation assets are custom: a set of 200–500 domain-specific Turkish prompts with gold-standard responses, written by native speakers who understand the target task. These should include adversarial cases — prompts that exploit known Turkish morphological ambiguities or register confusion — and edge cases from real user query logs if available.

Evaluation rubrics for Turkish should include an explicit morphological correctness dimension alongside the standard helpfulness and accuracy dimensions. A model can pass a semantic evaluation while still producing suffix-chain errors that native speakers flag immediately.

How to Scope a Turkish LLM Training Data Project

The key decisions to make before any Turkish LLM data collection begins are: (1) task domain, (2) target register, (3) dialect (Istanbul standard is the default; regional varieties require specialist annotators), and (4) whether you need SFT only, SFT plus preference data, or the full alignment pipeline.

Volume guidance: for fine-tuning a 7B–13B parameter base model on a narrowly defined task (e.g., customer service for one product category), 5,000–15,000 high-quality Turkish SFT pairs is typically sufficient. For broader instruction-following capability, 30,000–100,000 pairs is a more typical range. Preference data volume is usually 30–50% of the SFT volume for the same domain coverage.

Quality control should include a minimum 10% native-speaker expert review pass on morphological correctness, with a structured rubric covering vowel harmony compliance, correct evidentiality marker usage, and register consistency. This review pass, not automated validation, is the primary quality gate.

For Turkish language AI projects, see our full Turkish data annotation service page for scope templates and pricing guidance. For general multilingual LLM evaluation practice, our post on why native-speaker raters change multilingual LLM results covers the evaluation design principles that apply to Turkish and other low-resource languages. For why translating English training data fails, see our forensic guide on why translated training data fails.

For multilingual teams working across Turkish and other Turkic or low-resource languages, our multilingual annotation and localisation service provides a single-vendor solution with native-speaker pools across 120+ languages.

Frequently Asked Questions