Instruction-Tuning Dataset Design: Coverage, Diversity and Quality

What Instruction Tuning Actually Does to a Model

A pretrained language model is a next-token predictor. Given a sequence of tokens, it estimates the probability distribution over what comes next. This capability is extraordinarily broad — the model has seen enormous amounts of human text and has implicitly encoded patterns from virtually every domain. What it lacks is the ability to consistently interpret a user instruction, identify the relevant task, and execute it in the expected format.

Instruction tuning — also called supervised fine-tuning (SFT) — bridges this gap. By training the model on a curated set of (instruction, ideal response) pairs, you teach it to recognise the instruction-following register and to produce responses that match the format and quality of the examples in your dataset. The model's underlying world knowledge from pretraining remains intact; SFT shapes how it presents and applies that knowledge.

The critical implication: instruction tuning does not add knowledge to the model. It shapes behaviour. If your SFT dataset has coverage gaps — task types the model will be asked to perform but that are absent from the training set — the model will default to its pretrained behaviour on those tasks, which is typically verbose, unpredictable, and poorly structured. If your dataset has quality problems — incorrect responses, inconsistent formats, ambiguous instructions — the model will learn those patterns too.

Coverage: Building a Task Taxonomy First

Coverage is the most commonly neglected dimension of SFT dataset design. Teams collect examples opportunistically — reusing existing Q&A data, scraping helpdesk transcripts, generating from ChatGPT — without first mapping the complete set of task types the model needs to handle. The result is a dataset with hundreds of examples of some task types and zero examples of others.

Start with a coverage taxonomy. For a general-purpose assistant, this might look like:

For a domain-specific model, replace or extend these categories with the specific tasks the model will face in production. A legal document AI needs: contract clause extraction, obligation identification, risk flag classification, plain-language summary, cross-reference resolution. A customer service model needs: intent classification, FAQ response, escalation identification, order status queries, complaint handling.

Assign a minimum example count to each leaf node of the taxonomy before collecting. Aim for at least 50–100 examples per distinct task type for rare types and 200–500 for common ones. The ratio of examples across task types should reflect the distribution of tasks in production, not the ease of data collection.

Diversity: Why Uniform Data Produces Brittle Models

A dataset can have correct coverage — examples of every required task type — and still produce a brittle model if the examples within each task type are too uniform. This is the diversity problem: the model learns the training distribution of instruction phrasings, input characteristics, and response formats rather than the underlying task logic.

Diversity failures manifest in predictable ways. A summarisation model trained only on news articles fails on legal documents because the sentence length, passive voice frequency, and terminological density differ too much from the training distribution. A Q&A model trained on short-answer examples fails on multi-part questions that require structured output because the format diverges from what it learned. An email-writing model trained on formal business correspondence produces oddly formal outputs for casual requests because no casual examples existed.

For each task type, explicitly vary:

• Input length: short prompts (one sentence), medium (one paragraph), long (multi-paragraph document)
• Instruction phrasing: direct commands, questions, polite requests, implicit tasks ("Here is a contract. What are the key obligations?")
• Audience level: technical, general, executive summary
• Output format: prose, bullet list, numbered list, table, JSON, code block
• Topic and domain: avoid over-indexing on the topics easiest to collect data for
• Difficulty and edge cases: include examples that require disambiguation, that have trick aspects, or that involve legitimate uncertainty

The most reliable way to enforce diversity is to track it during collection. Maintain a spreadsheet or tagging system that records the diversity dimensions of each example as it is written. Review the distribution before finalising the dataset — most teams discover significant concentration in a few combinations (long formal inputs, short bullet outputs) that needs rebalancing.

Quality: The Four Failure Modes in SFT Data

Quality problems in instruction-tuning datasets are remarkably consistent across teams and projects. Most production SFT quality failures trace to one of four root causes:

Case Study: Legal Document AI, 8,000 Designed vs 50,000 Collected

In 2025, an Australian legal-tech company was building an LLM assistant for contract review, aimed at paralegal teams reviewing commercial agreements. Their first attempt used 50,000 instruction-response pairs assembled from three sources: legal Q&A forums, annotated case summaries from a public dataset, and ChatGPT-generated responses to contract-analysis prompts. Fine-tuning a 13B base model on this dataset produced a model that evaluated well on general benchmarks but performed poorly in user testing — paralegals rated it 2.8/5 on a structured 100-contract evaluation, with particular failures in obligation identification and risk flag precision.

An audit of the 50,000-example dataset revealed:

• 38% of examples were Q&A pairs on general legal topics unrelated to commercial contracts
• Obligation identification — the most-used task — had only 340 examples despite being the primary use case
• GPT-generated responses contained 14.2% factual errors on an 80-example spot-check by a senior solicitor
• Response format varied across all task types with no consistency — bullet lists, numbered lists, prose, and JSON appearing randomly for identical task types

The team rebuilt the dataset from scratch using a coverage taxonomy with 14 contract task types, each with a defined response format and a minimum of 500 examples. A commercial law solicitor reviewed all responses for factual accuracy, and a senior paralegal reviewed format consistency. Total dataset size: 8,200 examples.

Fine-tuned on this 8,200-example dataset, the same 13B base model scored 4.4/5 in user testing — a 57% improvement in the paralegal rating. Obligation identification precision improved from 61% to 89% on a 200-contract held-out evaluation set. The total annotation cost for the curated dataset was AUD 41,000 — less than the cost of generating and cleaning the original 50,000-example corpus. The LIMA finding held in production: quality and coverage dominated scale by a substantial margin.

Synthetic Data: Where It Helps and Where It Fails

Synthetic instruction data — generated by a capable teacher model such as GPT-4 or Claude 3 rather than authored by humans — is a legitimate tool for expanding coverage at lower cost, particularly for rare task types where human-authored examples are expensive to produce. Microsoft's Phi series demonstrated that "textbook quality" synthetic data, carefully curated from a strong teacher model, could produce surprisingly capable small models at a fraction of the data-collection cost of comparable human-labelled datasets.

However, synthetic data has predictable failure modes that make it unsuitable as the primary source for domain-specific SFT datasets:

• Knowledge boundaries: teacher models have knowledge cutoffs and domain gaps. GPT-4 writing legal examples will produce plausible but sometimes incorrect content in specialist areas.
• Style homogeneity: synthetic datasets tend to be stylistically uniform because the teacher model has consistent stylistic tendencies. This limits the diversity dimension discussed above.
• Instruction phrasing uniformity: teacher-generated instructions cluster around common phrasings, reducing the model's ability to handle varied real-world instruction styles.
• Evaluation contamination: if your evaluation benchmarks were also influenced by the teacher model's outputs (a risk if you use publicly available benchmarks), synthetic data can produce inflated benchmark scores without corresponding real-world quality gains.

The practical recommendation: use synthetic data to expand coverage in task categories where human examples are scarce (rare instruction types, low-frequency edge cases), but always include human-authored examples for the primary task types the model will be evaluated on in production. A 70/30 or 80/20 human/synthetic split for the core tasks, with synthetic data used more liberally for peripheral coverage, is a reasonable starting point.

Quality Filtering Before Fine-Tuning

No matter how carefully designed the collection process, instruction-tuning datasets benefit from a final quality filtering pass before training. The key filters:

For production datasets, document the filtering pass: how many examples were removed at each stage, what the final per-category distribution looks like, and what quality metrics were measured. This documentation matters for model iteration — when you retrain on an updated dataset, knowing the quality baseline of the previous version helps you attribute changes in model behaviour to dataset changes rather than training configuration changes. Our custom annotation service includes dataset auditing and filtering as part of the SFT data delivery process.

Frequently Asked Questions