RLHF vs DPO: Which Preference Data Should You Collect?

The Core Difference: Same Data, Different Pipelines

The debate between RLHF and DPO is primarily an engineering and training-pipeline question, not a data-collection question. From the annotation team's perspective, the task is identical: given a prompt, mark which of two model responses is better according to a set of quality criteria. The output of that task — a (prompt, chosen, rejected) triple — is the fundamental unit of preference data.

Where RLHF and DPO diverge is what happens to that data downstream. In Reinforcement Learning from Human Feedback (RLHF), the preference pairs train a separate reward model — a classifier that scores any response for a given prompt. That reward model then provides a scalar signal during PPO (Proximal Policy Optimisation) fine-tuning, allowing the LLM to explore response space and be shaped by reward signals rather than being directly pushed toward specific responses.

Direct Preference Optimisation (DPO), introduced by Rafailov et al. in 2023, skips the reward model. It reformulates the RLHF objective into a closed-form loss that operates directly on the preference pairs — pushing up the log-likelihood of chosen responses and down the log-likelihood of rejected responses relative to a reference model. There is no RL loop, no reward model inference, no PPO instability. The same preference data that would train a reward model is used directly to fine-tune the LLM.

What the Annotation Record Looks Like

A preference annotation record is structurally simple. Each record contains:

The preference_margin field — whether the annotator slightly preferred or strongly preferred the chosen response — is optional for basic DPO but valuable for both pipelines. Strong-margin pairs carry a cleaner signal; marginal pairs are more informative for reward model calibration in RLHF but can introduce noise in DPO.

Many teams also capture criteria_violated on rejected responses. This structured metadata serves two purposes: it lets you audit whether the rejected response is wrong for systematic reasons (factual errors, refusals, formatting failures) versus subjective reasons (tone, length), and it enables quality filtering before training — removing pairs where the "rejection" is based on annotator stylistic preference rather than a meaningful quality gap.

Dataset Size: How Much Do You Actually Need?

The most cited evidence on preference data scale comes from two directions. The Hugging Face Zephyr-7B experiment (2023) trained a 7B model using DPO on approximately 60,000 preference pairs from UltraFeedback and achieved MT-Bench scores that rivalled models trained with full PPO-based RLHF on much larger datasets. The result established that DPO can work well at moderate data volumes when annotation quality is controlled.

More surprisingly, the LIMA paper (Zhou et al., 2023) demonstrated that 1,000 carefully curated instruction-following examples could match the performance of models trained on 52,000 Alpaca-style examples. While LIMA was about supervised fine-tuning rather than preference data specifically, the finding applies: curation and annotation quality compound returns far beyond raw volume. 10,000 high-agreement preference pairs will outperform 100,000 pairs where annotators disagreed on 40% of labels.

For RLHF reward models, the calculus is somewhat different. The reward model benefits from larger datasets because it needs to generalise across the full response space the LLM will explore during PPO. Anthropic's Constitutional AI work (2022) trained reward models on approximately 150,000 human preference comparisons for its Helpful and Harmless criteria — a scale appropriate for a large-scale deployment where the reward model must handle diverse adversarial prompts. For domain-specific RLHF on a narrower task, 20,000–50,000 high-quality preference pairs typically suffice for reward model pretraining.

Why Annotation Quality Hits DPO Harder Than RLHF

Both pipelines are sensitive to annotation quality, but DPO is more directly exposed to individual-pair noise. In RLHF, the reward model is a trained classifier that implicitly averages over the entire preference dataset — a small percentage of mislabelled pairs gets averaged out during training and has limited impact on reward model predictions. The RL loop then provides a further noise buffer because the policy is shaped by reward trajectories, not individual decisions.

DPO has no such buffer. Each (chosen, rejected) pair directly enters the loss function. A preference pair where the "chosen" response is actually worse — because the annotator favoured length, confidence, or formatting over correctness — contributes a gradient step that pushes the model in the wrong direction. With 50,000 such pairs, the correct signal dominates. With 10,000, a 10% mislabel rate (1,000 incorrect pairs) can measurably degrade output quality on the evaluation dimensions that mattered to those pairs.

The practical consequence: DPO annotation guidelines need to be more specific, rubric-driven, and closely reviewed than reward model annotation. Define explicit criteria — factual correctness, task completion, instruction-following, safety — with worked examples of correct and incorrect preference judgements. Run a calibration phase with 200–300 pairs before production collection to measure inter-annotator agreement. Target Cohen's kappa ≥ 0.70 before scaling up.

When to Choose RLHF and When DPO Changes the Calculus

RLHF with PPO remains the choice when your task requires iterative online collection — generating new responses with the current policy, getting them rated, and updating the reward model and policy together. This is how OpenAI trained InstructGPT and how Anthropic trains Claude's helpfulness layer. The reward model can be queried at inference time, allowing human feedback to shape the distribution of responses dynamically. It is also more robust to label noise at scale.

DPO is the choice for teams that want a simpler training pipeline, have a fixed offline dataset of preference pairs, and are not running the RL infrastructure required for PPO. It is particularly strong for:

• Domain-specific fine-tuning where preference pairs can be collected once against a fixed base model
• Teams with ML engineering resources to collect data but not to maintain a stable PPO loop
• Smaller models (7B–13B) where PPO instability is a significant operational concern
• Projects where annotators can produce high-agreement labels — kappa ≥ 0.70 — making the direct-loss approach safe

The common middle path is to collect to DPO-quality standards from the start. This gives you data usable in either pipeline, preserves optionality, and enforces the annotation discipline that prevents sycophancy training — the failure mode where models learn to produce confident, well-formatted responses regardless of factual quality because annotators rewarded surface-level style.

Case Study: Switching From RLHF to DPO Mid-Project

In 2025, an Australian enterprise software team was building a domain-specific assistant for contract review. They began with a full RLHF pipeline: collecting preference pairs from in-house legal professionals, training a reward model on 35,000 pairs, and running PPO fine-tuning on a 13B model. The reward model performed well on their internal rubric — 84% accuracy on a held-out preference test set — but PPO fine-tuning introduced instability: the model began optimising for reward model score rather than actual contract review quality, producing verbose, hedge-heavy responses that scored well on their helpfulness reward dimension but frustrated legal staff in practice.

The team audited their preference data and found that annotators had been rating length and apparent thoroughness rather than legal accuracy. Pairs where the chosen response was longer but less precise outnumbered pairs where correctness drove preference by nearly 2:1 in the first 12,000 annotations, before rubric revision. The reward model had learned a length proxy for quality.

After rubric revision and a re-annotation pass on 8,000 ambiguous pairs, they switched to DPO on the revised 27,000-pair dataset (the 8,000 audited pairs plus their remaining clean collection). The DPO fine-tune ran in under 12 hours on a single 8xA100 node — compared with three days of iterative PPO runs. Legal staff rated the DPO-trained model 4.2/5 on a 100-case evaluation, compared with 3.1/5 for the PPO model at its best.

Total annotation budget was reduced by approximately 35% relative to the original RLHF plan — the iterative online collection cycles required for PPO were eliminated. The lesson: annotation quality and rubric precision matter more than pipeline architecture. The rubric revision that saved the project cost less than one week of annotation rework time.

The Sycophancy Problem and How to Prevent It

Sycophancy training is the most consistent failure mode in preference data collection for both pipelines. It occurs when annotators systematically prefer responses that are confident, verbose, or validating — regardless of factual content — because the surface properties of the response match their intuitions about "good answers." The model learns that long, assured responses earn higher preference ratings and optimises accordingly.

Preventing it requires annotation guidelines that explicitly decompose preference into measurable criteria:

Each criterion should have at least three worked examples in the annotation guidelines — two clear cases and one edge case — before annotators begin production work. Include "deceptive positive" examples: long, confident, well-formatted responses that are factually wrong, shown with the correct preference label (rejected), so annotators learn not to rate on style.

Practical Recommendation: Collect Once, Use in Both

The most cost-effective preference data strategy for production LLM teams is to collect to DPO-quality standards from the start — explicit rubrics, calibration phase, inter-annotator agreement target of 0.70+, preference margin metadata, and criteria-violation tagging on rejected responses. This data is fully compatible with DPO pipelines and provides a strong foundation for RLHF reward model training if you later scale up.

Sequence your collection in phases. Collect 1,000–2,000 pairs with a senior annotator cohort and measure agreement before scaling. If kappa falls below 0.65, revise the rubric before proceeding. After rubric stabilisation, scale to your target volume — typically 15,000–60,000 pairs for a production DPO fine-tune depending on task complexity and model size.

For domain-specific tasks — medical, legal, financial — ensure your annotator pool has genuine domain expertise. Our data collection service provides domain-routed annotator matching for preference annotation tasks, with calibration reporting and inter-annotator agreement measurement built into the project workflow.

Frequently Asked Questions