Code RLHF Data: Having Engineers Rank and Fix Model-Written Code

Why Code RLHF Is Different From General RLHF

In general-purpose RLHF, a well-calibrated generalist reviewer can assess whether a response is helpful, accurate, and well-written. These are properties a reader can evaluate without executing anything. Code is different: a response that looks well-formatted and confidently explained may contain a logic error visible only on execution, a subtle security vulnerability that requires domain knowledge to recognise, or an off-by-one bug that a surface reading will not catch.

A 2023 study published in arXiv by researchers at Stanford and DeepMind found that non-expert reviewers agreed with expert engineers on code correctness judgements only 58% of the time — barely above chance for binary classification. The error rate was highest for off-by-one errors (missed by non-experts 71% of the time), subtle logic errors (missed 64%), and type safety issues (missed 61%). Non-expert reviewers were reliable only for syntax errors and obvious non-completion.

This means non-engineer code RLHF data does not just underperform — it actively trains reward models toward the wrong signals, optimising for presentational code quality and confident commentary rather than functional correctness. The cost of using non-expert reviewers for code RLHF is not a small quality discount; it is a corrupted reward signal.

What Engineer Reviewers Evaluate in Code RLHF

The rubric for code RLHF preference tasks is more structured than general RLHF because code quality can be decomposed into assessable dimensions. A well-designed code RLHF rubric covers:

Designing Code Preference Pairs That Train Useful Reward Models

The most common mistake in code RLHF prompt design is using prompts that are too easy. When one model output is obviously correct and the other is obviously broken, the preference pair carries little training signal because the reward model has nothing to learn from clear-cut cases it will already predict correctly after initial training.

Informative code preference pairs are those where both responses are partially correct and the distinction requires genuine engineering judgement. Pairs where Response A handles the main case but misses edge cases and Response B handles edge cases but has unnecessary complexity teach the reward model the nuanced trade-offs that characterise senior engineering judgement.

Designing these prompts is a task that requires experienced engineers at the data collection design stage — not just at the reviewer stage. At AI Taggers, prompt design for code RLHF projects is done by a senior engineer who maps the model's known failure modes (from evaluation data) to a prompt taxonomy designed to surface those failure modes at collection time.

A Code RLHF Project: What Actually Happens

A representative code RLHF engagement for a mid-sized coding assistant team follows a consistent pattern. The team begins with an evaluation set of 200–300 hand-crafted prompts covering their highest-volume query types (typically code generation, debugging, refactoring, and code explanation). They run the current model against this set and have a small panel of senior engineers assess the outputs. This gives them a baseline: the proportion of correct, edge-case-handling, idiomatic responses the current model produces.

They then commission a code RLHF dataset starting from those known failure modes. Published research from CodeRL (Le et al., 2022) and subsequent work on execution-guided reinforcement learning shows that reward models trained with execution-verified preference data produce measurably larger gains in functional correctness than those trained on preference data alone — typically 8–15 percentage points on pass@1 benchmarks like HumanEval when the preference data specifically targets the model's current failure distribution.

After a round of 3,000–5,000 engineer preference pairs and reward model training, the team re-runs their evaluation set. Improvements cluster in the failure modes targeted by the preference data — correctness on edge cases, security handling, idiomatic output for the target languages. Generalisation to un-targeted failure modes is typically modest after a single round, which informs the next collection cycle.

Engineer Reviewer Vetting and Panel Management

Sourcing engineers for code RLHF is more difficult than sourcing medical or legal experts because there is no equivalent of AHPRA registration to verify. AI Taggers vets code RLHF reviewers through a four-stage process:

1. Portfolio and employment verification: Confirmed employment history in software engineering roles (not just coding education), with a sample of prior work reviewed for evidence of production-quality code.
2. Language proficiency screener: A timed coding screener for each nominated language, including a debugging task (identify and fix a seeded bug), an implementation task (write a function to spec), and a code review task (annotate a sample for issues). Screeners are calibrated against senior engineer gold standards.
3. RLHF task simulation: The candidate reviews 15 pre-judged code preference pairs. Agreement with the gold-standard judgements must exceed 75% to proceed to production.
4. Ongoing calibration: Production reviewers complete a calibration batch every two weeks. Those whose rolling agreement with the panel falls below 70% are paused and recalibrated.

This vetting process results in panel attrition of 30–50% at the screener stage. The remaining panel is smaller but consistently calibrated. For code RLHF, a well-calibrated panel of 30 engineers outperforms an un-vetted panel of 200 in data quality by a significant margin, based on the downstream reward model performance difference documented in published RLHF research.

Cost Drivers for Code RLHF Data

Code RLHF is more expensive per preference pair than general RLHF for two reasons: higher reviewer hourly rates (engineers earn more than generalists) and lower throughput (executing and evaluating code takes longer than reading text).

These ranges assume professional engineers with verified experience and proper quality controls. Volume discounts apply above 5,000 pairs. A minimum viable code RLHF pilot — 500 preference pairs across three languages — runs approximately AUD $8,000–$15,000 including task design and QA, and provides enough data to assess reward model responsiveness before committing to a full dataset.

Multilingual Code and Specialist Frameworks

Code RLHF for multilingual teams — where the code prompts and comments are in a language other than English — requires reviewers who are both engineering-proficient and native speakers of the prompt language. Arabic-language Python prompts, German-language SQL documentation, and French-language code comments all require engineers who can assess code quality and understand the stated requirements in context.

Specialist framework coverage (Terraform, Kubernetes YAML, dbt, or domain-specific languages) requires further reviewer matching beyond language. AI Taggers maintains a database of reviewer domain specialisations to match panel composition to your specific language and framework distribution.

Frequently Asked Questions