Clinician Review for Medical LLMs: How Human Evaluation Works

Why Automated Benchmarks Fall Short in Medical AI

Medical LLMs are routinely evaluated on standardised benchmarks before deployment. MedQA (based on US Medical Licensing Examination questions), MedMCQA, and PubMedQA are the most commonly cited. Google's Med-PaLM 2 model achieved 86.5% accuracy on MedQA (Singhal et al., Nature, 2023) — a result that attracted substantial industry attention. Yet the same paper noted that physician review of the model's reasoning identified significant differences between benchmark accuracy and clinically sound reasoning in complex cases.

The gap between benchmark performance and clinical safety is structural, not incidental. Multiple-choice medical examination questions test recall of isolated facts. Clinical AI operates in open-ended, contextual environments where the failure modes are qualitatively different: a drug interaction omitted from a discharge summary, a symptom cluster that should trigger urgent referral but is instead reassured away, or a correct general statement applied to a specific contraindicated patient scenario.

Automated metrics such as ROUGE, BERTScore, and G-Eval score surface similarity to reference outputs. They measure whether the model's response resembles a good answer. They cannot measure whether the response is safe for a specific clinical context, whether a dangerous omission is present, or whether the reasoning path — even when it reaches a correct conclusion — would mislead a clinician who follows it.

What Clinician Review Actually Involves

Clinician review for medical LLM evaluation is a structured annotation task, not an informal expert opinion. Each reviewer assesses a model response against a defined prompt or patient scenario and records scores across multiple dimensions using a rubric. The dimensions vary by use case, but a standard set for a clinical decision support LLM includes:

• Factual accuracy — Is the clinical content correct according to current guidelines?
• Completeness — Are material facts, contraindications, or warnings missing?
• Contextual appropriateness — Is the response appropriate for the specific patient scenario, not just generically correct?
• Safety level — If acted upon, could this response cause patient harm? Scored on a defined harm severity scale.
• Referral appropriateness — Does the response correctly identify when a patient needs urgent or specialist referral?
• Uncertainty calibration — Does the model appropriately express uncertainty rather than stating uncertain information with false confidence?

Reviewers record dimension scores and a structured justification for each safety-level rating. Responses flagged above a defined harm threshold — typically any response where harm is “possible if acted upon” — trigger escalation to a second senior reviewer or clinical lead before the evaluation record is closed.

Clinician Reviewer Selection and Credentialing

The quality of medical LLM evaluation is bounded by the quality of the reviewers. Reviewers must hold current registration or licensure in the relevant specialty, be active practitioners (not recently retired — clinical guidelines update constantly), and complete a structured calibration exercise before production evaluation begins.

Calibration involves reviewing a set of pre-scored “gold standard” prompts with known correct evaluations. Reviewers who do not reach a minimum inter-rater agreement threshold — typically Cohen's kappa ≥ 0.70 on the safety dimension — receive additional calibration or are reassigned to tasks within their demonstrated competence. This step is often skipped by teams that treat clinician review as an informal quality check rather than a structured annotation process.

Specialty matching matters. A general practitioner reviewing an oncology LLM's responses about second-line chemotherapy regimens will miss nuances that an oncologist would catch. At the same time, over-specialising reviewers inflates cost and creates availability bottlenecks. A tiered approach works well: GP-level reviewers handle primary-care scenarios, specialist reviewers handle domain-specific cases, and a clinical lead handles escalated reviews and rubric disputes.

Safety Review and Escalation Protocols

Safety review is the component of medical LLM evaluation that most distinguishes it from conventional NLP annotation. Every response that a clinician reviewer scores as potentially harmful — even if factually accurate in isolation — must trigger a defined escalation path. A minimal escalation protocol includes:

Documentation at each escalation level must be granular enough to support regulatory review. The FDA's action plan for AI/ML-based Software as a Medical Device (SaMD), updated in 2023, requires that developers of adaptive AI systems maintain records demonstrating ongoing performance monitoring and safety event capture. Clinician review escalation logs are a core component of this documentation trail.

Case Study: When Benchmark Scores and Clinician Review Diverge

A healthcare AI team developing a discharge summary generation system evaluated their initial model using ROUGE-L scores against reference summaries from the same hospital system. The model achieved a ROUGE-L score of 0.71, placing it above the threshold for the team's internal quality gate.

Prior to a planned pilot deployment, the team commissioned a structured clinician review across 400 discharge summaries from the held-out test set. Three board-registered general physicians reviewed each summary using a rubric covering accuracy, completeness, safety level, and follow-up instruction clarity. Key findings:

• 21% of summaries had a completeness failure — a material medication, allergy, or follow-up instruction missing from the generated output that was present in the underlying record
• 8% of summaries contained a Level 1 or Level 2 safety issue — incorrect dosage rounding, a missing drug interaction warning, or an inappropriately reassuring statement about a symptom that warranted follow-up
• Inter-rater agreement (Cohen's kappa) on the safety dimension was 0.79, indicating strong reviewer consistency on what constituted a safety flag

The team identified that the ROUGE-L score was high because the model was accurately reproducing the fluent language of the reference summaries — but the reference summaries themselves had been written by junior residents and contained the same systematic completeness gaps. The model had learned to replicate human errors, not to correct them.

After three targeted fine-tuning rounds using clinician-annotated corrections — annotators flagged failures and wrote preferred alternatives for each — the safety issue rate dropped to 1.9% and the completeness failure rate to 6%. The system passed clinician review for the pilot deployment scope. Automated ROUGE-L had not moved significantly (0.68, within noise), confirming that automated metrics were orthogonal to the dimensions that mattered clinically.

Regulatory Alignment: FDA 21 CFR and TGA Considerations

Medical AI teams in Australia must consider both FDA alignment (for US market access) and TGA requirements under the Therapeutic Goods Act 1989. The TGA classifies AI/ML-based medical devices using the same risk-tier framework as the FDA's Software as a Medical Device (SaMD) guidance, and both agencies have signalled increasing scrutiny of AI systems that make or support clinical decisions.

For teams pursuing FDA clearance, two requirements from the 21 CFR Part 11 and SaMD Action Plan are directly relevant to clinician review:

• Audit trails — Electronic records of who reviewed which response, when, and what safety determination was recorded. Part 11 requires timestamped, tamper-evident logs for clinical systems.
• Performance monitoring documentation — Ongoing clinician review after deployment, not just pre-launch evaluation. The FDA's predetermined change control plan (PCCP) framework requires that developers document how they will monitor and validate AI performance changes after clearance.

As of January 2024, the FDA had cleared or approved more than 950 AI/ML-enabled medical devices (FDA, Artificial Intelligence and Machine Learning in Software as a Medical Device, 2024). The regulatory volume reflects a maturing market — and increasing regulatory capacity to scrutinise submitted evaluation documentation. Teams that treat clinician review as an informal check rather than a documented process are creating compliance risk they will pay to retrofit later.

For TGA submissions, the same principles apply under the TGA's Medical Device Software (MDSW) guidance, which aligns closely with FDA and IMDRF frameworks. MDSW classification depends on the intended purpose and the severity of harm from failure — higher-risk systems require more extensive clinical evidence and evaluation documentation.

Cost, Throughput, and Practical Scoping

Medical LLM evaluation with clinician review is more expensive than general-purpose human evaluation, and scoping it correctly matters. Key cost drivers:

• Specialty depth — Specialist reviewers (radiologists, oncologists, cardiologists) cost significantly more than GP-level reviewers per response
• Rubric complexity — Multi-dimension rubrics with structured justification fields take longer per response than binary pass/fail
• Escalation volume — High-risk prompts that trigger escalation to a second reviewer roughly double the per-response cost for those items
• Documentation requirements — 21 CFR Part 11-aligned logging and audit trail infrastructure adds overhead on top of reviewer time

For planning purposes, structured GP-level clinician review with a standard rubric typically runs AUD $2.50–$5.00 per evaluated response at production volumes. Specialist review runs AUD $6.00–$14.00 per response depending on the specialty. Evaluation throughput — responses reviewed per reviewer per day — is typically 80–120 for simple rubrics and 40–70 for complex multi-dimension rubrics with escalation.

A practical minimum for a pre-launch clinician review is 300–500 responses across a stratified sample of the deployment prompt distribution, including oversampling of high-risk scenarios (dosage queries, urgent symptom triage, contraindication scenarios). This provides enough statistical power to estimate safety issue rates with reasonable confidence intervals, and produces a documentation set that supports regulatory review.

Teams building medical LLMs should budget for ongoing clinician review after deployment, not just pre-launch evaluation. The LLM evaluation services that support this ongoing cadence — with structured sampling, IAA tracking, and escalation documentation — are qualitatively different from a one-off consultant engagement.

Frequently Asked Questions