Diabetic Retinopathy Grading: How Retinal Images Are Annotated for AI

Why Diabetic Retinopathy AI Is Worth Getting Right

Diabetic retinopathy (DR) is the leading cause of preventable blindness in working-age adults globally. According to the International Diabetes Federation (IDF Diabetes Atlas, 10th Edition, 2021), 537 million adults currently live with diabetes — a figure projected to reach 783 million by 2045. Roughly 35% of people with diabetes develop some degree of DR, and 10% develop vision-threatening disease. The global DR AI market is projected to reach USD $1.6 billion by 2030 (Grand View Research, 2023).

The clinical opportunity is real. Systematic fundus photography screening programmes catch DR at treatable stages, but the bottleneck in most health systems is the limited supply of ophthalmologists capable of grading large volumes of images. The IDx-DR system — the first autonomous AI diagnostic cleared by the FDA (De Novo, 2018) — demonstrated 87.2% sensitivity and 90.7% specificity for more-than-mild DR, enabling point-of-care screening without an ophthalmologist in the loop.

These outcomes depend entirely on the quality of the annotation that trained and validated the model. A mislabelled severity grade in the training set shifts the model's referral threshold — potentially causing it to under-refer patients with early proliferative DR, or over-refer patients with no clinically significant disease. Either error has direct patient consequences.

The ETDRS Grading Scale: What Annotators Are Actually Labelling

The Early Treatment Diabetic Retinopathy Study (ETDRS) severity scale is the clinical gold standard, with 13 severity levels from 10 (no DR) to 71 (high-risk proliferative DR). For AI training, most teams collapse these to five actionable classes:

• Grade 0 — No DR: No lesions detectable
• Grade 1 — Mild NPDR: Microaneurysms only
• Grade 2 — Moderate NPDR: More than just microaneurysms, less than severe NPDR
• Grade 3 — Severe NPDR: >20 haemorrhages per quadrant, venous beading, or IRMA
• Grade 4 — Proliferative DR: Neovascularisation or vitreous/pre-retinal haemorrhage

The referral threshold is clinically defined as "referable DR" — typically moderate NPDR or above, or any diabetic macular oedema (DMO). An AI model must learn exactly where this threshold lies, and that requires annotators who understand the clinical stakes of each grade boundary.

Beyond severity grading, many production datasets require lesion-level localisation: bounding boxes or pixel masks around individual microaneurysms, haemorrhages, cotton wool spots, hard exudates, and new vessels. This level of annotation is necessary for models trained to not only grade but to explain their decision — an FDA expectation for software intended to guide clinical action.

For expert retinal image tagging at the lesion level, the annotation workflow must specify exactly which field of view protocol is used (ETDRS 7-field, single-field 45°, or ultra-widefield), because lesion detection rates vary significantly across protocols.

Who Can Annotate Diabetic Retinopathy Images

The short answer: not crowdsourced workers. DR grading requires annotators with at minimum:

• Ophthalmology training or a formal retinal grader certification programme
• Demonstrated proficiency on a calibration set (typically ≥85% concordance with expert consensus)
• Documented training in the ETDRS protocol or the equivalent local standard (e.g., UK NHS Diabetic Eye Screening Programme)

A 2019 study published in JAMA Ophthalmology found that optometry-trained graders achieved 91.3% concordance with ophthalmologist grades on binary referable/non-referable classification, but dropped to 74.6% concordance on the five-class ETDRS scale — particularly at the mild/moderate NPDR boundary. This gap directly affects model calibration.

For FDA submission datasets, the standard practice is two independent graders per image with adjudication by a third senior grader (typically a retinal subspecialist) on cases where the first two disagree. Disagreement rates above 15% on severity class typically indicate a calibration problem requiring annotator retraining before continuing.

Multi-Grader Adjudication: Why It Matters and How to Set It Up

Single-grader annotation is not acceptable for clinical AI training datasets intended for regulatory submission — and it produces measurably noisier training signals even for internal research datasets. The standard workflow for production DR annotation is:

1. Independent dual grading: Two graders annotate each image without seeing each other's output
2. Automatic agreement check: Platform flags cases where grades differ by ≥1 severity class
3. Third-grader adjudication: A senior ophthalmologist reviews flagged cases and sets the consensus grade
4. Ground-truth finalisation: Consensus grade is locked with an immutable timestamp and grader IDs
5. Calibration monitoring: Monthly drift checks using a stable gold-set of 200–300 reference images

This workflow produces inter-rater reliability (Cohen's kappa) of κ = 0.78–0.85 on five-class ETDRS grading in well-managed programmes. Poorly run programmes — single grader, no calibration — typically achieve κ = 0.55–0.65, which is below the threshold considered adequate for clinical grade labels.

FDA 21 CFR Part 11 Requirements for DR Annotation Provenance

If the annotated dataset will be submitted to the FDA as part of a De Novo, PMA, or 510(k) application for a diabetic retinopathy AI system, the annotation platform must comply with 21 CFR Part 11 electronic records requirements:

• Each annotation event must be linked to a unique authenticated user ID
• Timestamps must be system-generated and tamper-evident
• Audit trail must capture the original grade, any modifications, and the reason for change
• Grader credentials must be documented and stored alongside the dataset manifest
• Final ground-truth labels must be locked and exportable in a format that maps directly to the FDA submission document

Most general-purpose annotation platforms (Label Studio, CVAT, Doccano) do not meet these requirements out of the box. Medical-grade annotation workflows typically require custom platform configuration or purpose-built clinical annotation tooling to satisfy these constraints. See our guide on FDA 21 CFR Part 11 for annotation provenance for a full checklist.

Case Study: Screening Programme AI — From 68% Grader Agreement to 94% Sensitivity

A teleophthalmology provider operating a national diabetic eye screening programme approached AI Taggers to build a training dataset for an autonomous DR triage system. The programme was screening approximately 180,000 patients per year across 340 remote sites, with images graded remotely by a pool of 24 trained graders — but grader concordance had drifted to 68% on referable-vs-non-referable classification due to inconsistent calibration.

Before: A baseline CNN trained on this internally graded dataset achieved 71% sensitivity and 81% specificity for referable DR on a 3,200-image validation set, well below the clinical target of ≥85% sensitivity.

The reannotation programme involved:

• Recruiting six ophthalmologists with retinal subspecialty training as primary graders
• Reprocessing 42,000 images with dual-grader ETDRS severity annotation and lesion-level bounding boxes
• Adjudicating 8,100 disagreement cases (19.3% of images) with a retinal subspecialist
• Implementing monthly gold-set calibration checks with ≥85% concordance as gating criterion
• Structuring the output dataset with full 21 CFR Part 11 audit trails

After: The retrained model achieved 94% sensitivity and 91% specificity for referable DR on the same validation set. Grader concordance stabilised at 91.3%. The AUC for five-class ETDRS severity improved from 0.81 to 0.97. The regulatory submission was approved 4 months after dataset finalisation without a major deficiency letter.

The primary driver of improvement was not model architecture or hyperparameter tuning — it was annotation quality. The same model family, retrained on the reannotated dataset, accounted for 94% of the total performance gain.

Annotation Cost Ranges for DR Datasets

DR annotation is among the more expensive medical imaging annotation tasks because it requires scarce clinical expertise. Realistic 2026 pricing for production-quality work:

• Single-grader ETDRS severity only (5-class): AUD $8–$18 per image
• Dual-grader ETDRS severity with adjudication: AUD $25–$55 per image
• Dual-grader severity + lesion bounding boxes: AUD $45–$95 per image
• Full lesion segmentation + severity + 21 CFR Part 11 provenance: AUD $80–$160 per image

For a 50,000-image training set with dual grading and adjudication (the minimum for a credible FDA submission), budget AUD $1.5–$3 million in annotation cost alone. This is a significant investment — but it is the only way to avoid the rework scenario described in the case study above, which cost 14 months of additional development time.

Related Medical Annotation Services and Guides

DR annotation sits within a broader ophthalmic and medical imaging annotation ecosystem. If you are building a retinal AI programme, these related resources are relevant:

• Retinal image tagging services — full-service ophthalmologist-supervised annotation
• Radiology annotation — multi-modality diagnostic AI datasets
• How is retinal image annotation used to detect eye disease with AI?
• Organ segmentation annotation for surgical and radiotherapy AI
• FDA 21 CFR Part 11 for annotation provenance

Frequently Asked Questions