Chest X-ray Annotation: Labelling 14+ Pathologies for AI

The 14-Pathology Framework: Where It Comes From and Why It Matters

The standard 14-class schema for chest X-ray AI originated with the NIH ChestX-ray14 dataset (Wang et al., 2017), which labelled 112,120 frontal-view radiographs across 14 disease classes using NLP extraction from radiology reports. Stanford's CheXpert dataset (Irvin et al., 2019) extended this to 224,316 images and added a critical annotation innovation: the uncertainty label for findings that are possibly present but not confirmed. Together, these two datasets established the de facto labelling taxonomy used by most chest X-ray AI products in clinical deployment today.

The 14 standard pathology classes span a wide range of visual difficulty. Cardiomegaly can be annotated guideline-mechanically — the finding is positive when cardiothoracic ratio exceeds 0.5 on a PA view — while fibrosis, infiltration, and the distinction between pneumonia consolidation and atelectasis require radiologist expertise to classify consistently. A 2022 study published in Nature Medicine found that radiologist-level agreement on chest X-ray pathology classification varied from kappa 0.85 for cardiomegaly to kappa 0.47 for infiltration, confirming that the 14 classes cannot be treated as uniform in their annotation requirements.

Our chest X-ray annotation service structures the 14-class schema into three annotation tiers based on finding difficulty, with annotator qualification and QA intensity matched to each tier. This approach delivers the cost efficiency of trained-annotator throughput on guideline-specifiable findings while maintaining radiologist-grade accuracy on the ambiguous classes that drive model performance at clinical deployment thresholds.

The 14 Pathology Classes: Annotation Difficulty and Qualification Requirements

Understanding the annotation difficulty profile of each pathology class is essential for scoping a chest X-ray labelling project. The table below maps each class to its annotation complexity, typical inter-annotator agreement range, and minimum annotator qualification level.

The CheXpert Uncertainty Label: Handling What Radiologists Can't Confirm

Stanford's CheXpert dataset introduced a three-way annotation schema for each finding: positive (finding present), negative (finding absent), and uncertain (finding possibly present but not definitively confirmed on available imaging). The uncertainty label captures the clinical reality that chest X-ray interpretation frequently involves equivocal evidence — a subtle opacity that might be early infiltrate or might be overlapping vessels, a possible nodule that would require CT to confirm.

In production annotation workflows, the uncertain label must be operationalised through explicit annotation guidelines that specify which equivocal scenarios qualify as uncertain rather than positive or negative. Without these guidelines, annotators revert to binary decisions — typically marking equivocal findings as positive to avoid false negatives — which inflates positive rates on the difficult classes and produces training signal that misrepresents radiological uncertainty in the actual clinical population.

The uncertainty label also requires a training strategy decision before annotation begins: treating uncertain as positive, treating as negative, treating as a separate class, or using label-smoothing or probabilistic losses. This decision affects model calibration at the clinical deployment threshold and cannot be retrofitted once annotation is complete. Teams that annotate without resolving this upstream typically discover miscalibrated confidence scores at model evaluation — a problem that requires re-annotation of the uncertain-label subset, not just retraining.

Multi-Label Co-occurrence: The Annotation Complexity That Datasets Understate

Chest X-ray multi-label annotation is more complex than annotating 14 independent binary classifiers in parallel. Pathology co-occurrence creates annotation dependency — a patient with severe congestive heart failure may present with cardiomegaly, bilateral pleural effusions, and pulmonary oedema simultaneously, and the annotation guidelines must specify how overlapping findings interact at the bounding box level and whether severity grades are applied independently or as a composite picture.

Analysis of the CheXpert dataset found that 38.2% of positive studies contained two or more co-occurring positive findings. The most common co-occurring pairs were pleural effusion + cardiomegaly (appearing together in 21.7% of positive studies), and atelectasis + consolidation (14.3%). These co-occurrence patterns mean that annotation guidelines written for individual findings in isolation will produce systematic labelling errors when co-occurring findings interact — for example, when pleural effusion obscures the cardiac silhouette used for cardiothoracic ratio measurement.

Production annotation guidelines for multi-label chest X-ray datasets must address co-occurrence explicitly — specifying the priority order when findings compete for the same anatomical region, how to handle occluded landmarks, and whether bounding boxes for co-occurring findings may overlap or must remain spatially separate. These are annotation design decisions, not annotation execution questions, and they must be resolved before a single image is labelled.

Case Study: Multi-Label Chest X-ray Reannotation for a Clinical Decision Support Platform

A medical AI company developing a chest X-ray clinical decision support platform for Australian public hospitals engaged AI Taggers to reannotate an existing dataset after their model failed validation on three of the 14 pathology classes. The product was intended to automatically highlight findings for radiologist prioritisation — the AI would not make diagnostic decisions but would reorder the reporting worklist based on predicted findings.

The original dataset of 32,000 frontal chest radiographs had been annotated by a vendor using a single annotator tier for all 14 classes — trained non-specialists with medical image annotation experience but no radiology training. Model validation on a 4,000-image holdout set revealed three class failures: atelectasis AUC 0.63 (target: 0.80), pneumonia sensitivity 53.8% at 80% specificity (target: 75%), and infiltration AUC 0.59 (target: 0.75). Cardiomegaly, pleural effusion, and pneumothorax performance was within specification.

Root cause analysis: Review of 200 randomly sampled annotation disagreements revealed that the atelectasis failures were driven by two systematic annotation errors: (1) annotators were marking linear atelectasis (streaky opacity from incomplete lung expansion) as negative because the original guidelines only described lobar and segmental atelectasis patterns; and (2) atelectasis adjacent to pleural effusion — a common co-occurrence — was being incorrectly labelled as effusion-only when both findings were present. The pneumonia failures stemmed from annotators marking all areas of parenchymal opacity as consolidation regardless of clinical context clues in the image metadata.

Our approach: We rebuilt annotation guidelines for all three failing classes with board-certified radiologist input, producing expanded finding taxonomies that covered all subtypes (lobar, segmental, linear, and discoid atelectasis) with worked visual examples. Co-occurrence rules were defined explicitly — atelectasis adjacent to effusion receives two separate bounding boxes, with the atelectasis box drawn at the border of the collapsed lung rather than at the edge of the effusion. Two radiologists performed primary annotation on all 32,000 images for the three failing classes, with a third radiologist adjudicating disagreements on 8.3% of images. The four previously classification-only classes received trained annotator bounding box labelling under radiologist QA.

The atelectasis AUC improvement from 0.63 to 0.84 came almost entirely from guideline expansion to cover all atelectasis subtypes and the co-occurrence bounding box rules — the model had been learning to classify linear atelectasis as negative because the training data encoded that error. The pneumonia sensitivity improvement from 53.8% to 79.2% was achieved through radiologist primary annotation and the removal of context-dependent classification that had been introducing systematic false negatives on early pneumonia patterns. Mean IAA across all 14 classes improved from 0.58 to 0.76, crossing the 0.70 threshold required for the TGA documentation package. For ongoing monitoring annotation and dataset expansion programmes, our X-ray annotation service provides structured programmes covering all 14+ CheXpert-compatible pathology classes.

Regulatory Documentation for 14-Class Chest X-ray Datasets

FDA and TGA SaMD submissions for chest X-ray AI require annotation provenance documentation that covers more than a summary IAA score. The ground-truth methodology section of a 510(k) or De Novo submission must include: the annotation task specification for each finding class, the annotator qualification level and training documentation for each tier, per-class IAA statistics with statistical method identification, the adjudication process for disagreements, and the data management controls demonstrating annotation data integrity.

FDA 21 CFR Part 11 compliance requires electronic records and signatures — each annotation event must be timestamped, associated with an identified annotator, and stored in a system that prevents modification without audit trail. This requirement eliminates annotation platforms that store labels in editable spreadsheets or export-only workflows where the intermediate annotation history is not preserved. Our annotation operations are structured around Part 11-aligned workflows from project initiation, so the compliance documentation is generated as a byproduct of the annotation process rather than reconstructed afterward.

Australian TGA SaMD submissions under the Class IIa and IIb framework require equivalent ground-truth documentation standards, with particular attention to the post-market performance monitoring plan — which itself requires a pre-specified annotation protocol for labelling the real-world performance monitoring dataset. Teams that design annotation workflows for initial regulatory submission without considering the monitoring annotation protocol typically face a second annotation project design exercise six to twelve months later.

For broader radiology AI data programmes, see our post on how radiology annotation is done across imaging modalities, and the FDA 21 CFR Part 11 annotation documentation guide for the specific provenance fields required for SaMD submissions. For CT-based pulmonary AI — where chest X-ray findings escalate to CT confirmation — our lung nodule CT annotation guide covers the 3D multi-slice workflow that follows chest X-ray detection.

Frequently Asked Questions