How Is Radiology Annotation Done for Diagnostic AI?

What Radiology Annotation Covers: Four Modalities, Four Workflows

The term "radiology annotation" covers four imaging modalities that share DICOM as a file format but differ substantially in data structure, clinical conventions, and annotation requirements. Treating them as a single workflow is the most common planning mistake teams make when scoping medical AI data programmes.

X-ray (plain radiograph) annotation is the highest-volume task in radiology AI. Each study is typically one to four 2D images. Annotation involves placing bounding boxes on pathological findings, classifying view position (PA, AP, lateral), grading finding severity, and labelling image quality. Chest radiographs — covering pneumonia, pneumothorax, pleural effusion, cardiomegaly, and consolidation — represent the majority of chest X-ray annotation volume. See our detailed guide on X-ray annotation services for task-specific requirements.

CT scan annotation is volumetric by nature — a chest CT contains 300–600 axial slices and requires 3D segmentation or systematic multi-slice labelling with Hounsfield windowing to reveal different tissue types. Annotation performed in the wrong HU window (e.g. lung window only for abdominal pathology) produces systematic label errors that survive spot checks and only become visible at model benchmarking.

MRI annotation must be applied independently across multiple pulse sequences from the same imaging session. A brain MRI typically includes T1-weighted, T2-weighted, FLAIR, DWI, and sometimes contrast-enhanced T1 sequences — each revealing different tissue characteristics. An annotator who labels lesions on T2 only and does not cross-reference T1 contrast and FLAIR has produced incomplete ground truth. Multi-sequence consistency checking is the most labour-intensive QA step in MRI annotation.

Ultrasound annotation presents different challenges: lower spatial resolution, speckle noise, machine-dependent image characteristics, and real-time artefacts from probe angle and patient breathing. Ultrasound clips require frame-level annotation for dynamic structures (cardiac function, foetal movement) alongside still-image annotation for structural finding classification.

The Radiology Annotation Workflow: Five Stages

Regardless of modality, production radiology annotation follows a five-stage workflow. Compressing stages to accelerate volume delivery is the most reliable predictor of dataset failure.

Radiologist vs Trained Annotator: Who Should Do What

The global AI in radiology market is projected to reach USD $4.5 billion by 2030 (MarketsandMarkets, 2024), driven by applications in lung cancer screening, cardiac risk stratification, and fracture detection. The annotation quality requirements for these high-stakes applications are correspondingly demanding — and the radiologist vs trained annotator decision determines both data quality and project economics.

Trained medical image annotators working under radiologist QA can reliably perform: presence/absence detection of high-contrast, high-prevalence findings (pneumothorax, large consolidation, obvious fracture); view classification and image quality assessment; anatomical landmark placement on well-defined structures; and bounding box localisation of large, clearly demarcated pathology.

Board-certified radiologists are required as primary annotators for: pathology characterisation and malignancy grading; subtle or low-contrast findings (ground-glass opacities, early infiltrates, microbleeds); rare pathology categories with few training examples; and any finding category where the AI model's intended clinical use involves independent diagnosis rather than triage support.

A 2022 study in JAMA Network Open found that AI models for chest X-ray pathology trained on radiologist-validated annotations produced AUC scores 12.4 percentage points higher than models trained on crowdsourced annotations from the same imaging dataset — a difference large enough to change the clinical utility of the system entirely. Our radiology annotation service structures workflows to use trained annotators where appropriate and radiologists where required, keeping costs proportionate to task difficulty without cutting corners on quality gates.

Regulatory Compliance: FDA, TGA, and HIPAA

Radiology AI submitted to the FDA as Software as a Medical Device must satisfy FDA 21 CFR Part 11 for electronic records and signatures, and must document annotation methodology, annotator qualifications, and inter-rater agreement statistics in the 510(k) or De Novo submission package. The specific documentation required by FDA's 2021 AI/ML SaMD guidance includes the ground-truth determination process, the reference standard annotator qualification criteria, and the statistical method used to measure inter-annotator reliability.

In Australia, the TGA applies broadly equivalent requirements under the Software as a Medical Device framework for Class IIb and III devices. TGA requires conformity assessment documentation including training data methodology — teams cannot submit a clinical performance summary without being able to produce the underlying annotation provenance if requested. Our post on FDA 21 CFR Part 11 annotation documentation details the specific audit trail fields required for regulatory submissions.

HIPAA applies when annotation projects involve identifiable US patient imaging data. Safe Harbour de-identification removes 18 specified identifiers from DICOM headers; Expert Determination de-identification requires a statistical certification that re-identification risk falls below a defined threshold. Australian Privacy Act obligations and state health records legislation apply to Australian patient data. Both regimes require data processing agreements (DPA/BAA) before annotation begins, and both prohibit annotation in environments where PHI is accessible to unauthorised personnel.

Case Study: Multi-Modal Chest Pathology AI for Australian Health Networks

A medtech company developing a multi-modal chest pathology AI system for deployment across regional and metropolitan Australian hospitals engaged AI Taggers for their annotation programme. The product combined chest X-ray triage (pneumothorax, consolidation, cardiomegaly, pleural effusion), CT lung nodule characterisation, and MRI cardiac function assessment — three separate models trained from a shared annotation programme.

Initial annotation had been attempted using a mixed non-specialist and crowdsourced workforce. Preliminary model benchmarking revealed unacceptable performance across all three modalities, and the client's regulatory submissions consultant identified missing inter-rater agreement documentation as a blocker for the planned TGA Class IIb application.

Root cause of the original quality problem: Non-specialist annotators had applied uniform bounding box conventions across all finding categories, without modality-specific windowing (CT) or multi-sequence cross-referencing (MRI). Chest X-ray inter-rater agreement across the eight finding categories averaged kappa = 0.48 — below the 0.60 threshold at which annotation is considered sufficiently consistent for AI training. Cardiac MRI biventricular segmentation showed Dice similarity coefficients averaging 0.71 — adequate for research but insufficient for the intended clinical application of automated ejection fraction measurement.

Our approach: We deployed a radiologist-led annotation team segmented by modality: four thoracic radiologists for chest X-ray and CT, two cardiologists for cardiac MRI. Each annotator received modality-specific calibration sessions using gold-standard reference cases before production annotation began. CT annotation used OHIF v3 with pre-configured window presets; MRI annotation used ITK-SNAP with a multi-sequence linked viewer ensuring T1, T2, and FLAIR were visible simultaneously. All annotation provenance was captured in Part 11-compliant audit logs with electronic signature controls.

The chest X-ray IAA improvement from kappa 0.48 to 0.81 was driven by replacing vague finding definitions with guideline specificity at the sub-category level — distinguishing lobar from patchy consolidation, grading pleural effusion by estimated volume, and defining the minimum bounding box margin for each finding class. The cardiac MRI Dice improvement from 0.71 to 0.91 came from cardiologist primary annotation with multi-sequence simultaneous viewing, which reduced contour errors at the trabeculated endocardial boundary — the highest-variance region in biventricular segmentation. The client proceeded to TGA application 16 months after dataset completion.

Radiology Annotation Cost by Modality and Task Type

Cost in radiology annotation is primarily driven by two factors: the complexity of the clinical task and whether radiologists must perform primary annotation or can review trained annotator output. The table below reflects production pricing for Australian AI teams; prices are in AUD and include radiologist QA at all levels.

For teams building multi-modality radiology AI, our radiology annotation service supports cross-modality programmes from a single engagement — reducing the coordination overhead of managing separate vendors per modality. Related specialised services include X-ray annotation for chest and orthopaedic radiograph programmes, and CT scan annotation for volumetric datasets. For teams earlier in the radiology AI data planning process, our radiology AI annotation guide covers multi-modality data strategy, and our post on CT scan annotation for radiology AI covers the CT-specific workflow in greater depth.

Frequently Asked Questions