Radiology Report NLP: Annotating Findings, Impressions and Negation

Why Radiology NLP Annotation Is Its Own Discipline

Radiology reports are the most structured free-text documents in clinical medicine — they have predictable sections (Clinical History, Technique, Findings, Impression, Recommendation) — but within those sections the language is dense, highly abbreviated, and packed with clinical conventions that are invisible to annotators without radiology experience.

A sentence like 'No acute cardiopulmonary process identified' is a negative finding. 'Patchy ground-glass opacity in the right lower lobe, which could represent early pneumonia or aspiration' is a speculative finding with two competing differential diagnoses. 'Stable 1.4 cm nodule in the right lower lobe, unchanged compared with prior CT dated 14 March 2025' is a historical reference to a persistent finding, not a new abnormality. Each of these constructions requires a different annotation label — and each one will be misclassified by a non-radiologist annotator at a rate that renders the training data clinically useless.

A 2023 study published in the Journal of the American Medical Informatics Association (JAMIA) found that radiology NLP systems trained on annotation produced without clinical supervision misclassified negated findings at rates 3.1–4.7× higher than systems trained on radiologist-supervised annotation, on identical report corpora. The gap was widest on hedged findings ('cannot exclude', 'possible', 'suspicious but not diagnostic') — precisely the language radiologists use most frequently for clinically ambiguous cases.

The Three Annotation Tasks Radiology NLP Requires

Radiology NLP annotation is not a single task. It typically involves three separate annotation layers applied to the same report, each requiring a distinct schema:

1. Entity extraction from the Findings section

The Findings section describes imaging observations at the anatomical level. Annotation here labels clinical entities and their attributes: anatomical location (right lower lobe, hepatic segment VII, T4-5 intervertebral disc space), finding type (nodule, consolidation, effusion, fracture), morphological descriptors (spiculated, ground-glass, lytic), size or measurement, laterality, and — critically — the finding's epistemic status (present, absent/negated, speculative, historical).

Entity boundaries in radiology findings are complex. '1.2 cm speculated right lower lobe nodule' contains a single entity (pulmonary nodule) with four attributes (size 1.2 cm, morphology spiculated, location right lower lobe, modality-specific type). Whether those attributes are separate annotation spans or structured entity properties depends on the downstream model architecture — and the annotation schema must match the model's input format precisely.

2. Impression classification and entity extraction

The Impression section is the radiologist's synthesis and diagnostic recommendation. Annotation here operates at two levels: document-level classification (is the overall impression positive, negative, follow-up required, or emergent?) and entity-level extraction for ICD-10 coding or downstream structuring.

Impression-level annotation is where clinical document annotation expertise is most critical. Impression language uses clinical shorthand that changes meaning dramatically: 'Recommend CT pulmonary angiography to exclude PE' is a recommendation for further workup that implies a differential diagnosis (pulmonary embolism), not a confirmed PE. 'Findings are consistent with congestive heart failure' is a probable diagnosis. 'No acute abnormality identified' is a negative impression — yet it may follow a findings section describing chronic changes that require separate annotation.

3. Negation, uncertainty, and temporality annotation

This is the hardest annotation layer and the one where non-radiologist annotators fail most consistently. Every finding entity must be labelled with its assertion status:

• Present: the finding is confirmed as currently present ('There is a right pleural effusion')
• Absent: explicitly negated ('No pleural effusion is identified', 'The effusion has resolved')
• Speculative / uncertain: hedged ('possible early consolidation', 'cannot exclude pleural effusion on this limited study')
• Historical: referenced from a prior study, not currently observed ('Previously documented effusion is no longer present')
• Conditional: dependent on clinical context ('If clinically suspected PE, further workup is recommended')

The Radiological Society of North America (RSNA) has documented that clinical NLP tools processing radiology reports without negation-specific training data fail to correctly handle negation in 23–31% of findings statements in unselected report corpora. That failure rate means nearly one in three findings entities is labelled with the wrong assertion status — a dataset-level error rate that makes any downstream model trained on it clinically unreliable.

Case Study: Fixing a Chest CT NLP Pipeline With Radiologist Supervision

In late 2024, an oncology AI startup building a lung cancer risk stratification tool had accumulated 92,000 annotated chest CT radiology reports. The annotation had been performed over eight months by a team of general clinical NLP annotators — experienced in clinical text but without active radiology reading experience. The annotation schema included entity extraction and a binary positive/negative label for each finding, but negation had been handled via automated NegEx without human review.

An accuracy audit on a 600-report validation set reviewed by three board-certified thoracic radiologists revealed critical failures:

• Negation detection accuracy: 68.4% — one in three negated findings was labelled as present
• Speculative/uncertain finding classification: 41.2% accuracy — most hedged findings were classified as confirmed positives
• Historical finding classification: 55.7% accuracy — prior-study references were frequently annotated as current findings
• Entity boundary errors on complex anatomical entities: 22% of entities had incorrect span boundaries

The downstream lung cancer risk model trained on this data performed poorly on external validation: AUC 0.71 for malignant nodule prediction, sensitivity 74.2% at 80% specificity — substantially below the benchmark performance the team needed for clinical deployment.

The project was restructured over 11 weeks. All 92,000 reports were re-annotated by a team of eight thoracic radiologists using a revised schema that added assertion status (present / absent / speculative / historical / conditional) to every finding entity. A pre-annotation calibration session using 300 consensus reports established shared definitions. Disagreements between two independent radiologist reads (approximately 14% of findings entities) were adjudicated by a senior consultant thoracic radiologist.

Results after re-annotation: negation detection accuracy reached 94.3%, speculative finding classification reached 89.7%, and historical finding classification reached 91.2%. The downstream model retrained on the corrected dataset achieved AUC 0.91 for malignant nodule prediction and sensitivity 89.1% at 80% specificity on the same external validation cohort — a 19-percentage-point absolute improvement in AUC from annotation quality correction alone.

HIPAA De-identification Before Annotation Begins

Radiology reports contain Protected Health Information (PHI) across multiple categories: patient name (often in the Clinical History section), date of service, date of birth, accession number, referring physician name, facility name, and sometimes geographic identifiers or ages over 89. All 18 PHI categories under HIPAA's Safe Harbor method must be removed or generalised before reports are shared with annotation vendors.

De-identification should occur before annotation begins, not after. Annotators working on identified data create a separate breach exposure — even under a Business Associate Agreement (BAA) — because the data exists in their annotation environment in identifiable form. Safe Harbor de-identification for radiology reports typically includes:

• Replacing patient names with pseudonyms or placeholders in Clinical History and signature lines
• Shifting all dates by a consistent random offset (preserving temporal relationships while obscuring true dates)
• Removing accession numbers, MRN, and study UIDs
• Replacing facility names with generic identifiers (Site A, Site B)
• Generalising ages over 89 to '≥ 90'

Automated de-identification tools (Microsoft Presidio, AWS Comprehend Medical, ClinicalBERT-based NER models) achieve recall on common PHI categories but require human QA — particularly for radiology-specific identifiers and facility names embedded in protocol descriptions. A HIPAA de-identification audit after automated processing, before annotation, is the recommended practice.

Credentialing Requirements for Radiology Report NLP Annotators

The credential requirement for radiology NLP annotation depends on the annotation layer. For complex entity extraction and assertion-status annotation (present / absent / speculative / historical), board-certified radiologists or senior radiology residents under attending supervision are the minimum appropriate credential. General NLP annotators without radiology training should not perform these tasks — the case study above quantifies what happens when they do.

For ICD-10 coding tasks derived from radiology impressions, certified medical coders with radiology coding experience (AHIMA CCS with radiology specialisation) can perform the coding layer, but should not perform entity extraction or negation annotation. The two tasks require different expertise and should be annotated by different annotator pools.

Sub-specialty matching matters for complex modalities. Chest CT NLP annotation should use thoracic radiologists or general radiologists with active chest reading practice. Neuroradiology reports should use neuroradiologists; musculoskeletal reports should use MSK subspecialists. A general radiologist annotating a neuroradiology report will make systematic errors on anatomical location labelling and clinical hedging conventions that are subspecialty-specific. See also: our guidance on radiology annotation for imaging-level tasks.

Annotator qualification tasks — a set of pre-production reports with known labels — should be used to verify that each annotator meets the project's minimum agreement threshold before production annotation begins. Annotators falling below threshold on the qualification set should not proceed to production without targeted calibration and re-testing.

FDA 21 CFR Part 11 Provenance for Radiology NLP SaMD

If your radiology NLP model is classified as Software as a Medical Device (SaMD) — for example, a system that automatically codes ICD-10 diagnoses from radiology reports, or extracts structured findings for clinical decision support — FDA 21 CFR Part 11 applies to the annotation records used to train and validate it.

Part 11-compliant annotation provenance requires: a unique annotator identifier (not shared login) for every annotation record, a timestamped audit trail capturing every annotation and revision with the annotator's identity and timestamp, documented adjudication records naming the adjudicator and their credentials, and version-controlled annotation guidelines stored in a change-controlled document management system. The annotation platform must be validated to Part 11 requirements — which means most general-purpose open-source annotation tools require additional validation overhead before use in a Part 11 context.

FDA De Novo and 510(k) submissions for radiology NLP SaMD typically include inter-annotator agreement (IAA) statistics on the training data, the adjudication protocol and adjudicator credentials, and a description of the annotation schema and its clinical justification. Reviewers interpret high disagreement rates on the training annotation as a signal that the training labels are noisy — which raises questions about whether model test performance reflects true generalisation or label memorisation. For more detail on the inter-annotator agreement thresholds expected for clinical annotation, see our post on inter-rater reliability in medical annotation.

Building the Annotation Schema: Section-Specific Considerations

Radiology report annotation schemas must be designed section-by-section, not as a single flat entity schema. The Findings section and the Impression section require different entity types, different attribute sets, and different assertion-status vocabularies. Applying a single flat schema across both sections produces entity boundary errors and assertion misclassifications that compound as dataset size grows.

A minimal production schema for chest CT radiology NLP annotation includes:

• Entity types: Pathological Finding, Anatomical Location, Measurement, Morphological Descriptor, Temporal Marker, Laterality, Recommendation
• Assertion status: Present, Absent, Speculative, Historical, Conditional — applied to every finding entity, not just those with explicit negation markers
• Impression classification: Normal, Abnormal-Stable, Abnormal-New, Follow-up Required, Emergent/Critical, Incidental Finding
• Relation types: Finding-Location (links a pathological finding to its anatomical location), Finding-Measurement, Co-Reference (links a finding in Impressions to its corresponding Finding-section entity)

Co-reference annotation — linking an Impression-section diagnosis back to the Findings-section description that supports it — is frequently omitted in radiology NLP schemas but is essential for models that need to generate structured reports or support evidence retrieval. Without co-reference links, the NLP model cannot associate a diagnostic conclusion with the specific imaging observations that produced it.

Frequently Asked Questions