Ultrasound Annotation: The Hardest Modality to Label for AI

Why Ultrasound Is Harder to Annotate Than CT or MRI

CT and MRI produce images with consistent geometry and predictable signal characteristics. A lung nodule on CT looks like a lung nodule on CT — the Hounsfield unit range is defined, the anatomy is anatomically consistent, and the image quality varies within a narrow envelope across machines and acquisition protocols. Annotation guidelines written for one scanner generalise to another.

Ultrasound does not work this way. Speckle — the granular noise pattern inherent to coherent ultrasound — reduces tissue contrast and makes boundaries genuinely ambiguous. The same patient scanned by two different sonographers with different probes, probe angles, or focal zone settings can produce clips with visibly different image quality. A published review of point-of-care ultrasound (POCUS) lung assessment found inter-rater agreement across seven studies of κ = 0.51–0.74 for clinically meaningful findings — considerably lower than equivalent CT findings (Soni et al., Critical Care Medicine, 2015).

This variability is not a defect of the studies reviewed — it reflects the genuine difficulty of ultrasound interpretation. When you add annotation on top of interpretation, the challenge compounds. Annotators who cannot distinguish ultrasound artefacts from pathology will label shadow as tissue and reverb as lesions. These errors propagate directly into training data and degrade model performance in ways that are difficult to diagnose post hoc.

The consequence for annotation is that each ultrasound sub-specialty effectively requires its own annotator pool, credentialing standard, and quality protocol. AI Taggers' radiology annotation service addresses this by maintaining credentialed annotator pools across obstetric, cardiac, abdominal, and musculoskeletal ultrasound — each with sub-specialty QA workflows distinct from those used for CT and MRI work.

Credentialing Requirements by Sub-Specialty

The American Registry for Diagnostic Medical Sonography (ARDMS) credential system provides the clearest framework for understanding annotator qualification requirements. Each credential corresponds to a specific anatomy and clinical context — and annotation tasks should be matched to the relevant credential.

HIPAA De-identification: The Pixel Problem

Ultrasound has a de-identification challenge that cross-sectional modalities largely avoid: patient-identifiable information is often burned directly into the pixel data by the acquisition system. Patient name, date of birth, accession number, and institution name are rendered as text in the image corners — typically white text on a dark background — by the ultrasound machine's onboard software.

Standard DICOM header de-identification addresses the 64 mandatory tags specified in PS3.15 Annex E but does not touch pixel data. A compliant de-identification pipeline for ultrasound must include a separate step to detect and redact pixel-level PHI — either through OCR-based text localisation and masking or by applying a fixed cropping region that excludes header regions for known scanner models.

This matters practically because annotation vendors working from de-identified DICOM headers but not pixel data are routinely processing images that still contain PHI. Under HIPAA, this constitutes improper disclosure of protected health information. FDA 21 CFR Part 11-compliant annotation workflows must document the de-identification method, audit it, and maintain records of which de-identification approach was applied to each study.

Case Study: Fetal Biometry Annotation for an OB AI Programme

A maternal-foetal medicine AI company brought an obstetric ultrasound project to AI Taggers after their initial annotation round — performed by general medical annotators with no formal sonography credential — produced unacceptable measurement error rates.

The dataset comprised 22,000 cine clips from 14 clinical sites in Australia and New Zealand, covering first-trimester crown-rump length (CRL), second-trimester biparietal diameter (BPD), head circumference (HC), abdominal circumference (AC), and femur length (FL). Initial annotation by non-credentialed annotators yielded:

• CRL MAE: 4.2 mm (acceptable threshold: ≤1.5 mm)
• BPD MAE: 3.8 mm (acceptable threshold: ≤2.0 mm)
• Plane selection accuracy: 61% (annotators frequently labelled off-plane or suboptimal images as diagnostic quality)

AI Taggers reannotated the full dataset using ARDMS-credentialed RDMS-OB sonographers, with a two-tier QA protocol: independent annotation by two sonographers per clip, followed by blinded adjudication where measurements differed by more than 2 mm. Sonographers were recruited from perinatal centres in Australia to ensure familiarity with Australian clinical standards for plane selection.

Post-reannotation outcomes across the same dataset:

The client's gestational age estimation model — retrained on the reannotated data — improved AUC from 0.71 to 0.93, with mean gestational age prediction error reducing from 6.4 days to 2.1 days. The reannotation cost was approximately 18% of the initial annotation budget; the model performance lift justified it within one validation cycle.

Multi-Frame Annotation: Cine Clips and Temporal Consistency

Ultrasound is routinely acquired as cine clips (multi-frame sequences) rather than static images. Cardiac echo clips capture a full cardiac cycle. Fetal biometry clips span multiple breathing movements to identify the optimal measurement frame. POCUS clips capture real-time anatomical assessment over several seconds.

This creates an annotation challenge distinct from static CT or MRI slice annotation. The annotator must first identify the optimal frame or frames within the clip for a given measurement or label, then apply annotations consistently across selected frames. For cardiac segmentation, this means maintaining consistent chamber boundaries across systole and diastole — requiring temporal interpolation between keyframes.

Annotation tools must support frame-by-frame navigation, keyframe designation, and interpolation review. Annotation stored at the clip level without frame references is not useful for training temporal models. The output schema should specify: selected frame index, frame selection rationale (e.g., end-systole, end-diastole, standard plane achieved), measurement values in pixel coordinates with conversion to physical units where available, and annotator credential tier.

FDA 21 CFR Part 11 Documentation for Ultrasound AI

For ultrasound AI intended for clinical decision support — including foetal growth monitoring, cardiac function assessment, and POCUS-guided diagnosis — FDA 21 CFR Part 11 compliance requires that the annotation provenance record captures specific elements that most general-purpose annotation platforms do not log by default.

Required elements include: annotator identity and credential tier (ARDMS certificate number or equivalent), annotation timestamp (UTC), annotation tool and version, de-identification method applied to the source image, any adjudication record if multiple annotators reviewed the case, and the final accepted label with the rationale when conflicting annotations were resolved. Electronic signatures on completed annotation records must comply with 21 CFR Part 11.50 — handwritten signatures applied to printed PDFs do not meet the electronic record requirement.

Teams building ultrasound AI for regulatory submission should review these requirements before choosing an annotation platform or vendor. A platform that produces JSON label files without annotator identity linkage cannot produce compliant provenance records retroactively.

How to Scope an Ultrasound Annotation Project

Before approaching an annotation vendor for ultrasound work, document the following. Each item has direct implications for credentialing requirements, timeline, and cost.

1. Modality and sub-specialty: Which scanner type (linear, phased, convex, endocavitary) and which anatomical sub-specialty?
2. Task type: Classification, measurement, segmentation, or multi-frame tracking?
3. Frame format: Static frames or cine clips? What frame rate and clip duration?
4. De-identification status: Is pixel-level PHI present? Has header de-identification been applied? By what method?
5. Regulatory context: Is this FDA pre-submission, CE-mark, or research? Which compliance framework applies?
6. IAA target: What minimum inter-annotator agreement is required, and for which metrics?
7. Adjudication protocol: Single annotator, dual annotator with adjudication, or expert panel review?
8. Output format: DICOM SEG, JSON coordinates, CSV measurements, or platform-native export?

Vendors who do not ask these questions upfront are unlikely to have the sub-specialty annotator pools or compliance infrastructure needed for production ultrasound AI. Our ultrasound and radiology annotation service covers all major sub-specialties with ARDMS-credentialed annotators and FDA 21 CFR Part 11-compliant documentation. For comparison with other imaging annotation workflows, see our guides on CT lung nodule annotation and cardiac MRI annotation.

Frequently Asked Questions