MRI Annotation for Neuroradiology AI: Workflows That Pass Clinical Review

Why MRI Annotation Is the Hardest Task in Medical Imaging AI

MRI annotation demands a fundamentally different skill set from CT annotation. Where CT uses Hounsfield units — a standardised physical scale — MRI contrast is acquisition-parameter-dependent. The same brain tissue can appear hyperintense on T2 and hypointense on T1 in the same patient on the same scanner, with the exact signal depending on repetition time, echo time, inversion time, and field strength. There is no universal threshold annotators can apply. Boundary decisions require knowledge of what sequence shows which pathology and why — a distinction that separates annotation that passes neuroradiologist review from annotation that fails it.

The scale of the challenge is significant. The global neuroimaging AI market was valued at USD $1.2 billion in 2024 and is projected to grow at a 14.8% CAGR through 2030 (Grand View Research, 2024), driven by diagnostic AI tools for glioma grading, multiple sclerosis (MS) lesion monitoring, stroke triage, and dementia progression modelling. Each of these applications requires training data annotated to a different standard with different sequence requirements, and errors that would be acceptable in one domain are disqualifying in another.

Our MRI annotation service covers the four primary neuroradiology AI domains:

Multi-Sequence Consistency: The Annotation Problem Unique to MRI

The central annotation engineering challenge in brain MRI is maintaining structural consistency across four sequences that visualise the same anatomy differently. An annotation guideline that specifies “delineate the tumour on T1 post-contrast” sounds simple until an annotator encounters a case where the T1c enhancing region is smaller than the T2 signal abnormality but larger than the DWI restricted diffusion region. Which sequence defines the “true” tumour boundary? The answer depends on the clinical definition of the tumour subregion being annotated, not on the annotator's individual judgement — and if the guideline does not specify this, annotators will each answer the question differently.

The BraTS challenge resolved this by defining four tumour subregions with explicit sequence-based definitions: the enhancing tumour (ET) region is visible as hyperintense on T1c; the tumour core (TC) is ET plus necrotic non-enhancing core (T1c hypointense within the T2 signal abnormality); and the whole tumour (WT) is TC plus the peritumoral oedema region (T2/FLAIR hyperintense, T1c isointense). These definitions make the annotation protocol reproducible — each annotator applies the same sequence-specific rule rather than interpreting the overall appearance.

For MS lesion annotation, the MAGNIMS consensus criteria provide the equivalent protocol framework: T2 FLAIR hyperintense lesions at least 3 mm in minimum diameter are candidate lesions; periventricular lesions must contact the ventricular surface; and infratentorial lesions in the brainstem and cerebellum are classified separately from supratentorial white matter lesions. Without this level of protocol specificity, inter-rater agreement on MS lesion burden in a 400-case training dataset will vary by as much as 30–40% between annotators depending on their lesion-inclusion threshold, directly degrading model performance on lesion count and volume estimation.

Case Study: MS Lesion Segmentation AI for an Australian Neurology Network

An Australian neurology network covering four hospital sites was developing an AI tool for automated MS lesion volume measurement to replace time-consuming manual FLAIR segmentation for annual MRI monitoring. The initial model, trained on 650 cases annotated in-house by two radiology registrars, was achieving a mean Dice of 0.58 on an independent test set — well below the 0.75 threshold required for clinical monitoring equivalence with manual neuroradiologist measurement.

Error analysis identified three root causes. First, the two registrars were applying different minimum lesion size thresholds: one was annotating lesions as small as 2 mm, the other applying a 5 mm threshold — a four-fold difference in lesion count for patients with high lesion burden. Second, the protocol did not specify whether periventricular lesions were delineated including or excluding the ventricular wall partial-volume voxels, producing systematic volume discrepancies at the periventricular border. Third, infratentorial lesions (brainstem and cerebellum) were sometimes excluded and sometimes included depending on the annotator, with no guideline specification.

Our approach: We rebuilt the annotation protocol from the MAGNIMS 2016 consensus criteria before any production annotation began. The minimum lesion diameter was set at 3 mm. Periventricular lesion delineation was specified to exclude voxels with >50% partial-volume contribution from the ventricular wall (determined using a T1w reference). Infratentorial lesions were included as a separately labelled class, enabling the model to treat them as a distinct category rather than treating their presence or absence as noise. Six neuroradiologist-calibrated annotators completed a 90-case inter-rater calibration exercise before production, achieving a mean Dice of 0.82 across the calibration set — set as the minimum individual annotator quality threshold for production assignment.

The whole-lesion Dice improvement from 0.58 to 0.81 came primarily from the protocol-level fixes rather than annotator quality: standardising the minimum lesion size threshold eliminated the largest source of inter-rater variance. The periventricular volume error reduction from ±31% to ±8% came from the partial-volume exclusion rule, which removed the ambiguous boundary voxels that had been the main source of lesion volume discrepancy at the ventricular interface. On retraining with the corrected dataset, the deployed model achieved mean Dice of 0.79 on the independent test set — above the 0.75 clinical monitoring threshold — and the network proceeded to TGA submission. Our full MRI annotation service covers the full spectrum of neuroradiology AI annotation tasks.

Brain Tumour Annotation: BraTS Protocol and Subregion Standards

The BraTS benchmark dataset and challenge have become the de facto standard for brain tumour segmentation AI evaluation. State-of-the-art models trained on BraTS 2023 data achieve Dice scores of approximately 0.91 for whole tumour, 0.85 for tumour core, and 0.82 for enhancing tumour — representing the ceiling for models trained on BraTS-protocol annotation. For production annotation intended for clinical AI development rather than academic benchmarking, the BraTS protocol provides the reference framework, but must be adapted for the specific clinical application.

The enhancing tumour subregion is the most clinically important and the hardest to annotate consistently. Gadolinium enhancement depends on blood-brain barrier breakdown — a biological process that varies with imaging timing (typically 5–20 minutes post-injection), patient blood pressure, and prior treatment (steroids reduce enhancement even in the presence of active tumour). An annotator who annotates a post-contrast T1 series without knowing the injection-to-acquisition time, or who encounters a case where dexamethasone has been recently administered, will produce an enhancing tumour volume that is technically correct for that acquisition but not comparable to other cases in the dataset.

For production tumour annotation datasets, case metadata — injection-to-acquisition time, steroid status, and prior treatment history — must be captured alongside the imaging data and incorporated into the annotation protocol as conditional rules. Cases with atypical enhancement patterns (T1c isointense tumour core, ring-enhancing metastases versus GBM necrosis) should be flagged for neuroradiologist primary annotation rather than proceeding through the standard trained-annotator workflow.

Regulatory Requirements for Neuroradiology AI Annotation Data

Brain MRI annotation for clinical AI development is subject to the same regulatory documentation requirements as all SaMD (Software as a Medical Device) training data. Under FDA 21 CFR Part 11, annotation records must be maintained as electronic records with audit trails that capture who annotated each case, when, which protocol version was active, what changes were made and by whom, and the final quality acceptance decision. A provenance log that answers “who annotated this, when, and under which guideline version” must be maintainable for each annotation in the training set.

For Australian TGA SaMD Class IIb submissions, the annotation provenance requirements align with FDA de Novo expectations under the IMDRF SaMD framework. Annotator credentialling records — professional qualifications, calibration exercise results, and per-annotator Dice performance on the calibration set — must be documented as part of the AI development evidence package. Post-market surveillance requirements for the deployed model trace back to the training data quality documentation, meaning inadequate provenance records at the annotation stage create a regulatory gap that cannot be closed retroactively.

For a detailed breakdown of the specific provenance log fields required for medical AI annotation, our guide to FDA 21 CFR Part 11 annotation documentation specifies the electronic record requirements that apply to brain MRI training data. For the broader medical imaging annotation context, our guide to radiology annotation for diagnostic AI covers CT, X-ray, and MRI annotation workflows across oncology, emergency radiology, and screening AI applications.

Cost and Throughput for Brain MRI Annotation

Brain MRI annotation throughput is highly task-dependent. A standard T2 FLAIR white matter lesion segmentation case with moderate lesion burden (10–20 lesions, typical MS monitoring scan) takes an experienced annotator approximately 20–35 minutes. A four-sequence glioblastoma case with full subregion annotation (ET, TC, WT, and necrotic core) takes 45–90 minutes per case at annotator level, plus 15–25 minutes of neuroradiologist review per case. Brain parcellation to 50+ structures takes 3–5 hours per scan even with atlas-guided semi-automated tools.

For related medical imaging annotation contexts, our post on organ segmentation annotation for surgical and radiotherapy AI covers the CT-based segmentation workflows and Dice/HD95 quality standards that apply to non-neuro anatomical structures. For annotation in the adjacent histopathology domain, our guide to whole-slide histopathology annotation workflows covers tile-level vs slide-level tasks and pathologist credentialling requirements. Our full radiology annotation service covers CT, MRI, and X-ray annotation across all clinical AI applications.

Frequently Asked Questions