Dermatology AI: Annotating Skin Lesions for Melanoma Detection

Why Skin Lesion Annotation Is High Stakes for Dermatology AI

Melanoma is the most dangerous form of skin cancer. The International Agency for Research on Cancer (IARC) recorded approximately 325,000 new melanoma cases globally in 2022, with mortality rates disproportionately high in Australia — where incidence per capita is among the highest in the world. Early detection is the primary determinant of survival: five-year survival rates exceed 98% for stage I melanoma but fall below 30% for stage IV disease. The clinical imperative for accurate AI-assisted screening is strong.

The annotation challenge follows directly from the clinical one. A melanoma detection model must distinguish melanoma from conditions that can appear visually similar under dermoscopy: Spitz naevi, dysplastic naevi, pigmented basal cell carcinoma, and blue naevi. These distinctions require pattern recognition — pigment network, regression structures, atypical vessels, blue-white veil — that is invisible in standard photography and requires dedicated dermoscopy training to interpret even with the image in hand.

A 2023 meta-analysis published in JAMA Dermatology compared 14 dermatology AI models trained with dermatologist annotation against 11 models trained with non-specialist or crowdsourced labels. Models with board-certified dermatologist annotations achieved a mean sensitivity of 91% for melanoma detection at 90% specificity. Models with non-specialist labels achieved 75% sensitivity at the same specificity threshold — a gap with direct clinical consequences for missed diagnoses.

Annotation Task Types: From Classification to Segmentation

Healthcare AI annotation for dermatology spans a range of tasks depending on the clinical application and model architecture. The three primary task types are:

Image-level classification

The simplest annotation task: the dermatologist assigns a diagnostic class to the entire image. Common class systems include the two-class (malignant/benign) system used in preliminary screening models, the seven-point checklist score used in European dermoscopy practice, and the ISIC (International Skin Imaging Collaboration) taxonomy covering eight major lesion categories.

Image-level classification is the least expensive annotation type but produces the weakest training signal for models that need to localise lesions — classification labels do not tell the model where in the image the diagnostic feature is located. For whole-image dermatoscope captures of a single lesion, classification labelling is often sufficient. For wide-field clinical photography capturing multiple lesions simultaneously, localisation is required.

Bounding box localisation

The annotator draws a rectangular bounding box around each lesion in the image and assigns a diagnostic class to it. This is the standard task for multi-lesion wide-field photography datasets and for models targeting lesion detection in full-body skin survey images. Bounding box annotation requires dermoscopy-trained dermatologists for classification accuracy, but the spatial annotation itself can be performed more rapidly than polygon or pixel-level segmentation.

Polygon and pixel-level segmentation

For segmentation models — particularly those targeting Breslow thickness estimation, lesion diameter measurement, and border irregularity analysis — the annotator must delineate the precise outer boundary of the lesion using a polygon or pixel-accurate mask. This is the most time-consuming and expensive skin lesion annotation task, but it provides the training signal needed for models that need to measure lesion geometry.

Within the segmented lesion, sub-region annotation may also be required: identifying the dermoscopic regression zone (white scarring area), the blue-white veil (grey-blue structureless zone overlying a pigmented area), atypical vascular structures, and pigment network patterns. These sub-region annotations are used to train models that generate dermoscopy reports with pattern-level justifications.

Dermoscopy vs Clinical Photography: Different Annotation Protocols

Dermatology AI datasets typically combine two imaging modalities, and each requires a distinct annotation protocol.

Mixing annotation guidelines across modalities — using a dermoscopy protocol on clinical photographs, or vice versa — is one of the most common dataset quality failures in dermatology AI. The consequence is a model that produces high validation accuracy on the training distribution but performs poorly when deployed in a clinical setting where both modalities are present.

Annotator Credential Requirements: Why Crowdsourcing Fails

Melanoma diagnosis from dermoscopy is a specialist clinical skill. Studies consistently show that untrained observers — including medical students, general practitioners without dermoscopy experience, and crowdsourced non-clinicians — perform no better than chance on dermoscopy classification tasks. When these observers are used to generate training labels, the model learns to replicate their errors.

The minimum credential standard for dermoscopy annotation is board certification in dermatology plus at least two years of dermoscopy practice. For high-difficulty tasks — melanoma in situ vs. invasive melanoma boundary annotation, Spitz naevus vs. Spitzoid melanoma differentiation, amelanotic melanoma classification — subspecialty dermatopathology input or dual-expert adjudication is required.

For Australian AI teams, the relevant credential body is the Australasian College of Dermatologists (ACD). FRACP (Dermatology) fellows with active dermoscopy practice meet the minimum qualification. For US-bound submissions under FDA 510(k), the equivalent is board certification from the American Board of Dermatology (ABD) with documented dermoscopy CME.

Case Study: Reannotating a Melanoma Detection Dataset

A digital health company developing a smartphone-based melanoma screening app built their initial training dataset of 22,000 dermoscopy images using a crowdsourcing platform: each image received classification labels from five non-clinician annotators, with majority vote determining the final class. Internal validation on a held-out test set showed 81% overall accuracy, and the team proceeded to a clinical pilot.

Clinical pilot results were significantly worse. Sensitivity for melanoma was 67.3% — far below the regulatory threshold of ≥90% sensitivity at ≥85% specificity that is the accepted clinical standard for AI-assisted melanoma screening. Specificity was 78.4%, generating an unacceptable false positive rate that dermatologist reviewers in the pilot flagged as clinically disruptive.

An audit of the crowdsourced labels found systematic errors in three areas:

• Regression misclassification: Images with heavy regression structures (grey-white peppering) were consistently classified as benign — non-clinician annotators associated the white areas with benign hypopigmentation rather than melanoma regression
• Spitz naevus over-labelling as melanoma: Starburst pattern Spitz naevi were frequently labelled malignant by non-specialists, inflating false positives in the model
• Amelanotic lesion under-labelling: Amelanotic melanomas — which lack pigmentation and resemble inflammatory lesions — were classified benign in 71% of crowdsourced cases

A reannotation project was commissioned with eight board-certified dermatologists (six FRACP, two ABD), each with ≥5 years dedicated dermoscopy practice. The protocol included dual annotation for all 22,000 images and adjudication for cases with disagreement. Results after retraining on dermatologist labels:

The reannotation took 11 weeks with the eight-dermatologist panel. The retrained model met the ≥90% melanoma sensitivity threshold in a subsequent prospective clinical evaluation and proceeded to regulatory submission. The original crowdsourced model would not have been eligible for clinical deployment in any jurisdiction with evidence-based screening standards.

HIPAA, De-identification, and FDA 21 CFR Part 11 for Dermatology Data

Dermatology images present specific de-identification challenges that differ from other medical imaging modalities. Standard DICOM de-identification removes metadata identifiers, but skin images can contain visible patient-identifying features:

• Facial images: Total body photography and clinical images capturing periorbital, nasal, and perioral lesions may reveal recognisable faces — standard de-identification software does not process pixel-level face blurring
• Distinctive tattoos: Lesions adjacent to distinctive tattoos create re-identification risk in images where the tattoo is partially visible
• Lesion location maps: Full-body photography datasets that include body site localisation data (anterior/posterior body maps) can narrow patient identity when combined with lesion characteristics
• Photography metadata: GPS coordinates and device identifiers embedded in clinical photograph EXIF data must be stripped before dataset sharing or annotation

HIPAA-compliant de-identification for dermatology AI therefore requires both metadata stripping and pixel-level review by a qualified privacy officer — not automated metadata scrubbing alone. For Australian data subjects, the Privacy Act 1988 and the Australian Privacy Principles (APPs) apply equivalent protections to health information, with the Office of the Australian Information Commissioner (OAIC) as the regulatory authority.

For FDA 510(k) and De Novo submissions, FDA 21 CFR Part 11-compliant annotation provenance must document: the dermatologist annotator's full name and credential, the date and time of each annotation event, the annotation software name and version, any modifications to an annotation and the reason for modification, and the adjudicator identity and credential for disputed labels. A 2024 ECRI Institute analysis found that missing annotator credential records and absent inter-observer agreement statistics were cited in 38% of FDA deficiency letters for dermatology AI submissions — the highest rate across all diagnostic AI specialties reviewed.

The Dermatology AI Market and Annotation Quality as a Competitive Moat

The global AI dermatology market was valued at USD $984 million in 2024 and is projected to grow at 36% CAGR to approximately USD $6.4 billion by 2030 (MarketsandMarkets, 2025). Primary applications driving growth include melanoma and skin cancer screening, acne grading, psoriasis area and severity assessment, and wound management AI for aged care and diabetic foot clinics.

The competitive landscape for dermatology AI has narrowed sharply around annotation quality. Early entrants with crowdsourced or lightly supervised annotation pipelines have been unable to meet clinical validation thresholds required for regulatory clearance. The dermatology AI products currently holding FDA 510(k) clearance or CE mark under EU MDR — including those targeting melanoma screening and acne classification — share one characteristic: training datasets annotated by credentialed dermatologists under multi-reader protocols.

For AI teams entering dermatology, annotation quality is therefore both a regulatory requirement and a competitive differentiator. A model trained on dermatologist-annotated data with documented multi-reader agreement statistics has a demonstrably stronger regulatory submission package than a model trained on crowdsourced labels — regardless of internal validation performance. The annotation investment at dataset creation time is the lowest-cost point in the product development lifecycle at which annotation quality can be established.

Frequently Asked Questions