Surgical and Endoscopy Video Annotation for Operating-Room AI

Why Surgical Video Annotation Cannot Be Crowdsourced

The operating room generates more video data than any other clinical setting. A single laparoscopic cholecystectomy produces 20–45 minutes of high-definition footage. A robotically-assisted radical prostatectomy captures 3–5 hours. Colonoscopy screening programmes generate millions of endoscopy minutes annually. The AI applications targeting this data — phase recognition for operating room workflow automation, polyp detection for colonoscopy, instrument tracking for robotic guidance, and critical view of safety verification — all require annotations that a general labeller cannot produce accurately.

The annotation challenge is domain expertise. Identifying that a laparoscopic cholecystectomy is in the "calot triangle dissection" phase rather than the "gallbladder dissection" phase requires recognising the anatomical context of the instrument movements — a distinction that takes surgeons years of training to make reliably and that non-clinicians cannot learn from an annotation guideline document alone. Similarly, labelling the "critical view of safety" — the specific configuration of dissected hepatocystic triangle and exposed lower third of gallbladder that must be achieved before clipping — requires understanding what an unsafe configuration looks like clinically, not just visually.

According to a 2024 systematic review in Surgical Endoscopy, surgical AI models trained with board-certified surgeon annotations achieved mean phase detection F1 scores of 0.88–0.93 across laparoscopic cholecystectomy and appendicectomy benchmarks. Models trained with non-surgeon annotations from a general annotation platform achieved F1 scores of 0.61–0.74 on the same benchmarks — a gap that represents the difference between a model useful in clinical practice and one that is not.

The Four Core Annotation Tasks in Surgical Video

Surgical video annotation encompasses four distinct task types, each requiring different annotation tools and credential levels. Most production surgical AI datasets combine all four.

1. Surgical phase segmentation

The annotator assigns a phase label to every frame — or to contiguous temporal segments — indicating which step of the procedure is currently occurring. For laparoscopic cholecystectomy, the most widely used taxonomy is the seven-phase Cholec80 schema (preparation, calot triangle dissection, clipping and cutting, gallbladder dissection, gallbladder packaging, cleaning and coagulation, gallbladder retraction). Phase boundaries must be marked at the exact frame where the transition occurs, not approximately — phase boundary precision directly affects model performance on intraoperative phase prediction tasks.

2. Instrument detection and tracking

Annotators draw bounding boxes or segmentation masks around each visible instrument in each frame, classifying by instrument type (grasper, clipper, scissors, hook, irrigator, specimen bag, stapler). For instrument tracking models — which follow a specific instrument across frames — inter-frame annotation consistency is critical: the instrument identity must remain consistent across frames even when the instrument partially exits the field of view and re-enters.

Multi-instrument frames — where two or more instruments are simultaneously visible and may overlap — require careful annotation of occlusion order (which instrument is in front). Annotation tools with depth ordering support are required for these frames; tools that only support single-layer annotation cannot encode the overlapping instrument stack correctly.

3. Anatomy segmentation

Anatomical structure segmentation is the most demanding credential task in surgical video annotation. For cholecystectomy, the target structures are the cystic duct, cystic artery, gallbladder body and fundus, common bile duct (which must be identified but not clipped), and hepatocystic triangle tissue planes. For colorectal surgery, target structures include the mesorectal envelope, ureter, iliac vessels, and autonomic nerve bundles.

Anatomy segmentation errors are directly safety-relevant: a model trained on annotations where the common bile duct is incorrectly labelled as cystic duct will produce guidance that, if acted upon, would cause a bile duct injury — the most serious complication of cholecystectomy. This is why anatomy segmentation annotation must be performed by subspecialist surgeons, not general surgeons or non-clinicians.

4. Critical view of safety (CVS) labelling

The critical view of safety is the internationally accepted intraoperative criterion for safe clipping in laparoscopic cholecystectomy: the hepatocystic triangle must be cleared of fat and fibrous tissue, the lower third of the gallbladder must be separated from the liver bed, and only two structures (cystic duct and cystic artery) should be seen entering the gallbladder. CVS labelling marks the exact frame range where all three criteria are simultaneously met.

CVS annotation is among the highest-value annotation tasks in surgical AI because AI models trained to verify CVS have direct patient safety implications: an AI system that confirms or flags CVS achievement before the surgeon clips can reduce bile duct injury rates. CVS annotation requires HPB (hepato-pancreato-biliary) or advanced laparoscopic surgeon annotators — not general surgeons, and certainly not non-clinicians.

Endoscopy Video Annotation: Colonoscopy and Gastroscopy

Endoscopy AI annotation differs from surgical video annotation in scale and frame density. A typical colonoscopy screening programme generates 20–30 minutes of video per patient and the high-value annotation task — polyp detection and classification — occurs in a minority of frames. Annotators must review the full video to identify the frames where a polyp appears, often for fewer than 30 seconds of a 25-minute procedure.

Colonoscopy polyp annotation for AI must include both polyp bounding boxes and Paris classification (morphology: 0-Ip pedunculated, 0-Is sessile, 0-IIa flat elevated, 0-IIb completely flat, 0-IIc flat depressed) because the model's utility depends on classifying not just the presence of a polyp but its histological risk category. NBI (narrow-band imaging) frames require separate annotation guidelines from white-light endoscopy frames — the vascular patterns visible under NBI require specific training to interpret.

Case Study: Improving Laparoscopic Cholecystectomy Phase Detection

A surgical AI company developing an operating room workflow intelligence platform needed a training dataset of 4,200 laparoscopic cholecystectomy videos for phase detection and CVS verification models. Their initial dataset of 1,100 videos had been annotated by a general annotation team using an annotation guide written by one surgeon. Phase boundaries were estimated from the guide rather than surgeon-verified, and CVS labelling was performed by non-surgeons following a three-criterion checklist.

Internal validation showed phase detection F1 of 0.81. On intraoperative deployment in a 12-week clinical evaluation, actual phase detection accuracy fell to F1 0.72, and the CVS verification model achieved 61% sensitivity for correctly identifying CVS-achieved frames — far below the ≥85% threshold the surgical team considered clinically useful.

Audit of the original annotations identified three systematic failure modes:

• Phase boundary drift: Non-surgeon annotators had applied phase boundaries 30–120 seconds early on average — marking the start of calot triangle dissection when the surgeon was still completing port placement — because visual instrument positioning did not capture the surgeon's tactical intent
• CVS overcalling: Non-surgeons had marked CVS as achieved in 44% of frames where an HPB surgeon assessment found at least one of the three criteria unmet — typically the hepatocystic triangle clearing criterion
• Anatomy misidentification: The cystic duct was incorrectly identified (with common bile duct labelled as cystic duct) in 8.3% of dissection-phase frames across the dataset

A reannotation project was commissioned with 12 board-certified surgeons — eight HPB surgeons for CVS and anatomy annotation, four advanced laparoscopic surgeons for phase segmentation. All 4,200 videos were dual-annotated with adjudication. Results after retraining:

The reannotation took 16 weeks with the 12-surgeon panel. The retrained model entered a prospective randomised surgical trial comparing AI-assisted CVS verification against standard practice and received a favourable safety profile from the independent data monitoring committee. The original non-surgeon-annotated model would have posed an unacceptable patient safety risk if deployed in a real operating theatre environment.

HIPAA, De-identification, and FDA 21 CFR Part 11 for Surgical Video

Surgical and endoscopy video sourced from hospital systems contains PHI that requires careful de-identification before annotation. The specific challenges for video are more complex than for static imaging:

• Burned-in overlays: Most surgical video capture systems embed patient name, date of birth, procedure date, surgeon name, and hospital identifier directly into the video pixel data as overlaid text — these cannot be removed by metadata-only stripping and require per-frame pixel scrubbing or masking
• Audio track: Surgeon-to-team communication captured in the operating room audio contains patient-identifiable information and must be either muted or transcribed and de-identified before annotation
• Surgeon identity: Operating surgeon identity can be re-identifiable from surgical technique patterns in a small-volume training dataset — de-identification protocols should include temporal and procedural shuffling to prevent technique-based re-identification
• Timestamp metadata: Procedure timestamps must be shifted consistently for datasets where temporal ordering of a patient's surgical history is an identifiable attribute

For FDA 510(k) De Novo submissions of surgical AI devices, FDA 21 CFR Part 11-compliant annotation provenance is required: a tamper-evident audit trail per annotated segment documenting surgeon annotator identity, board certification number, the date and duration of each annotation session, and the adjudicator identity for disputed annotations. The FDA Software as a Medical Device (SaMD) guidance explicitly identifies annotator qualifications as a required component of training data documentation in regulatory submissions.

In Australia, surgical AI applications targeting clinical deployment are regulated under the Therapeutic Goods Administration (TGA) as Class IIb or Class III medical devices under the Medical Devices framework. TGA follows a similar evidence standard to the EU MDR for training data provenance — annotator qualifications and inter-observer agreement must be documented in the technical file.

The Surgical AI Market and Why Annotation Quality Determines Product Outcomes

The global surgical AI market was valued at USD $1.2 billion in 2024 and is projected to grow at 31% CAGR to USD $5.1 billion by 2030 (Grand View Research, 2025). Primary growth drivers include robotic surgery augmentation (instrument tracking and tissue differentiation for Da Vinci and Hugo platforms), operating room efficiency AI (surgical phase prediction for workflow scheduling), and endoscopy AI (colonoscopy polyp detection, which has already achieved FDA 510(k) clearance for multiple products).

The surgical AI products currently cleared or approved by FDA — including the colonoscopy polyp detection systems and the operating room workflow management platforms — were all trained on surgeon-annotated datasets with documented multi-expert adjudication. None of the currently cleared surgical AI products used crowdsourced annotation pipelines.

For AI teams entering the surgical space, this creates a clear imperative. Annotation quality is not a secondary consideration for surgical AI — it is the primary determinant of whether a product can reach clinical deployment and regulatory clearance. The cost of surgeon annotation is higher per minute than general annotation by an order of magnitude, but the cost of a failed clinical validation or a patient safety event in a clinical trial is substantially higher again.

Frequently Asked Questions