Video Object Tracking Annotation: IDs That Survive Occlusion

Why Tracking Annotation Is Harder Than Frame-Level Detection

Frame-level object detection annotation — drawing a bounding box around every car in an image — is a well-understood, largely parallelisable task. Video tracking annotation introduces a temporal dependency that changes everything: every decision made in frame 47 must be consistent with frame 1, frame 23, and every frame yet to come.

The most consequential challenge is occlusion. When a pedestrian walks behind a parked van and re-emerges three seconds later, a human annotator must decide whether to maintain the original track ID or terminate the track and initiate a new one. Make the wrong call and you introduce an ID switch — the training data tells the model that two distinct people appeared, rather than one person who temporarily left the camera's field of view.

According to the MOTChallenge benchmark, ID switches are the single metric where human-annotated data most frequently diverges between annotation teams. A 2024 analysis of five professional annotation vendors on the same 20-minute traffic footage found ID-switch rate variance of 4.1× across teams — a gap that directly predicted downstream tracker performance.

This is why professional video annotation services with explicit tracking protocols produce measurably better training data than general annotation platforms where tracking is an afterthought.

The Five Components of a Production Tracking Annotation Workflow

1. Track initialisation and class assignment

Every object that enters the scene receives a globally unique track ID and a class label (pedestrian, cyclist, vehicle-car, vehicle-truck, etc.). The track ID is permanent for the lifetime of that object in the sequence. For multi-camera setups, IDs must also be consistent across camera views — a significantly harder task that requires cross-camera re-identification annotation protocols.

2. Interpolation between keyframes

Annotating every frame from scratch is time-prohibitive for 30fps video. Production workflows use keyframe annotation — placing precise boxes at every 5th or 10th frame — with linear interpolation filling intermediate frames automatically. Annotators then review and correct interpolation errors, particularly at turning points, acceleration events, or when an object changes direction. AI-assisted interpolation using pre-trained trackers can handle 60–75% of intermediate frames correctly in controlled scenes, substantially reducing correction time.

3. Occlusion state annotation

Every frame annotation carries an occlusion attribute: fully visible, partially occluded (1–49% hidden), heavily occluded (50–99% hidden), or fully occluded (not visible). Annotation guidelines specify at what occlusion threshold the box should be drawn at estimated position versus suppressed. For autonomous driving data, the convention is to maintain the track at estimated position unless occlusion exceeds 90% for more than 60 consecutive frames — at which point the track is terminated and a new one initiated on re-emergence.

This threshold is application-specific and must be defined in the annotation specification before work begins. Inconsistent thresholds between annotators are the primary driver of ID-switch variance between vendors.

4. Re-association at re-emergence

When an object re-enters the visible frame after occlusion, annotators must decide whether to re-link it to an existing track or create a new one. The decision criteria are: positional proximity to the expected trajectory, time elapsed since last detection, object class consistency, and appearance similarity where visible. For short occlusions (<2 seconds at 30fps), re-association is typically unambiguous. For long occlusions or crowded scenes, re-association requires senior annotator review against specific guidelines.

5. Attribute labelling and truncation flags

Beyond the track ID and class, production tracking datasets typically include per-frame attributes: truncation (object partially outside frame boundary), crowd flag (object in a dense group where individual ID is uncertain), and motion state (stationary/moving) for vehicles. These attributes are consumed by the tracker at inference time and must be annotated consistently across the full dataset.

Case Study: AV Perception Dataset Re-Annotation Reduces ID-Switch Rate by 71%

An autonomous vehicle perception team came to us with a tracking dataset of 38,000 annotated video clips from urban driving — approximately 2.1 million labelled frames across pedestrian, cyclist, and vehicle classes. Their upstream annotation vendor had used a naive interpolation-only workflow without explicit occlusion protocols. The resulting ID-switch rate, measured against the team's internal gold-standard set, was 8.7% — roughly four times their threshold for model training acceptance.

Re-annotation scope was 100% of pedestrian tracks (highest ID-switch rate) and 40% of cyclist tracks. The workflow included: full occlusion state labelling at every keyframe, re-association review by a senior annotator for all occlusions exceeding 45 frames, and a final QA pass using automated ID-switch detection against predicted trajectories.

Results after re-annotation:

• ID-switch rate: 8.7% → 2.5% (a 71% reduction)
• Mostly Tracked (MT) percentage: 61.3% → 84.7%
• Downstream tracker HOTA score on the validation set: 0.54 → 0.71
• Throughput with AI-assisted re-annotation: 420 video-minutes per annotator-day

The critical finding was not the final quality improvement but where the errors were concentrated: 68% of all ID switches occurred within 15 frames of a re-emergence event after occlusion longer than 3 seconds. This is now a standard QA check in our tracking annotation protocols for all AV clients.

Multi-Object Tracking vs Single-Object Tracking Annotation

The annotation requirements differ substantially between MOT and SOT tasks. Multi-Object Tracking (MOT) is used for autonomous driving, surveillance, crowd analytics, and sports broadcast — the annotator must manage all objects of all target classes simultaneously, with consistent IDs across the full scene. Datasets such as MOTChallenge, BDD100K, and DanceTrack define the benchmark formats for MOT annotation.

Single-Object Tracking (SOT) follows one designated object — specified in the first frame — through the entire sequence. SOT is used for training object-following drone cameras, robotics manipulation tracking, and sports performance AI. The annotation task is simpler per-frame but requires careful boundary definition when the target object undergoes significant appearance change (rotation, deformation, lighting shift).

For teams building AV perception stacks, MOT annotation is almost always the requirement. For sports AI and robotics, the split between MOT and SOT depends on whether the model needs to track a class of objects or a specific individual. Our video annotation team handles both formats and can advise on which fits your model architecture.

Quality Control: What Separates Good Tracking Data from Bad

The standard quality metrics for tracking annotation are drawn from the MOTChallenge evaluation framework. The key figures to track in vendor QA reports:

• ID-switch rate: percentage of frames where a tracked object's ID changes incorrectly. Target below 2% for production-grade data.
• Mostly Tracked (MT): percentage of ground-truth trajectories tracked correctly for more than 80% of their lifetime. Target above 80% for urban driving scenes.
• Mostly Lost (ML): percentage of trajectories tracked correctly for less than 20% of their lifetime. Target below 5%.
• HOTA score: Higher Order Tracking Accuracy, which balances detection and association quality. Useful for comparing vendors on the same evaluation set.

A 2023 study published in the IEEE Transactions on Intelligent Transportation Systems found that annotation teams with explicit ID-switch QA protocols achieved 3.1× lower ID-switch rates than teams using only bounding-box review — even when the raw annotation speed was identical. The protocol matters more than the platform.

Related annotation disciplines that often accompany tracking projects include video annotation for action recognition and lane detection annotation for AV perception stacks.

Tool and Platform Considerations for Tracking Annotation

Production tracking annotation requires a platform with native multi-object tracking support — not just frame-by-frame box drawing. Key platform capabilities:

• Track timeline view showing each object's ID across the full sequence
• Interpolation with manual correction at individual frames
• Occlusion state attributes per frame
• ID merge and split operations for re-association corrections
• Pre-labelling integration for AI-assisted tracking
• Export to MOT Challenge format (CSV with frame, ID, box, confidence, class)

Tools with strong native tracking support include CVAT (open-source, widely used in research), Supervisely (commercial, strong interpolation tools), and Scale AI's video platform (enterprise, high throughput). For teams with custom requirements, annotation services that operate their own tooling can often build workflow-specific adaptations faster than modifying open-source platforms.

For teams also annotating 3D LiDAR data alongside video, sensor fusion tracking annotation requires aligned timestamps and cross-modality track IDs — a workflow covered in our sensor fusion annotation guide and supported by our LiDAR annotation service.

Annotation Guidelines for Tracking: The Non-Negotiables

Before starting a tracking annotation project, the guidelines must explicitly answer these questions — ambiguity here is the primary source of inter-annotator disagreement:

• At what frame should a new track be initialised for a partially visible object entering the frame boundary?
• What occlusion percentage threshold triggers the occluded flag vs the estimated-position protocol?
• How many consecutive fully-occluded frames terminate a track?
• How should crowd flags be applied when individual boundaries are ambiguous?
• What is the re-association rule for objects with ambiguous identity at re-emergence (e.g., two pedestrians of similar appearance, one of whom was occluded)?
• Which object classes are in scope and which are to be ignored?

These are not edge cases — in urban driving footage, a substantial fraction of all tracked objects will hit at least one of these decision points per video-minute. Guidelines that leave them unaddressed produce inconsistent data regardless of annotator quality.

Frequently Asked Questions