Sensor-Fusion Annotation: Aligning Camera, LiDAR and Radar Labels

Why Sensor Fusion and Why It Requires Specialised Annotation

No single sensor provides the complete environment perception that production autonomous vehicle or robotics systems require. Camera images offer rich texture and colour at high resolution but fail in low light and cannot measure depth or velocity directly. LiDAR provides precise 3D geometry and works in darkness but produces sparse point clouds at long range and is degraded by heavy rain or fog. Radar measures radial velocity precisely and works in all weather but has poor angular resolution and cannot recover object shape.

A 2023 study published in Nature Machine Intelligence found that AV perception systems using camera-only inputs had a 4.7× higher pedestrian miss rate in adverse lighting conditions compared to systems using fused camera-LiDAR inputs, and a 2.1× higher rate under normal conditions. For rain and fog conditions, camera-only miss rate was 6.3× higher than camera-LiDAR-radar fusion systems. These numbers explain why every production AV platform at L3+ uses sensor fusion — and why the training data must be annotated in a way that preserves that fusion benefit.

The annotation challenge is that multi-modal data is only as good as the alignment between the labels across modalities. If a pedestrian's camera bounding box is spatially misaligned with their LiDAR cuboid by 40 centimetres due to a calibration error in the annotation workflow, the model is trained with contradictory inputs: the camera says the person is here, the LiDAR says they are there. Rather than learning to fuse the modalities, the model learns that camera and LiDAR disagree — which is the opposite of the perception robustness the sensor suite was designed to provide.

The Three Technical Requirements for Sensor-Fusion Annotation

1. Extrinsic calibration accuracy

Extrinsic calibration defines the spatial transformation between sensor coordinate frames: the rotation and translation matrices that map a point in LiDAR space to the equivalent point in camera space, and vice versa. Annotation platforms use these matrices to project LiDAR points into the camera image plane (for overlay visualisation) and to project camera-image bounding boxes into the LiDAR coordinate frame (for 3D position estimation).

Production-quality sensor-fusion annotation requires extrinsic calibration accuracy of ±0.2–0.5 degrees in rotation and ±1–3 cm in translation. The practical impact of calibration error is significant at long range: a 0.5-degree rotation error between camera and LiDAR produces a label offset of approximately 44 cm at 50 metres range and 87 cm at 100 metres range. At 100 metres, an 87 cm offset is large enough to place a pedestrian's LiDAR points entirely outside their camera bounding box — producing annotation that the model cannot learn to fuse coherently.

Calibration should be verified at the start of each annotation project and after any vehicle maintenance event that could have repositioned a sensor. Annotation platforms should report per-frame calibration residual: the mean reprojection error between LiDAR points projected into camera space and their expected image positions on calibration targets. Frames with calibration residual above threshold (typically 1.5–2.0 pixels mean reprojection error) should be flagged for re-calibration before annotation proceeds — annotating with degraded calibration cannot be corrected in post-processing and produces permanently misaligned labels.

2. Temporal synchronisation

AV sensor suites operate at different frequencies: cameras typically capture at 25–30 Hz, LiDAR at 10–20 Hz, and radar at 10–25 Hz. When an AV platform travels at 60 km/h (16.7 m/s), a 50-millisecond temporal misalignment between camera and LiDAR produces a position offset of approximately 83 cm — enough to misalign labels on objects at highway speeds.

Temporal synchronisation in annotation is achieved by: (1) hardware-level timestamping using a shared clock signal (GPS-PPS, IEEE 1588 PTP) that timestamps each sensor return with microsecond accuracy; and (2) software interpolation at annotation time, where objects annotated in the LiDAR frame at timestamp T are projected into the camera frame at the nearest camera timestamp using the platform's ego-motion data to account for vehicle movement between the two timestamps.

Annotation platforms that do not support motion-compensated projection — using only the closest-timestamp frame without ego-motion correction — produce systematic temporal offset errors that scale with vehicle speed. This is a common quality failure in annotation workflows that were initially designed for static or slow-moving scenes and later applied to highway AV data without updating the temporal alignment approach.

3. Cross-modal label consistency

Cross-modal label consistency means that the same physical object has the same class label and instance ID in camera, LiDAR, and radar streams simultaneously. Achieving this requires annotation workflows where annotators see all modalities together rather than annotating each sensor stream independently and merging afterwards.

The most common cross-modal consistency failure is independent annotation: camera annotations are done first, LiDAR annotations second, and the two are merged by matching bounding geometries in post-processing. This approach fails in four scenarios: (a) objects visible in camera but below LiDAR scan height (ground-level debris, low-profile vehicles in depressions); (b) objects visible in LiDAR but outside camera field of view (objects behind the vehicle or in blind spots between cameras); (c) objects with different class assignments in different modalities (a motorcyclist annotated as 'motorcycle' in camera and 'vehicle' in LiDAR due to different annotator conventions); and (d) objects with different instance boundaries in different modalities (a vehicle with camera box tightly enclosing the body but LiDAR cuboid including the mirrors and roof rack).

Production sensor-fusion annotation requires integrated multi-modal annotation interfaces where annotators see camera images, LiDAR point clouds, and radar sweeps in a synchronised view, and where a label placed on one modality is immediately projected into the other modalities for consistency review. Annotation platforms that support this integrated view — nuScenes-compatible platforms, Scale AI's multi-modal task type, Annotell — reduce cross-modal consistency errors by 60–75% versus independent-modality workflows, based on internal benchmarks from production AV annotation projects.

Case Study: Camera-LiDAR Fusion Annotation for L3 Urban AV

In mid-2025, an autonomous vehicle company developing an L3 urban shuttle system was experiencing a significant gap between perception performance in structured test-track environments and performance in live urban deployment. The platform used a six-camera surround array, one 128-beam Ouster LiDAR, and four corner 77 GHz radar units. The annotation dataset of 280,000 camera-LiDAR paired frames had been annotated using an independent-modality workflow: camera bounding boxes first, LiDAR cuboids second, merged in post-processing by spatial proximity matching.

An audit of the annotation dataset identified three systematic problems from the independent-modality workflow:

• 17.3% of pedestrian camera-LiDAR pairs had cross-modal IoU below 0.65 due to calibration drift and temporal offset errors at speeds above 50 km/h
• 8.9% of camera bounding boxes for cyclists had no matching LiDAR cuboid (cyclists with low LiDAR return due to small cross-section were undetected in the independent LiDAR pass)
• Cross-modal instance ID mismatches affected 6.2% of object pairs in multi-object scenes (two adjacent pedestrians swapping IDs between camera and LiDAR streams)

Evaluation on a held-out 8,000-frame test set showed:

• Pedestrian detection accuracy (fused camera-LiDAR model): 71.3% at 50m range
• Cyclist detection accuracy: 58.6% at 50m range
• False negative rate in adverse lighting (dawn/dusk): 29.4% on the pedestrian class

The team re-annotated 95,000 frames — selected to cover urban intersection sequences, mixed pedestrian-cyclist scenes, and adverse lighting conditions — using an integrated multi-modal annotation workflow. Annotators worked in a synchronised camera-LiDAR-radar view with motion-compensated projection verified against calibration residuals. Average annotation throughput was 18 frames per annotator per hour (versus 28 frames per hour in the original independent workflow), with per-frame cost approximately 1.9× higher.

After retraining on the re-annotated dataset, the same held-out evaluation showed: pedestrian detection accuracy at 50m improved from 71.3% to 93.8%. Cyclist detection accuracy improved from 58.6% to 87.1%. False negative rate in adverse lighting dropped from 29.4% to 8.7%. The annotation investment for 95,000 frames at AUD $4.80/frame (integrated workflow rate) was approximately AUD $456,000. The operations team estimated the improvement in pedestrian detection alone reduced their required safety-driver intervention frequency by 3.4× in urban deployment, which was the gating metric for autonomous operation approval in the test jurisdiction.

Radar Annotation in Multi-Modal Fusion Datasets

Radar is the most underspecified modality in multi-modal AV annotation because its sparse returns and poor angular resolution make standard bounding-box annotation ambiguous. A 77 GHz automotive radar typically returns 200–2,000 points per scan — compared to 40,000–120,000 LiDAR points — and adjacent objects within the same angular resolution cell (typically 1–4 degrees) may be unresolvable as separate targets.

Radar annotation in fusion datasets is best structured as attribute annotation on objects already identified from camera and LiDAR, rather than independent radar object detection annotation. For each object identified in the camera-LiDAR annotation, the radar annotation adds:

• Velocity label: radial velocity measured by radar for the object (stationary, approaching, receding, cross-traffic)
• Radar-visible flag: whether the object has a matching radar return within the expected position range (confirmed by radar) or no matching return (not detected by radar — important for coverage analysis)
• Ghost detection flag: radar returns in the scan that do not correspond to a camera-LiDAR object (clutter, multipath reflections, stationary infrastructure returns) that the model should learn to suppress

This structure allows the fusion model to learn when radar confirms camera-LiDAR detections (and can be used to boost confidence), when radar does not confirm them (the object may be radar-absorptive material or at a geometry that produces specular reflection away from the sensor), and when radar detects something that camera-LiDAR do not (partially occluded objects that break the LiDAR scan angle but produce strong radar returns).

A 2024 nuScenes dataset analysis found that objects with confirmed radar returns had 2.3× higher detection recall in camera-LiDAR-radar fusion models compared to objects without radar confirmation, particularly in low-light and adverse weather conditions where camera performance degraded. The practical implication for annotation is that radar-visible flags on each object — even when annotated as simple binary attributes — provide significant model training signal at relatively low annotation cost compared to the camera and LiDAR annotation they are attached to.

QA Controls Specific to Sensor-Fusion Annotation

Sensor-fusion annotation requires QA controls that go beyond per-modality quality checks. The three QA controls that catch the failure modes most damaging to fusion model training are:

Formats and Platforms for Multi-Modal Fusion Annotation

The nuScenes dataset format is the most widely adopted open standard for multi-modal fusion annotation. It stores calibration data, sensor timestamps, ego-poses, and per-object annotations in a scene-level JSON structure where each physical object has a single token ID referenced across all sensor modalities. This shared token ID structure is what makes cross-modal consistency verification tractable — every modality's annotation points to the same object record.

Annotation platforms that natively export nuScenes-compatible JSON are strongly preferable to platforms that require custom post-processing to create multi-modal datasets. Manual post-processing to merge camera and LiDAR annotation exports — matching objects by spatial proximity or timestamp — introduces the same cross-modal ID inconsistency errors that integrated annotation workflows are designed to prevent. The merge step itself becomes an annotation error source.

For teams using the LiDAR annotation services and camera annotation pipelines at AI Taggers, multi-modal fusion projects are scoped to use integrated annotation tools with calibration-verified spatial alignment from the start — avoiding the independent-then-merge workflow that was the root cause of the annotation quality failures in the case study above. Related service: autonomous vehicle data annotation.

Frequently Asked Questions