Panoptic Segmentation: When You Need Both Instances and Background

Semantic, Instance, and Panoptic: What Each Annotation Type Captures

The three segmentation annotation types answer different questions about an image:

The distinction matters for annotation planning because panoptic requires annotators to make two decisions per pixel: (1) which semantic class does this pixel belong to, and (2) if it is a thing class, which specific instance does it belong to. The second decision — instance ID assignment — requires careful boundary management at occlusion edges and consistency across adjacent objects of the same class. These are annotation judgements that add substantial annotator time relative to semantic-only labelling.

When Panoptic Segmentation Annotation Is Worth the Cost Premium

A 2022 survey published in IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) found that panoptic annotation costs 2.3–3.8× more per image than semantic segmentation on equivalent urban driving scenes, and that teams who had started with semantic-only data and later needed instance IDs spent 1.7× the original annotation cost on rework — more than if they had annotated panoptic from the start. The implication is straightforward: if your downstream model architecture requires instance IDs, annotating panoptic from the start is almost always cheaper than retrofitting instances onto existing semantic labels.

Panoptic annotation is worth the cost premium in these scenarios:

• Autonomous driving perception: Multi-object tracking, pedestrian counting, vehicle gap estimation, and turn prediction all require individual instance IDs that semantic segmentation cannot provide. ADAS systems at L2+ and AV systems at L3+ require panoptic-grade training data for production perception stacks.
• Robotic manipulation and navigation: Robots reasoning about individual objects (pick this bottle, avoid that person) need instance IDs. Semantic labels alone cannot distinguish one bottle from another in a cluttered scene.
• Smart city and crowd monitoring: Pedestrian density estimation, crowd flow modelling, and individual tracking in public spaces all require instance-level labels. Semantic segmentation identifies 'crowd' but cannot count individuals or assign persistent IDs across frames.
• Industrial inspection with multiple defect instances: When the model must count defects per unit, locate each individually, and classify the full scene simultaneously, panoptic is the appropriate label format.

Panoptic is not worth the cost premium when only one of the two tasks is required. If your model needs scene understanding without individual object tracking (satellite imagery classification, medical image segmentation, agricultural field mapping), semantic annotation is appropriate. If your model needs to detect and count objects in a sparse scene where background is unimportant (product detection in e-commerce images, document layout analysis), instance annotation is appropriate. Paying for panoptic annotation when your model architecture uses only one of the two label types is unnecessary cost with no quality benefit.

The Three Annotation Challenges Unique to Panoptic

1. Instance boundary assignment at occlusion edges

When two objects of the same class overlap — a pedestrian partially occluded by another pedestrian, two vehicles side-by-side at the image boundary — the annotator must assign every boundary pixel to one instance ID. The convention matters: boundary pixels are typically assigned to the front-most (least occluded) instance, but annotators must apply this consistently across tens of thousands of frames for the downstream model to learn correct edge behaviour.

Occlusion handling is where panoptic annotation inter-annotator agreement drops most sharply. Annotators without explicit calibration on occlusion-edge cases assign boundary pixels inconsistently — sometimes to the front object, sometimes to the rear, sometimes splitting ambiguously between both. This inconsistency shows up as low Panoptic Quality (PQ) on validation even when overall mask quality is high. Calibration sessions using occlusion-edge reference cases before production annotation begins are essential.

2. Consistent instance ID assignment across frames in video

For autonomous driving and video surveillance datasets, panoptic annotation must maintain consistent instance IDs across consecutive frames — pedestrian #7 in frame 1 must still be pedestrian #7 in frame 47 even if the pedestrian leaves and re-enters the frame. This is a tracking annotation task layered on top of segmentation, requiring annotators to maintain an instance registry and apply IDs consistently through sequences.

ID-switch errors — where a tracked object receives a new instance ID mid-sequence — are the most common quality failure in video panoptic annotation. An ID-switch on a pedestrian in a 200-frame sequence creates a training example where the model is asked to simultaneously predict that a person is 'new' and that the same person disappeared, injecting conflicting signals into the tracker training. Production video panoptic annotation requires explicit ID tracking tools and a QA pass specifically for ID-switch detection.

3. Stuff class boundary precision at complex textures

Stuff classes — road surface, vegetation, sky, building facades — have complex boundaries at fine texture transitions that require careful polygon or mask annotation. A tree canopy overlapping a building facade, or vegetation encroaching on a road surface, requires the annotator to make precise boundary decisions at pixel level. Unlike thing classes where the object boundary is usually visually distinct, stuff class boundaries at texture transitions are judgement calls that vary significantly between annotators without calibrated reference examples.

Coarse stuff class boundaries propagate into low Segmentation Quality (SQ) scores on PQ evaluation — which then suppresses the overall PQ score even when thing-class instance annotation is high quality. Stuff class annotation guidelines should include minimum boundary precision requirements (e.g., no boundary pixel more than 3 pixels from the true edge), reference images for common texture-transition edge cases, and a separate QA pass on stuff-class boundaries in addition to the standard thing-class QA.

Case Study: Switching From Semantic to Panoptic on Urban Driving Scenes

In mid-2025, an autonomous vehicle startup developing an L3 urban driving system had 180,000 annotated driving frames in semantic-only format. The semantic annotation had been performed over six months and covered 25 classes across urban, suburban, and motorway scenes. The perception model trained on this data was performing well on scene classification but failing on the multi-object tracking task required for urban intersection navigation.

The core problem was that the semantic model could identify 'pedestrian' pixels but could not distinguish pedestrian #3 from pedestrian #7 when they were close together or momentarily occluded. The tracking module was generating instance IDs from detection outputs rather than from annotation-trained instance features, leading to high ID-switch rates on the pedestrian class.

An audit of the tracking module performance on a 5,000-frame held-out set showed:

• Pedestrian ID-switch rate: 23.4% (one in four pedestrians received an incorrect or new ID within a 10-frame window around an occlusion event)
• False positive rate on pedestrian detection: 18.1% (background pixels misclassified as pedestrian instances)
• Multi-pedestrian scene accuracy: 61.3% at correctly counting the number of pedestrians in crowded crossings

The team re-annotated 60,000 representative frames (urban intersections, pedestrian crossings, and multi-vehicle scenarios) in panoptic format over 9 weeks, using a specialist AV annotation team with explicit instance ID tracking and occlusion-edge calibration. The remaining 120,000 frames (motorway and suburban scenes with lower pedestrian density) retained semantic-only labels, saving annotation cost on scenes where instance IDs provided minimal model benefit.

Results after retraining on the hybrid panoptic/semantic dataset: pedestrian ID-switch rate dropped from 23.4% to 7.8%. False positive rate on pedestrian detection dropped from 18.1% to 9.3%. Multi-pedestrian count accuracy at crossings improved from 61.3% to 84.7%. The panoptic annotation cost for 60,000 frames was approximately AUD $5.10/frame versus the original AUD $1.80/frame for semantic, a cost differential of approximately AUD $198,000 — which the team estimated was recouped within two sprint cycles of previously planned rework on the tracking module.

Measuring Panoptic Annotation Quality: Panoptic Quality (PQ)

Panoptic Quality (PQ) is the primary metric for evaluating both panoptic annotation quality and panoptic model performance. Introduced by Kirillov et al. (CVPR 2019), PQ decomposes into two components:

For annotation QA purposes, PQ-based inter-annotator agreement is more informative than simple pixel accuracy. Two annotators annotating the same urban driving scene can have high pixel accuracy (90%+) while having low PQ agreement (0.55–0.65) due to instance ID inconsistencies and occlusion boundary disagreements — which is exactly the kind of annotation quality issue that causes tracking model failures.

Production panoptic annotation for AV systems typically targets PQ ≥ 0.80 on a gold standard validation set. Below 0.75, instance ID inconsistencies are frequent enough to train incorrect tracking behaviour. The COCO benchmark PQ for state-of-the-art models on the COCO 2023 panoptic challenge ranges from 0.58 to 0.71 — annotation quality at PQ < 0.80 on your training set means you are training against noisier labels than the benchmark models used.

Tooling for Panoptic Segmentation Annotation

Not all segmentation annotation platforms support panoptic output formats. The minimum requirements for a panoptic-capable annotation platform are: pixel-level mask annotation tools (polygon, brush, or SAM-assisted tools), per-instance ID assignment with an instance registry that persists across a dataset, export in COCO panoptic JSON format or equivalent (per-image PNG maps with instance-encoded segment IDs), and video ID-tracking tools for multi-frame datasets.

CVAT, Label Studio with custom panoptic templates, Segments.ai, and SuperAnnotate all support panoptic annotation natively or with configuration. SAM (Segment Anything Model)-assisted annotation has significantly improved panoptic annotation throughput — AI-assisted mask proposals that annotators accept, refine, or reject reduce per-frame annotation time by 35–55% on urban driving scenes compared to manual polygon annotation (based on internal benchmarks from production AV annotation workflows).

For the semantic segmentation annotation layer within panoptic, the tooling requirements are identical to standard semantic annotation. The additional work in panoptic is the instance ID assignment layer — which requires platform support and annotator training but does not require a fundamentally different annotation interface for the segmentation mask itself. Teams already producing high-quality semantic segmentation can typically onboard to panoptic annotation in 1–2 calibration cycles without a complete workflow change.

Frequently Asked Questions