Quick answer
Image annotation is the structured labelling of photographs, frames, or scans with bounding boxes, polygons, pixel masks, keypoints, or classification tags so that a computer vision model can learn object detection, scene segmentation, or pose estimation. The right annotation type depends on the model architecture and task: bounding boxes for object detection, semantic or instance segmentation for pixel-level scene understanding, keypoints for pose and landmark tasks, and polygon annotation for irregularly shaped objects requiring precise boundaries without full pixel coverage.
The Five Primary Image Annotation Types
Each annotation type encodes different spatial information. Using the wrong type for a task does not just waste annotation budget — it produces training data that cannot serve the model architecture. Here is a plain-language map of what each type does and when it applies.
1. Bounding Box
A rectangle defined by four coordinates (x-min, y-min, x-max, y-max) placed around an object with a class label. The dominant format for object detection architectures including YOLO, Faster R-CNN, and DETR. Bounding boxes are appropriate when the model needs to answer "what is in this image and roughly where" — not "what shape is it." They over-include background pixels for non-rectangular objects, which can degrade detection precision on dense or irregular-shape scenes.
2. Polygon Annotation
A polygon is a closed multi-point outline drawn around the exact boundary of an object. It preserves the precise shape of non-rectangular objects — vehicles seen at an angle, furniture, tools, fruit, anatomical structures — without pixel-level annotation cost. Polygon annotation is typically 2–4 times slower per object than bounding box but 30–50% faster than semantic segmentation for object-level tasks. Use it when shape fidelity matters but you do not need every pixel classified.
3. Semantic Segmentation
Every pixel in the image is assigned a class label. Two pedestrians in the same frame share the "person" class — they are not distinguished as separate entities. Semantic segmentation is the foundation of scene parsing for autonomous vehicles (road vs footpath vs vegetation vs building vs sky), satellite and aerial image analysis, and medical imaging where tissue types rather than individual lesions are the target. Grand View Research valued the global computer vision market at USD $19.1 billion in 2023 and forecasts 19.0% CAGR through 2030 — a large share of that growth is semantic understanding applications.
4. Instance Segmentation
Instance segmentation extends semantic segmentation by assigning each individual object its own unique pixel mask, so two overlapping pedestrians receive separate masks labelled "person-001" and "person-002." This is the annotation type required for Mask R-CNN, SAM-based architectures, and any task that must count, track, or individually isolate objects. Retail shelf monitoring (count units facing forward), medical cell counting, and waste sorting on conveyor belts all require instance segmentation rather than semantic.
5. Keypoint and Landmark Annotation
Keypoints are single-coordinate markers placed at predefined anatomical or structural positions — shoulder, elbow, wrist for human pose; pupil, nose tip, mouth corners for facial landmark detection; wheel centre, headlight position for vehicle orientation. The annotation schema defines the keypoint index order and visibility flags. Any deviation from the schema at annotation time corrupts the model's spatial understanding, so keypoint projects require strict guideline adherence and a separate QA review step.
Matching Annotation Type to Model Architecture
The annotation type is not a stylistic choice — it is determined by the model family you are training. Feeding the wrong label format to a model architecture either produces a training error or forces the framework to silently discard information. Here is the canonical mapping:
| Model task | Architecture examples | Required annotation type |
|---|---|---|
| Object detection | YOLOv9, Faster R-CNN, DETR | Bounding box |
| Precise object detection | YOLO with polygon support, Detectron2 | Polygon |
| Scene parsing / pixel classification | SegFormer, DeepLab, UNet | Semantic segmentation mask |
| Individual object masking | Mask R-CNN, SAM, YOLACT | Instance segmentation mask |
| Human pose estimation | HRNet, OpenPose, MediaPipe | Keypoint skeleton |
| Face alignment / facial analysis | InsightFace, FAN, RetinaFace | Facial landmark keypoints |
A common and costly mistake is starting with bounding boxes for a task that eventually requires polygon or segmentation labels, then discovering the model's precision falls short of the product requirement. Switching annotation types mid-project means re-annotating the entire corpus. According to Scale AI's State of AI Report (2024), object detection accounts for 58% of all computer vision annotation volume — bounding box is the default — but teams serving retail, medical, or robotics applications increasingly require polygon or segmentation formats from the start.
What Drives Image Annotation Cost
Image annotation cost per image is determined by three factors: annotation type complexity, object density per image, and domain expertise required. Here are realistic AUD production rates for 2026:
- Bounding box (2–5 objects/image): AUD $0.05–$0.20 per image. Rises with density — a retail shelf image with 40 products may cost $0.80–$1.20 despite the simple format.
- Polygon annotation (irregular objects): AUD $0.15–$0.60 per object, 40–80% premium over equivalent bounding box annotation.
- Semantic segmentation (complex scenes): AUD $0.80–$4.50 per image. Autonomous vehicle street scenes at the high end; simpler two-class industrial images at the low end.
- Instance segmentation: AUD $1.20–$6.00 per image, depending on instance density. Crowded retail shelves or medical cell-counting tasks are at the upper end.
- Keypoint annotation (human pose, 17 points): AUD $0.20–$0.60 per person instance. Multiple people per frame multiply the cost proportionally.
Volume discounts of 20–35% are standard above 50,000 images on any annotation type. The bigger cost lever is annotation quality: a 2023 analysis by Landing AI found that switching from bounding box to polygon annotation for irregularly shaped industrial components reduced false-positive rates by 34% in defect detection models — meaning the higher annotation cost paid back in reduced production line errors within three months of deployment.
Need image annotation for a computer vision project?
AI Taggers provides bounding box, polygon, semantic segmentation, instance segmentation, and keypoint annotation with expert QA across retail, medical, industrial, and AV domains.
See our image annotation servicesCase Study: Multi-Type Image Annotation for Australian Retail Visual Search
In mid-2025, an Australian online fashion retailer needed to improve both product search relevance and a "shop the look" feature that allowed users to click any item in an outfit photograph and find that item in the catalogue.
The initial dataset of 95,000 product images used only bounding box annotation for object detection (identify garment category and location). The downstream product attribute classifier — which extracted colour, pattern, sleeve length, neckline, and fit — was trained separately on image-level classification labels. The results were poor:
- Product attribute classification accuracy: 64.2% on the held-out test set
- Visual search top-5 match rate: 51.8%
- "Shop the look" click-through-to-purchase conversion: 2.1%
The problem was that bounding boxes over-included background, furniture, and overlapping garments in the crop fed to the attribute classifier. A fold on a dress was classified as a pattern; a chair visible inside the bounding box was contributing noise to colour predictions.
The annotation rebuild used two types in combination. Polygon annotation replaced bounding boxes for all garment instances — capturing the exact outline of each garment and excluding background and overlapping items. Attribute classification labels (colour, pattern, 14 garment attributes) were added as image-region labels on the polygon crops. The full 95,000-image corpus was re-annotated over six weeks with a team of 14 annotators and a two-stage QA process (10% random sampling + 100% review of polygon boundary quality on a 2,000-image gold set).
Results after retraining:
- Product attribute classification accuracy: 89.7% (up from 64.2%)
- Visual search top-5 match rate: 78.4% (up from 51.8%)
- "Shop the look" click-through-to-purchase conversion: 4.9% (up from 2.1%)
The polygon annotation corpus cost approximately 2.3× the original bounding box corpus. The revenue lift from the improved visual search feature recovered that cost within seven weeks of deployment. The case illustrates a wider principle: annotation type selection is a product decision with a direct revenue impact, not a back-office data hygiene task.
Quality Controls for Image Annotation
Image annotation quality degrades in predictable ways. Annotators drift on boundary tightness, class coverage drops on small or partially occluded objects, and consensus breaks down at class boundary cases (a motorbike vs a scooter, a van vs a truck). Production-grade QA for image annotation services includes four controls:
Gold-set benchmarking
A set of 200–500 pre-annotated images with known correct labels is included in every annotator's task queue without their knowledge. Accuracy on gold-set items is tracked per annotator; annotators falling below threshold are flagged for retraining before their production output is accepted.
Inter-annotator agreement (IAA) on overlapping tasks
5–10% of production tasks are assigned to two independent annotators. IoU (Intersection over Union) between their bounding boxes or masks is computed. For bounding box tasks, IoU below 0.75 on a consistent basis indicates annotator calibration problems. For segmentation, a mean IoU below 0.80 triggers a guideline review.
Edge case taxonomy in guidelines
Annotation guidelines must define what to do with occluded objects (annotate the visible portion or skip), truncated objects at image edges (annotate to the boundary or exclude), and crowd instances (separate individual masks or group). Ambiguity on these cases causes 60–80% of IAA failures in image annotation projects.
Model-in-the-loop early warning
Training a lightweight detector on the first 10,000 annotated images and running inference on the next batch reveals systematic annotation errors: classes that the model cannot learn (ambiguous guidelines), consistent false negatives (objects being missed), and spurious positives (background being labelled). This feedback loop shortens the correction cycle from end-of-project to week two.
Choosing the Right Image Annotation Partner
The annotation type determines which tooling and annotator skills are needed. Bounding box annotation at scale can be handled by general-purpose crowdsourcing with strong QA controls. Semantic and instance segmentation at high quality requires annotators trained on boundary precision and familiar with the domain. Medical image annotation — histology, radiology, retinal fundus — requires credentialed domain experts, not general-purpose annotators.
The right partner should be able to demonstrate: inter-annotator agreement results on the annotation type you need, a defined escalation path for edge cases, and a QA pipeline that includes gold-set benchmarking rather than relying on annotator self-correction. For regulated applications (medical device AI, autonomous vehicle certification), request provenance logging and annotator credential documentation as part of the initial scoping conversation.
If you are sourcing image annotation services for a production model, specify the annotation type in your brief, include sample images that represent your edge cases, and ask the vendor for a 500-image pilot with IAA reporting before committing to full-scale production.
Related resources
- Image Annotation Services — bounding box, polygon, segmentation, keypoints
- Bounding Box Annotation — the fastest path to object detection training data
- Semantic Segmentation Services — pixel-level scene understanding
- Bounding Box Annotation Cost and Speed — production case study
- Semantic Segmentation Annotation — when and why pixel-level labels pay off
- Instance Segmentation Annotation — use cases and case study
Frequently Asked Questions
What is image annotation in AI?▼
What is the difference between semantic and instance segmentation annotation?▼
How much does image annotation cost per image?▼
What image annotation type is best for object detection?▼
Can AI tools replace human image annotation?▼
How many annotated images does a computer vision model need?▼
Get a quote for image annotation
Tell us your annotation type, volume, domain, and quality requirements. We'll respond with a scoped proposal within one business day.
Neel Bennett
AI Annotation Specialist at AI Taggers
Neel has over 8 years of experience in AI training data and machine learning operations. He specializes in helping enterprises build high-quality datasets for computer vision and NLP applications across healthcare, automotive, and retail industries.
Connect on LinkedIn