Skeletal Pose Estimation Annotation for Sports and Health AI

Why Pose Annotation Quality Determines Model Biomechanical Accuracy

Pose estimation models — MediaPipe, OpenPose, MMPose, ViTPose — all learn from keypoint annotations. The model's understanding of where a human shoulder is when viewed from 45 degrees, partially behind a cricket bat, at low resolution, is entirely determined by how annotators placed that keypoint in similar training images.

A 2022 analysis published in the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition found that annotation teams without anatomy training produced mean OKS scores of 0.71–0.79, compared to 0.88–0.93 for teams with structured biomechanics training. This gap translated directly to 12–18% lower AP50:95 on downstream pose models trained on the respective datasets — a performance difference that makes the anatomy-trained dataset effectively a different product category.

The gap widens further for difficult poses. For a standing frontal subject with all joints visible, inter-annotator agreement on the COCO benchmark is consistently above 0.93 OKS across teams. For a bowling cricketer at the point of release — where the front knee may be partially occluded by the body, the bowling arm partially behind the head, and the spine in a non-standard extension — the same teams diverge to 0.61–0.89 OKS. That variance is the quality risk that sports AI teams consistently underestimate.

This is why professional keypoint and landmark annotation services with anatomy-trained annotators and biomechanics QA protocols exist as a distinct offering from general image annotation.

Keypoint Schemas: COCO, MPII, and Domain Extensions

The choice of keypoint schema determines your model architecture, loss function, and downstream compatibility with benchmark evaluation frameworks. The three most common schemas:

• COCO 17-keypoint: Nose, left/right eyes, left/right ears, left/right shoulders, elbows, wrists, hips, knees, ankles. The most widely used schema; compatible with most pre-trained models and benchmark evaluation pipelines.
• MPII 16-keypoint: Slightly different joint definitions (adds thorax, pelvis, neck), used primarily in research settings and some sports biomechanics applications.
• Whole-body (COCO WholeBody, 133 keypoints): Extends COCO with 68 face landmarks, 21 hand keypoints per hand, and 6 foot keypoints. Used for sign language recognition, fine-grained gesture AI, and detailed rehabilitation monitoring.

Domain-specific extensions for Australian sports applications typically add: toe and heel points for running gait analysis (AFL, athletics), grip-point keypoints for cricket batting/bowling, and spine mid-point for swimming stroke analysis. These extensions must be defined before annotation begins because retro-fitting additional keypoints to an existing dataset requires complete re-annotation of every image.

Occlusion: The Hardest Part of Pose Annotation

The COCO visibility convention defines three states for each keypoint: v=0 (not labelled), v=1 (labelled but not visible — occluded or out of frame), and v=2 (labelled and fully visible). The v=1 case — placing a keypoint at the estimated anatomical location of a joint that is not directly visible — is where annotation quality diverges most dramatically between teams.

An annotator without anatomy training, asked to place a left shoulder keypoint on a subject facing away from the camera with the left arm tucked behind the back, has no principled basis for placing the point accurately. They will typically place it at the visible skin boundary or skip it as v=0. An annotator with anatomy training reasons from the visible skeletal landmarks: the right shoulder position implies left shoulder position via bilateral symmetry, the spine midpoint constrains the shoulder axis, and the left elbow position constrains the arm geometry. The resulting v=1 placement is not a guess — it is a constrained anatomical inference.

Dropping occluded joints as v=0 instead of estimating them as v=1 introduces training bias: the model learns that partial-body poses are "complete" poses, and fails to infer missing joints at inference time. For sports AI applications where athletes are frequently partially occluded by equipment, opponents, or camera angles, this is a critical quality failure.

Case Study: Sports Performance AI for AFL Running Mechanics

An Australian sports science company building a running mechanics AI for AFL clubs came to us with 22,000 annotated video frames from game footage and training sessions. Their initial dataset had been annotated by a crowdsourcing platform using COCO 17-keypoint schema. Mean OKS against their internal physiotherapist-validated gold set was 0.73 — acceptable for generic pose estimation but insufficient for the biomechanical precision needed to detect injury-risk gait patterns in elite athletes.

Specific failure modes identified in the initial annotation:

• Ankle keypoints placed at the visible boot top rather than the anatomical lateral malleolus position — 34% error rate in occluded frames
• Occluded hip keypoints dropped as v=0 rather than estimated from pelvis geometry
• Elbow and wrist placement inconsistency during high-speed arm swing (motion blur frames)

Re-annotation scope: 100% of frames with partial occlusion, 60% of high-motion frames. Annotator team: four annotators with sports physiology or exercise science qualifications, plus a lead annotator with AFL biomechanics background reviewing all occluded joint placements.

Results after re-annotation:

• Mean OKS against gold set: 0.73 → 0.91 (a 24.7% improvement)
• Ankle keypoint accuracy in occluded frames: 66% → 94%
• Downstream model AP50:95 on held-out AFL footage: 0.58 → 0.77
• Injury-risk gait pattern detection sensitivity: 48% → 79%

The annotation throughput with anatomy-trained annotators was 340 frames per annotator-day on average — slower than the crowdsourcing baseline (580 frames/day) but producing measurably different downstream model performance that justified the investment.

Multi-Person Pose Annotation: Assigning Joints to the Right Individual

Single-person pose annotation is straightforward: one skeletal graph per image or frame. Multi-person pose annotation — the norm in team sports footage, physiotherapy group sessions, and dance AI — requires annotators to assign every keypoint to the correct individual before placing it. Misassignment (placing a right shoulder point on the wrong person in a crowded scene) is a catastrophic error that generic quality checks will not catch.

Production workflows for multi-person scenes assign each visible person a unique instance ID before keypoint placement begins. This mirrors the approach used in video object tracking annotation, where person identity must persist across frames. For video datasets, the person ID assigned at keypoint placement must align with the track ID from object tracking if both annotation layers are present.

A 2023 benchmark study on multi-person pose annotation found that instance assignment error (keypoint assigned to the wrong individual) accounted for 41% of all OKS failures in crowded scenes, despite representing a much smaller fraction of total annotation decisions. Dedicated instance-verification steps before keypoint placement reduce this error by approximately 85%.

Health and Clinical Applications: Physiotherapy and Rehabilitation AI

Clinical pose estimation annotation has requirements that differ from sports AI in several important ways. First, the subject population includes elderly, injured, and mobility-impaired individuals who do not move like athletes — atypical postures, compensatory gait patterns, and assistive device interactions must be handled in the annotation schema. Second, the use case (fall detection, exercise form checking, rehabilitation progress tracking) demands high precision at specific joints rather than global skeleton accuracy.

For fall detection in aged care, the critical keypoints are hip, knee, and ankle — the pelvis-to-floor angle at the moment of fall initiation is the primary detection signal. For physiotherapy exercise monitoring (squat form, shoulder abduction, knee rehabilitation), the relevant joints are specific to the exercise and the relevant precision is clinical-grade: the difference between a 10-degree and a 15-degree knee valgus is clinically meaningful and must be annotatable from standard camera footage.

Annotation for clinical AI applications typically requires annotators with physiotherapy, occupational therapy, or exercise physiology qualifications — the same credentialling requirement that applies to clinical document annotation for healthcare NLP. Data handling must comply with the Australian Privacy Act and, for data exported to the US, HIPAA de-identification requirements.

Quality Measurement: OKS, PCKh, and Per-Joint Accuracy

OKS (Object Keypoint Similarity) is the benchmark standard for measuring pose annotation quality, defined in the COCO evaluation protocol. It measures per-keypoint similarity between annotation and reference, weighted by per-joint scale factors based on anatomical size. The resulting score ranges from 0 to 1, where 0.85+ is considered production-quality annotation.

PCKh (Percentage of Correct Keypoints relative to head size) is the MPII benchmark standard, used primarily in research settings. It counts a keypoint correct if it falls within a distance threshold (typically 50% of head length) from the ground-truth position. PCKh@0.5 of 90%+ is the research community's threshold for high-quality annotation.

For domain-specific applications, per-joint accuracy on the most critical joints is the most useful internal metric. An AFL biomechanics model may require ankle OKS of 0.92+ and knee OKS of 0.93+ while accepting lower accuracy at less-important landmarks such as ears or nose. Per-joint accuracy reports from QA audits reveal which joints are driving quality failures — almost always the occluded, fast-moving, or small joints — and focus annotation improvement effort where it matters most.

Related annotation types that often complement pose estimation projects include facial landmark annotation for face-level pose tracking and instance segmentation annotation when precise person boundary separation is required alongside skeleton labelling.

Annotation Guidelines for Pose: Non-Negotiable Definitions

The following questions must be explicitly answered in the annotation specification before any pose annotation work begins. Ambiguity in any of these generates systematic inter-annotator disagreement that no QA process can fully correct:

• What is the anatomical definition of each keypoint? (e.g., shoulder = anterior tip of acromion process, or centre of glenohumeral joint?)
• For occluded joints: what is the placement rule — skip (v=0), estimated position (v=1), or conditional on occlusion percentage?
• For truncated subjects (partially outside frame boundary): which keypoints should be labelled, and how?
• For motion-blurred frames: is the keypoint placed at the estimated centre of the blur streak, or skipped?
• For multiple people: what is the minimum visible body area to initiate annotation on a partially visible person?
• For domain extensions (heel, toe, spine mid): exactly which anatomical landmark does each keypoint reference?

The joint-definition question is more consequential than it appears. "Shoulder" in the COCO annotation protocol references a different anatomical point than "shoulder" in sports biomechanics or clinical physiotherapy. A dataset annotated with COCO shoulder convention cannot be directly merged with one annotated with clinical convention without systematic bias being introduced into the combined training set.

Frequently Asked Questions