Speaker Diarisation and Audio-Event Annotation for Voice AI

Two Distinct Tasks: Diarisation vs Audio Event Detection

Voice AI annotation encompasses two related but distinct labelling tasks that are often conflated in project scoping — to the detriment of both.

Speaker diarisation annotation answers the question "who spoke when?" It produces a timeline of speaker turns — typically in RTTM (Rich Transcription Time Marked) format — where each segment is labelled with a speaker ID, an onset timestamp, and a duration. The speaker IDs are relative to the recording (Speaker A, Speaker B) and must be consistent across the full file. For multi-session diarisation, IDs may need to be consistent across multiple recordings of the same speakers.

Audio event annotation answers the question "what sounds were present and when?" It produces onset/offset timestamps for non-speech acoustic events — ringing phones, background music, hold tones, machinery noise, applause — each labelled with an event class from a predefined taxonomy. Audio event annotation is used to train acoustic event detection (AED) models for smart speakers, contact-centre telephony analytics, environmental monitoring, and industrial fault detection.

The two tasks share annotation infrastructure (timeline-based tools, timestamp precision requirements) but require different annotator skills and produce different output schemas. Our audio annotation services cover both — and the first step on any voice AI annotation project is determining which task or combination of tasks the model actually needs.

What Makes Diarisation Annotation Hard

Speaker diarisation is one of the perceptually harder annotation tasks in audio. Unlike transcription, where the target is the words, diarisation requires annotators to make boundary decisions at the millisecond level — where exactly does one speaker's turn end and another's begin? — while simultaneously tracking speaker identity across minutes or hours of audio.

Three factors drive most annotation errors:

• Boundary precision — annotators disagree on where one speaker's turn ends and another's begins, particularly during overlapping speech, false starts, and back-channels. A 2022 analysis of the VoxConverse diarisation benchmark found that even expert annotators showed mean boundary disagreement of 210ms on clean telephone audio — wide enough to affect DER by 3–5 percentage points on multi-speaker recordings.
• Speaker ID consistency — on long recordings (30 minutes or more), annotators drift in their assignment of speaker IDs, particularly when a speaker who was quiet for several minutes re-enters the conversation. Consistency errors inflate speaker confusion error in the downstream model.
• Overlapping speech handling — most annotation tools are designed for single-channel, single-speaker timelines. Representing simultaneous speech from multiple speakers requires multi-track workflows, and inter-annotator agreement on overlap onset/offset is consistently lower than for clean turn-taking segments.

A 2023 study published in the IEEE/ACM Transactions on Audio, Speech, and Language Processing compared diarisation annotation from three annotator types — expert linguists, trained non-linguist annotators, and crowdsourced annotators — on 120 hours of telephone conversation data. Expert linguists achieved median DER of 4.8% on clean audio. Trained non-linguists achieved 7.3%. Crowdsourced annotators achieved 19.4% — more than four times higher error. The primary driver of the crowdsourced gap was speaker ID inconsistency on long recordings and failure to mark overlapping speech.

Case Study: Contact-Centre Diarisation for Australian Banking

A major Australian financial services group engaged AI Taggers to build diarisation training data for their contact-centre analytics platform. The existing diarisation system — a pre-trained model without domain fine-tuning — achieved 17.3% DER on their recorded call corpus. The primary failure modes were speaker confusion on accented Australian English and incorrect turn attribution on customer calls with background hold music.

The annotation scope was 480 hours of telephony recordings across four call types: inbound retail banking, inbound home loan, outbound collections, and internal escalation calls. Each recording was annotated by two trained annotators in ELAN, with RTTM export, and adjudicated by a third annotator on segments with boundary disagreement greater than 300ms. Annotators received a four-hour briefing on the bank's specific call structure — including the pattern of agent hold notifications ("I'll just place you on hold") that immediately precede hold music — before annotation began.

Results after fine-tuning on the annotated corpus:

• DER dropped from 17.3% to 6.8% overall
• Speaker confusion error dropped from 11.2% to 3.4% — the primary source of previous failures
• False alarm rate on hold-music segments dropped from 8.7% to 1.2%
• Downstream sentiment analysis accuracy (which depends on correctly attributed speaker turns) improved from 71.3% to 88.9% on agent-sentiment classification

The 480-hour annotation project took nine weeks with a team of six annotators and two senior adjudicators. The improvement in downstream sentiment accuracy — from 71.3% to 88.9% — was the metric the business had been targeting for two years with model architecture changes alone.

The Annotation Pipeline: From Raw Audio to Validated RTTM

A production diarisation annotation pipeline typically runs in four stages:

1. Pre-segmentation — a voice activity detection (VAD) model segments the audio into speech and non-speech regions, producing a draft timeline that annotators refine rather than create from scratch. Pre-segmentation reduces annotation time by 30–40% but requires annotators to check and correct VAD errors, particularly at the boundaries of overlapping speech and at low signal-to-noise segments.
2. Speaker turn labelling — annotators assign speaker IDs to each speech segment, marking turn boundaries and flagging overlapping speech. For recordings with more than three speakers, a speaker roster (listing known speakers with brief audio examples from each) is provided to help annotators maintain ID consistency.
3. Overlap and back-channel marking — a second annotation pass specifically reviews segments where the waveform indicates simultaneous energy from multiple sources. Overlap segments are marked with a dedicated overlap flag and assigned speaker IDs to all concurrent speakers.
4. Adjudication and QA — disagreements between annotators are resolved; a sample of 5–10% of the corpus is double-annotated by a senior annotator as a gold check; DER is computed on the gold sample to confirm the annotation meets the agreed IAA threshold before the dataset is released.

Audio Event Detection Annotation: Taxonomies and Timestamps

Audio event detection annotation follows a similar pipeline but uses event-class taxonomies rather than speaker IDs. The most widely used reference taxonomy for general-purpose AED annotation is AudioSet (Google, 2017), which provides a hierarchical ontology of 632 audio event classes. For domain-specific annotation, custom taxonomies are typically narrower — a contact-centre AED system might use 12–20 event classes covering hold music, ringing, background speech, keyboard noise, and silence — and annotators must be briefed on the specific taxonomy before annotation begins.

Unlike speaker diarisation, audio event annotation often involves strong labelling (onset and offset timestamps for each event instance) and weak labelling (clip-level binary presence/absence labels without timestamps). Strong labels are more expensive but train better localisation models. Weak labels are cheaper and sufficient for classification tasks that don't need event localisation.

Polyphonic event annotation — where multiple event classes may be simultaneously present — requires multi-track annotation interfaces and explicit handling of the case where one event class masks another. This is the audio equivalent of overlapping speech in diarisation, and it produces similar inter-annotator agreement challenges when guidelines don't specify the minimum event duration to annotate and how to handle masked events.

Cost and Throughput Benchmarks

Multilingual annotation adds a language-loading factor: annotators who are not native speakers of the recorded language are slower to make turn-boundary decisions because they cannot rely on prosodic and syntactic cues the same way native speakers can. For Australian English recordings, annotators fluent in Australian English are significantly more accurate on boundary detection than annotators working from phonetic cues alone.

Our audio annotation services include multilingual diarisation for Arabic, Mandarin, Hindi, and 40+ other languages — see also our multilingual speech transcription annotation guide for ASR-focused requirements.

QA and Validation: How to Know Your Diarisation Data Is Good

Quality control for diarisation annotation is different from QA for transcription or image annotation, because the primary quality signal is the IAA DER — the diarisation error rate computed between two independent annotators on the same recording — rather than a human-readable label you can inspect visually.

Standard QA protocols for production diarisation annotation:

• Double annotation of 10–15% of the corpus — compute IAA DER on this gold sample. If IAA DER exceeds the agreed threshold, the batch fails QA and the annotators receive additional calibration.
• Speaker ID consistency audit — automatically check that speaker IDs used in each annotated file are consistent with the known speaker roster (for constrained-speaker scenarios) or that the number of unique IDs matches the expected speaker count.
• Boundary jitter analysis — flag segments where annotator A's boundary and annotator B's boundary differ by more than 500ms for senior review. Systematic boundary jitter on specific audio conditions (e.g. hold music onset/offset) indicates a guideline gap.
• Model-in-the-loop validation — run the annotation batch through a pre-trained diarisation model and flag segments where the annotation disagrees with the model prediction by more than 2 seconds. These are likely annotation errors or genuinely ambiguous segments that need adjudication.

For a broader view of annotation quality validation methods, see our guide on how to validate annotation quality before it reaches your model — the gold-set and audit approaches apply directly to audio annotation.

Frequently Asked Questions