Speaker Diarization

Speaker diarization answers "who spoke when" in multi-speaker audio — it segments an audio stream by speaker identity without knowing who the speakers are in advance. Two models dominate local deployment: **PyAnnote Audio 3.1** (pyannote.audio, the de facto standard) and **NVIDIA NeMo diarization** (NVIDIA, competitive but heavier setup). Neither transcribes speech (that's [speech-to-text](/tasks/speech-to-text)); diarization outputs speaker-labeled time segments: "Speaker A: 0:00-0:12, Speaker B: 0:12-0:18, Speaker A: 0:18-0:30." PyAnnote 3.1 pipeline works in four stages: (1) Voice Activity Detection (Silero VAD or pyannote's own VAD) identifies speech vs silence segments, (2) speaker embedding extraction (ECAPA-TDNN or WavLM model) computes a fixed-length vector for each short audio segment that captures voice characteristics, (3) clustering (Agglomerative Hierarchical Clustering on cosine distances between embeddings) groups segments by speaker, (4) segmentation refinement (Hidden Markov Model smoothing) corrects impossibly short speaker turns (<0.5s). Key metric: Diarization Error Rate (DER) — the sum of missed speech, false alarm speech, and speaker confusion errors, normalized by total speech time. **DER benchmarks by difficulty**: Clean single-mic, 2 speakers (AMI clean condition): DER 3-7%. Meeting audio, 4 speakers, single distant mic (AMI actual meetings): DER 12-22%. Telephone audio, 2 speakers: DER 5-10%. Overlapping speech present (2+ speakers talking simultaneously): DER jumps 10-20 points (pyannote's primary weakness). 8+ speakers: DER 25-40% (clustering fails with many speakers). **NVIDIA NeMo diarization** offers a GPU-optimized alternative with similar DER on standard benchmarks (within 2-5% of PyAnnote depending on condition) but requires CUDA toolkit + NeMo framework (heavier install, ~10 GB dependencies vs pyannote's ~2 GB). NeMo's advantage: scales to multi-GPU for batch processing, and supports multi-channel audio (microphone arrays) where pyannote is single-channel. For most deployments, pyannote is simpler; NeMo for GPU-cluster batch processing. For recommended models, see [Llama 3.3 70B](/models/llama-3-3-70b). Deploy locally with [Ollama](/tools/ollama) on [RTX 4090](/hardware/rtx-4090).

Install PyAnnote Audio 3.1 via pip: `pip install pyannote.audio`. You need a Hugging Face token (free) to download the pretrained models. Create an account at huggingface.co, generate a token, accept the model terms for "pyannote/speaker-diarization-3.1" and "pyannote/segmentation-3.0." Basic Python script for your first diarization: ```python from pyannote.audio import Pipeline pipeline = Pipeline.from_pretrained( "pyannote/speaker-diarization-3.1", use_auth_token="YOUR_HF_TOKEN" ) diarization = pipeline("meeting.wav") for turn, _, speaker in diarization.itertracks(yield_label=True): print(f"{speaker}: {turn.start:.1f}s - {turn.end:.1f}s") ``` Hardware: PyAnnote runs on CPU — no GPU required. A 30-minute meeting processes in 2-5 minutes on a modern laptop CPU (10-25× real-time). GPU acceleration (CUDA) gives 2-3× speedup but isn't necessary for occasional use. The models are small: segmentation model ~100 MB, embedding model ~20 MB. Test with a 2-person conversation first — recorded on your phone in a quiet room, one person speaking at a time, 3-5 minutes. PyAnnote should achieve <5% DER on this easy condition. Then test with 3-4 people in a natural room (some overlap, varying distance from mic) — expect 15-25% DER. This jump in error rate is normal; diarization degrades quickly with speaker count and recording quality. For recommended models, see [Llama 3.3 70B](/models/llama-3-3-70b). Deploy locally with [Ollama](/tools/ollama) on [RTX 4090](/hardware/rtx-4090).

Production diarization requires matching pipeline components to deployment scale, integrating with downstream [speech-to-text](/tasks/speech-to-text) and [dubbing](/tasks/dubbing) pipelines, and implementing accuracy monitoring. **Batch pipeline architecture**: (1) Audio ingest → VAD segmentation (Silero VAD, speech probability threshold ≥0.5, minimum segment 0.5s, maximum 15s) → (2) PyAnnote diarization on speech segments → (3) Overlap detection (identify segments with >1 speaker, flag for degraded performance — overlapping speaker DER is 20-40%) → (4) Optional: re-diarize overlapping segments with spectral clustering (different algorithm, handles overlap better but slower) → (5) Merge diarization output with [faster-whisper](/tools/faster-whisper) transcription → (6) Speaker-labeled transcript + speaker embeddings stored in [pgvector](/tools/pgvector) for speaker retrieval across recordings → (7) DER monitoring (compute on a gold-standard validation set monthly to detect pipeline drift). **Throughput**: PyAnnote on CPU processes ~10-25× real-time. On [RTX 4090 24GB](/hardware/rtx-4090): ~40-80× real-time (GPU primarily accelerates embedding extraction, not clustering). A single GPU server handles ~200-400 hours of audio per day. For call centers processing 10,000+ hours/month: 2-4 GPU servers. **Overlapping speech handling**: PyAnnote 3.1's primary weakness — DER increases 15-30 percentage points when >2 speakers overlap. Three mitigations: (1) Pre-process with speech separation (Demucs or SepFormer) to isolate individual speaker streams before diarization (adds 5-10× processing time but reduces DER on overlaps by 40-60%). (2) Use multi-channel microphone arrays (2-8 mics) — NVIDIA NeMo processes multi-channel audio natively, achieving 30-50% lower DER on overlapping speech by exploiting spatial separation. (3) Accept overlap errors and flag segments for human diarization in quality-critical workflows. **Speaker counting uncertainty**: Most clustering algorithms need a known or estimated speaker count. Overestimating by 1-2 speakers: mild DER increase (2-5%). Underestimating by 1-2: severe (10-20% DER) — two speakers merged into one. Strategy: overestimate speaker count by 20-50% (if you think 4 speakers, tell the model 6-8), then post-process by merging adjacent segments with cosine similarity >0.7 (re-merging over-split speakers). This reduces the underestimate penalty while accepting a small overestimate penalty. **DER by use case and acceptance thresholds**: Legal/courtroom transcripts: DER must be <5% (human-verified diarization). Call center analytics: DER <15% acceptable (trends over hundreds of calls compensate for individual errors). Meeting notes: DER <20% acceptable (attendees correct speaker labels). Podcast transcription: DER <10% desired (listeners notice speaker confusion). If your use case is legal/courtroom, plan for human diarization on all content — AI diarization isn't reliable enough.

**Speaker confusion in same-gender groups.** Symptom: two similar-sounding speakers (same gender, similar age, similar pitch) are frequently confused — Speaker A labeled as Speaker B for 20-30% of turns. Cause: speaker embeddings collapse when voice characteristics overlap — ECAPA-TDNN embeddings have limited intra-gender discrimination. Mitigation: use WavLM embeddings (newer, better discrimination) instead of ECAPA-TDNN, or add video-based speaker identification (face recognition from webcam) as supplementary signal — face + voice strongly reduces same-gender confusion. **Short-turn segmentation failure.** Symptom: quick back-and-forth dialogue ("Yes." — "No." — "Maybe.") is segmented as one long turn from one speaker. Cause: pyannote's segmentation minimum turn length (default ~1s) is longer than these utterances. Mitigation: reduce minimum segment length to 0.3s (increases false speaker changes but catches quick turns), or use a specialized word-level diarizer (TS-VAD, but not well-supported locally) that operates at word-timestamp granularity. **Music/sound as "speaker."** Symptom: background music, phone ringtones, and door slams are assigned speaker labels ("Speaker C: 12:00-12:02" is actually a phone ringing). Cause: the VAD doesn't distinguish speech from non-speech sound — it detects "audio energy" as speech. Mitigation: add a speech/music classifier post-VAD (pre-trained on AudioSet) to filter non-speech segments before diarization, or use Silero VAD's speech probability threshold (≥0.7 is conservative — fewer false positives, more false negatives). **Clustering failure on long recordings.** Symptom: 2-hour meeting with 6 speakers — clustering produces 14 speaker labels (over-segmentation). Cause: the same speaker's voice varies across 2 hours (fatigue, emotion change, varying distance from mic) — embeddings drift, and clustering sees multiple clusters for one person. Mitigation: apply speaker embedding smoothing across time (3-minute rolling average of embeddings within each segment's temporal neighborhood) — reduces intra-speaker variance while preserving inter-speaker differences.

**CPU-only ($0)**: Modern CPU (Apple M4, Ryzen 9 7950X, Core Ultra 9). PyAnnote processes audio at 10-25× real-time. GPU not required — diarization's bottleneck is clustering (CPU-bound, not GPU-accelerated). A 1-hour recording processes in 2.5-6 minutes on CPU. Sufficient for <10 hours/week of audio. **Entry GPU (any 4 GB+)**: Any GPU with 4 GB+ VRAM accelerates embedding extraction (2-3× faster than CPU) but clustering remains CPU-bound. [RTX 3060 12GB](/hardware/rtx-3060-12gb): 20-50× real-time overall. [Apple M2 Max](/hardware/apple-m2-max): 15-30× real-time via MPS. GPU provides marginal benefit for diarization compared to GPU-dependent tasks. **SMB/throughput (8-16 GB)**: For batch processing. [RTX 4060 Ti 16GB](/hardware/rtx-4060-ti-16gb): 2-4 concurrent diarization pipelines. [RTX 4070 12GB](/hardware/rtx-4070): same. The VRAM headroom allows concurrent embedding extraction; CPU handles concurrent clustering. **Enterprise/scale (16-48 GB)**: [RTX 4090 24GB](/hardware/rtx-4090) or [NVIDIA L40S](/hardware/nvidia-l40s). 4-8 concurrent pipelines. For call centers processing 100,000+ hours/month: 4-8 GPU servers horizontally scaled. NVIDIA NeMo scales better to multi-GPU than pyannote (NeMo supports distributed processing of a single audio file across GPUs; pyannote processes one file per GPU). **Memory scaling**: PyAnnote pipeline (segmentation + embedding + clustering): 1-3 GB RAM, 0-2 GB VRAM — extremely lightweight. NeMo full pipeline: 4-12 GB VRAM. Neither model needs more than 8 GB VRAM. This is one of the most hardware-accessible AI tasks — diarization is bottlenecked by algorithm quality, not hardware capability.

**If you want diarization + transcription in one tool** → WhisperX (`pip install whisperx`). Runs faster-whisper for transcription + pyannote for diarization, outputs word-level timestamps with speaker labels. Single command: `whisperx audio.wav --model large-v3 --diarize --hf_token YOUR_TOKEN --output_format json`. Output: JSON with words, timestamps, speaker labels. The default choice for most operators — no manual pipeline chaining needed. Hardware: CPU for diarization, GPU for transcription. On [RTX 4090](/hardware/rtx-4090): ~30-60× real-time end-to-end. **If you need standalone diarization (no transcription)** → PyAnnote Python pipeline. Use when transcription is provided by another system, or when you only need "who spoke when" without what they said. `Pipeline.from_pretrained("pyannote/speaker-diarization-3.1")`. Then `pipeline(audio, num_speakers=N)` — specify num_speakers if known, or omit for automatic estimation. Integrate with downstream [faster-whisper](/tools/faster-whisper) or [whisper.cpp](/tools/whisper-cpp) for transcription. **If processing very long recordings (2+ hours)** → Pre-segment long audio into 15-30 minute chunks (at natural silence boundaries via VAD). Diarize each chunk independently, then merge speaker labels across chunks via embedding matching (cosine similarity between chunk-edge speaker embeddings >0.75 = same speaker). This prevents the clustering degradation that occurs on very long recordings and enables parallel processing. Chunk-and-merge diarization: ~20-30% faster than monolithic, with 3-5% DER penalty from chunk-edge speaker label mismatches. **If you have overlapping speech as a primary concern** → Add speech separation pre-processing. Demucs (htdemucs_ft) or SepFormer separates mixed audio into individual speaker streams. Pipeline: mixed audio → speech separation → 3-4 isolated speaker tracks → diarize each track independently (trivially, since each track is single-speaker) → merge results. This adds 5-10× processing time but reduces overlapping-speech DER from 30-40% to 10-15%. Use for quality-critical applications where accuracy matters more than speed.