Whisper

OpenAI's Whisper family is the dominant open-weight speech-to-text architecture, spanning four major iterations since September 2022. All Whisper variants share a common encoder-decoder [Transformers](/tools/transformers) architecture trained on 680,000 hours of weakly supervised multilingual speech data — a scale that remains unmatched in open-weight STT. The encoder processes 80-channel log-Mel spectrogram frames (25ms window, 10ms stride) into audio embeddings. The decoder is an autoregressive Transformer that generates text tokens conditioned on the encoder output, with a multi-task training format that jointly learns language identification, timestamp prediction, and transcription within a single sequence. Whisper v1 (September 2022, sizes tiny/base/small/medium/large at 39M/74M/244M/769M/1550M parameters) established the architecture and demonstrated the "weak supervision works at scale" finding: training on noisy, web-scraped audio with automatically generated transcripts produced a model competitive with supervised LibriSpeech-trained models on zero-shot benchmarks. The v1 models were English-optimized despite multilingual training data — non-English languages received roughly 1/3 of English's token allocation in the training mixture. Whisper v2 (December 2023, large-v2) increased the decoder width from 1280 to 1500 dimensions and retrained on an improved data mixture with 30% more non-English audio. The v2 model improved non-English word error rate (WER) by 15–25% across 50+ languages while maintaining English WER parity. Whisper v3 (June 2024, large-v3) further expanded the decoder to 1600 dimensions, increased the vocabulary from 51,864 to 69,328 tokens (better coverage for CJK and Indic scripts), and introduced Flash Attention 2 for 40% faster inference. [Whisper large-v3](/families/whisper)-v3-turbo (November 2024) is the current frontier: a large-v3 model distilled to 809M parameters (matching the medium size class) that achieves large-v3-quality transcription at 2× the speed. The distillation uses speculative decoding — the smaller model generates draft tokens, and the frozen large-v3 teacher verifies and corrects — during training only. At inference time, large-v3-turbo is a standalone 809M model that matches large-v3's 5.7% English LibriSpeech WER and achieves <8% average WER across the top 10 languages. This is the operational sweet spot for most deployments.

Whisper is the most locally-viable STT family by a wide margin. Whisper tiny (39M parameters, ~150 MB FP16) runs in real-time (>10× faster than real-time) on any CPU from the last decade — a Raspberry Pi 4 transcribes audio at 2–3× real-time using whisper.cpp. Whisper base (74M, ~280 MB) provides the best accuracy-per-compute ratio for edge devices and mobile deployment (on-device transcription on [iPhone 16 Pro](/hardware/iphone-16-pro) via Core ML at 5–8× real-time). Whisper medium (769M, ~1.5 GB FP16) is the VRAM-friendly production sweet spot. It runs fully on an [RTX 3060 12GB](/hardware/rtx-3060-12gb) at 40–60× real-time (a 1-hour audio file transcribes in ~60–90 seconds), or CPU-only at 3–5× real-time on a modern laptop. Medium achieves ~85% of large's multilingual accuracy with 1/4 the compute — sufficient for transcription of clear speech in consumer and SMB applications. The [MacBook Pro 16" M4 Max](/hardware/macbook-pro-16-m4-max) transcribes via whisper.cpp's Metal backend at 50–80× real-time for medium and 20–30× real-time for large-v3 — laptop-class transcription throughput matching desktop GPUs. Whisper large-v3 (1.55B, ~3.1 GB FP16) on an [RTX 4060 Ti 16GB](/hardware/rtx-4060-ti-16gb) transcribes at 20–30× real-time. Large-v3-turbo (809M, ~1.6 GB) achieves the same quality at 40–60× real-time on the same hardware — essentially, you get large-v3 accuracy at medium speed. This is the recommended configuration for production STT on consumer hardware: large-v3-turbo via whisper.cpp with CUDA acceleration. Multi-GPU scaling is unnecessary for Whisper — even large-v3 at FP16 fits in 3.1 GB VRAM, well within any dedicated GPU's capacity. The Whisper workload is embarrassingly parallel across audio files (batch-process independent files across multiple GPUs), not within a single file. A server with 4× [RTX 4090](/hardware/rtx-4090) can process 4 concurrent audio streams at 20–30× real-time each — sufficient throughput for a call center with hundreds of simultaneous calls. The primary local viability constraint is language coverage, not hardware. Whisper supports 99 languages, but quality varies dramatically: English WER is ~3–5%, Spanish ~5–7%, Mandarin ~10–12%, and low-resource languages (Welsh, Swahili, Urdu) can exceed 30% WER. For deployment in a specific language, validate Whisper's WER on your domain's audio before committing to the model family.

Whisper's benchmark dominance is built on its zero-shot generalization. On the LibriSpeech clean test set (English read speech), [Whisper large-v3](/families/whisper)-v3 achieves 2.7% WER — within 0.5 percentage points of supervised SOTA models trained specifically on LibriSpeech. On LibriSpeech "other" (noisy speech), large-v3 achieves 5.7% vs the supervised Whisper fine-tune benchmark of 5.0% — the zero-shot model performs within 15% of a dataset-specific fine-tune. This "good enough out of the box" quality, servable locally via [llama.cpp](/tools/llama-cpp), drove mass adoption. Multilingual benchmarks illustrate Whisper's uneven coverage. On Fleurs (102-language speech benchmark), Whisper large-v3 achieves <10% WER for 32 languages (primarily European + Mandarin/Japanese/Korean), 10–20% for 35 languages, and >20% for 31 languages. The WER correlates strongly with training data volume — languages with >5,000 hours of training data achieve <10% WER. Low-resource language performance is the family's primary quality limitation. On the challenging CORAAL (African American Vernacular English) and Common Voice accent benchmarks, Whisper large-v3's WER is 2–3× higher than on standard American English — 12–15% vs 3–5%. This accent generalization gap is a well-documented bias in Whisper's training data (skewed toward broadcast-standard pronunciation) and has spawned a community of accent-specific Whisper fine-tunes. On noisy/reverberant speech benchmarks (CHiME-5, REVERB), Whisper large-v3 degrades more gracefully than supervised models trained on clean speech only — the weak supervision training on diverse web audio provides implicit robustness. CHiME-5 WER for large-v3 is 42.1% vs 55.3% for supervised baselines — still high in absolute terms but 13 points better than clean-speech-trained alternatives. Whisper large-v3-turbo's benchmark performance is the quality-per-compute story of 2024–2025. On LibriSpeech clean, turbo achieves 2.9% WER (vs large-v3's 2.7%) — a 0.2-point regression for a 2× speedup. On Fleurs multilingual average, turbo achieves 14.2% vs large-v3's 13.8% — a 0.4-point regression. For most applications, the speed gain outweighs the negligible quality loss, and the gap narrows further with language-specific fine-tuning — distilled models often match the teacher's accuracy, a pattern also seen in [Phi](/families/phi)'s distillation lineage.

Whisper's runtime ecosystem is the broadest in STT. whisper.cpp (C++ inference, maintained by ggerganov) is the gold-standard runtime: supports all Whisper variants, all major GPU backends (CUDA, Metal, Vulkan, OpenCL, ROCm, Core ML), all operating systems (Windows, macOS, Linux, iOS, Android, WebAssembly), and runs on hardware from Raspberry Pi to [H100](/hardware/nvidia-h100-pcie). whisper.cpp is the deployment path for edge devices, mobile apps, and embedded systems — it's the primary reason Whisper is the dominant local STT model. The project ships pre-built binaries for Windows/macOS/Linux and has language bindings for Python, Node.js, Go, Rust, and Swift. [faster-whisper](https://github.com/SYSTRAN/faster-whisper) (CTranslate2 backend) achieves 3–4× faster transcription than the reference PyTorch implementation on NVIDIA GPUs. It supports INT8 quantization for the encoder and decoder, reducing large-v3 VRAM from 3.1 GB to ~1.2 GB with <0.5% WER regression. faster-whisper is the production choice for GPU-accelerated Whisper serving — it's the default backend in most Whisper API wrappers (WhisperX, whisper-api, openai-whisper-server). [Transformers](/tools/transformers) Whisper support provides the reference implementation with Hugging Face model hub integration. The WhisperForConditionalGeneration pipeline handles all variants, languages, and task modes (transcribe, translate to English). The Transformers implementation is accuracy-optimized (matching OpenAI's reference) but throughput is 3–4× slower than faster-whisper due to PyTorch overhead. Whisper has no [Ollama](/tools/ollama) or [vLLM](/tools/vllm) integration — these runtimes target text generation models. Whisper serving uses either: (1) faster-whisper behind a FastAPI REST server, (2) whisper.cpp as a subprocess called from application code, or (3) the OpenAI-compatible Whisper API (OpenAI's hosted service, not local). The WhisperX library adds word-level timestamps, speaker diarization, and voice activity detection (VAD) on top of faster-whisper — it's the de facto production pipeline for meeting transcription, podcast indexing, and call center analytics.

Whisper quantization spans INT8 (faster-whisper's default) through 2-bit GGUF (whisper.cpp), with the encoder and decoder quantized separately due to their different precision sensitivities. The encoder processes the entire audio input in a single forward pass — its quantization sensitivity is low because errors don't compound autoregressively. Encoder INT8 reduces VRAM by ~50% with <0.1% WER impact. Encoder Q5_K_M reduces VRAM by ~60% with ~0.5% WER impact — the practical minimum for most use cases. The decoder is autoregressive (generates one token at a time conditioned on previous tokens), making it ~5× more quantization-sensitive than the encoder. Decoder INT8 produces <0.3% WER regression. Decoder Q5_K_M introduces ~1–2% WER regression — still acceptable for non-critical transcription. Q4 introduces 3–5% WER regression and occasional token repetition loops (the decoder gets stuck in a token loop due to quantization-induced probability distribution distortion). Q4 is acceptable for keyword-spotting and language identification but not for transcription where WER matters. whisper.cpp's GGUF quantization supports the full range from Q2_K through Q8_0 across all Whisper sizes. The Q5_K_M profile (encoder Q5 + decoder Q5) is the community-standard balance: large-v3 at Q5_K_M is ~1.5 GB (vs 3.1 GB FP16), transcribes at ~80% of FP16 speed, and achieves within 1–1.5% of FP16 WER. On an [RTX 3060 12GB](/hardware/rtx-3060-12gb), Q5_K_M large-v3 transcribes at 25–35× real-time — real-time transcription for live audio streams with headroom. Whisper quantization's unique characteristic: the impact is heavily language-dependent. High-resource languages (English, Spanish, French) show <1% WER regression at Q5_K_M vs FP16. Low-resource languages (Swahili, Welsh, Urdu) show 3–8% regression at the same quantization level because the model's parameter precision for low-resource token representations is already marginal at FP16 — quantization further degrades an already-weak signal. Language-specific quantization selection (Q8 for low-resource languages, Q5 for high-resource) is the operational best practice.

Whisper's ecosystem is the most mature in STT, comparable to [Stable Diffusion's](/families/stable-diffusion) position in image generation. The whisper.cpp project alone has 35,000+ GitHub stars, 200+ contributors, and implementations across 12 programming languages. The ecosystem layers from low-level inference (whisper.cpp, faster-whisper) through processing pipelines (WhisperX for diarization, WhisperLive for real-time streaming) to application frameworks (OpenAI-compatible API servers, Nextcloud integration, Jellyfin subtitle generation). The Whisper fine-tune ecosystem is extensive. The most popular fine-tunes cover: Japanese (kotoba-whisper, 4.8% Japanese WER vs base Whisper's 9.2%), medical terminology (whisper-medical, specialized vocabulary), and multi-accent English (distil-whisper accents). Each fine-tune is a drop-in replacement — same Architecture, same API, same runtime — making Whisper fine-tunes operationally trivial to adopt. The tooling ecosystem for post-processing is comprehensive: WhisperX for word-level timestamps and speaker diarization, pyannote.audio for speaker identification, stable-ts for timestamp refinement, and whisper-ctranslate2 for production serving. These tools compose into a pipeline: audio → VAD (segment audio) → Whisper (transcribe segments) → WhisperX (word timestamps) → pyannote (speaker labels) → formatted output. This pipeline serves ~80% of production STT use cases with the default model and libraries. The application-layer ecosystem includes: Otter.ai alternatives (MacWhisper, Buzz, Whisper Transcription), home assistant integration (Whisper for Home Assistant voice commands), accessibility tools (live captioning in VLC, browser extensions), and developer tools (Cursor/Copilot voice coding via Whisper). The breadth of integrated applications means Whisper is the "Intel Inside" of local STT — users may not know they're using Whisper, but they're using Whisper. The ecosystem gap is real-time streaming STT. Whisper's encoder-decoder architecture processes complete audio segments, not streaming audio frames — it transcribes "after the fact" rather than "as you speak." Real-time streaming Whisper requires segment-chunking with VAD (transcribe every 0.5–2s of audio) which introduces latency and potential word-boundary artifacts. For true streaming STT, dedicated streaming architectures (Deepgram, AssemblyAI, [whisper.cpp's](/tools/llama-cpp) streaming mode) remain necessary.

Whisper fine-tuning is the most accessible in STT. A language-specific Whisper fine-tune on 50–200 hours of transcribed audio (medium or large-v3 base) requires 8–16 GB VRAM and 4–12 hours on an [RTX 4090](/hardware/rtx-4090) using Hugging Face's Seq2SeqTrainer. The fine-tuning recipe is well-documented: freeze the encoder for the first 30% of training, train the decoder end-to-end for the remainder, use a low learning rate (1e-5 to 5e-5), and apply SpecAugment for robustness. The Hugging Face Whisper fine-tuning guide provides a Google Colab notebook that completes a language-specific fine-tune in ~2 hours (with Colab's T4 GPU). Domain-specific Whisper fine-tunes (medical, legal, technical) converge in 10–50 hours of in-domain audio. The key finding: fine-tuning on as little as 10 hours of domain-specific audio with a frozen encoder and LoRA-small (rank-4 on the decoder cross-attention layers) improves domain WER by 20–40% with only ~10 MB of trainable parameters. This makes Whisper domain adaptation economically viable for small businesses — a solo practitioner with a laptop can fine-tune Whisper on their industry's vocabulary in a weekend. The Distil-Whisper project (Hugging Face) provides knowledge-distilled Whisper variants that match large-v2 quality at 5.8× speedup and 51% fewer parameters. The distillation recipe is published and reproducible — operators can distill their own Whisper fine-tunes for speed. The distilled model achieves 3.9% WER on LibriSpeech clean vs large-v2's 4.2% (the student actually outperforms the teacher on this benchmark due to the distillation process filtering teacher errors). Distil-[Whisper large-v3](/families/whisper)-v3 is in development, targeting the large-v3 quality tier at medium-size speed. The Whisper fine-tuning ecosystem's limitation: evaluating fine-tunes requires language-specific test sets that most fine-tuners don't have. The typical pattern is: fine-tune on available data, subjectively evaluate on 10–20 samples, deploy. This produces fine-tunes that overfit to the training domain and degrade on out-of-domain audio — a persistent quality issue in community Whisper fine-tunes. The Hugging Face Open STT Leaderboard (evaluating fine-tunes on Common Voice + Fleurs) partially addresses this, but coverage of non-European languages and specialized domains is sparse.

Whisper's "hallucination on silence" behavior is the most common production issue when self-hosting with [llama.cpp](/tools/llama-cpp)'s whisper backend. The model was trained to always produce text output, and silence was underrepresented in the 680K-hour training set. Mitigation: prepend a VAD (voice activity detection) stage that segments audio into speech and non-speech regions, passing only speech segments to Whisper. The Silero VAD model (MIT license, ONNX runtime, <1ms per frame) is the standard companion to Whisper deployments, available alongside [Transformers](/tools/transformers) pipelines. Timestamp accuracy is inconsistent across Whisper variants. The timestamp prediction (trained as a secondary task alongside transcription) is coarser than dedicated timestamp models — Whisper timestamps are accurate to ±0.5–1.0 seconds depending on speaking rate and audio quality. For applications requiring word-level timestamps (captioning, karaoke, forensic analysis), Whisper's native timestamps are insufficient. The WhisperX library's forced alignment post-processing (using a phoneme-level ASR model) improves timestamp accuracy to ±50ms — essential for production captioning. Budget 1.5–2× the transcription time for WhisperX-style alignment post-processing. Language auto-detection is reliable for the top 20 languages (>95% accuracy) but degrades significantly for low-resource languages (<70% accuracy for the bottom 30 languages). Whisper's language detection is based on the first 30 seconds of audio — short utterances (<5 seconds) and code-switching (multiple languages in one utterance) are common failure modes. Mitigation: explicitly set the language parameter when the language is known, bypassing auto-detection entirely. For multi-language audio (e.g., a Spanish tourist speaking English-accented Spanish), expect degraded transcription quality — code-switching was underrepresented in training data. Whisper's output formatting is unpredictable across variants. The model sometimes adds punctuation, sometimes doesn't; sometimes normalizes numbers ("123" vs "one hundred twenty-three"), sometimes doesn't; and the behavior varies across model sizes and quantization levels. This is a pervasive quality-of-life issue that every Whisper deployment eventually encounters and solves with a ~50-line post-processing script.