Speech Processing

Speech processing refers to the analysis, synthesis, and manipulation of human speech by AI models. Operators encounter it when running automatic speech recognition (ASR) to transcribe audio, text-to-speech (TTS) to generate spoken output, or voice activity detection (VAD) to segment audio streams. On local hardware, these tasks are typically handled by specialized models like Whisper (ASR) or Coqui TTS, which must fit within available VRAM and meet real-time latency requirements. Speech processing pipelines often combine multiple models—e.g., VAD to isolate speech, ASR to transcribe, then a language model to interpret—each adding latency and memory overhead.

Speech processing encompasses several sub-tasks: automatic speech recognition (ASR) converts audio waveforms into text; text-to-speech (TTS) generates audio from text; voice activity detection (VAD) identifies speech segments; and speaker diarization assigns speech to different speakers. Modern local approaches use transformer-based models like Whisper (ASR) or VITS (TTS), which run on GPU or CPU via llama.cpp, Ollama, or Hugging Face Transformers. Operators must consider model size—Whisper large-v3 requires ~3 GB VRAM—and real-time factor (RTF), where RTF < 1 means faster than real-time. Quantization (e.g., Q4_0) reduces memory but may degrade accuracy. Latency is critical for interactive use; batch processing (e.g., transcribing a recording) can tolerate higher latency. Speech pipelines often chain VAD + ASR + optional LLM for command understanding, each step adding latency and memory pressure.

An operator runs Whisper large-v3 via whisper.cpp on an RTX 3060 (12 GB VRAM). The model takes ~3 GB, leaving room for a 30-second audio buffer. Transcribing a 1-hour lecture at Q4 quantization yields ~30 minutes wall-clock time (RTF ~0.5). If the operator switches to Whisper medium (1.5 GB), RTF drops to ~0.2 but word error rate increases from 5% to 10% on accented speech.

In a local voice assistant pipeline, the operator uses Silero VAD (via silero-vad Python library) to detect speech segments, then pipes audio chunks to Whisper.cpp for transcription. The transcribed text is sent to a local LLM (e.g., Llama 3.1 8B via Ollama) for intent parsing. The LLM response is fed to Coqui TTS for spoken output. Each step is monitored for latency: VAD < 50ms, ASR ~200ms per 5-second chunk, LLM ~1s, TTS ~300ms. If total latency exceeds 2s, the operator may switch to a smaller ASR model or quantize the LLM to Q4_K_M.