Audio Enhancement

Audio enhancement covers noise reduction, upsampling (bandwidth extension), de-reverberation, and restoration of degraded audio. Two models define the local landscape: **Resemble Enhance** (Resemble AI, 2024) for speech denoising + super-resolution, and **AudioSR** (2023/2024) for general audio upsampling including music. Both operate in the frequency domain — converting audio to spectrograms, applying learned transformations, and reconstructing the waveform. Resemble Enhance is the default recommendation for speech enhancement. It performs two operations in sequence: (1) denoising — removes background noise, reverb, compression artifacts via a UNet-based denoiser trained on paired clean/noisy speech at 8-48kHz, and (2) super-resolution — upsamples speech from 8-16kHz to 48kHz by hallucinating high-frequency content (8-24kHz) based on learned speech spectral patterns. On standard benchmarks (VCTK + DEMAND noise at 0dB SNR → denoised), Resemble Enhance achieves PESQ 3.2-3.8 (out of 4.5) and STOI 0.92-0.95 on severe noise. This is competitive with commercial solutions (Adobe Podcast Enhance: PESQ ~3.5-3.9) and substantially better than classical DSP (Wiener filtering: PESQ ~2.0-2.5, spectral subtraction: ~1.8-2.3). AudioSR handles general audio (music + speech + sound effects) upsampling from any input rate to 48kHz. Architecture: latent diffusion on mel-spectrograms — adds noise to low-res spectrogram, denoises toward high-res equivalent. Key advantage over Resemble Enhance: works on music (reveals lost cymbal shimmer, string harmonics, room ambience above 16kHz). Disadvantage: slower (diffusion vs Resemble Enhance's direct prediction), GPUs required, more artifacts on speech (occasional metallic ringing in reconstructed highs). **Classical DSP alternatives** (no AI): RNNoise (Mozilla, LSTM-based, ~1 MB) for real-time noise suppression on CPU — used in Firefox/Chromium WebRTC, handles 48kHz in <1ms per 20ms frame. DeepFilterNet (2022, ~3 MB) for speech enhancement with better quality than RNNoise on non-stationary noise (cafe babble, keyboard typing). Classical DSP is 10-100× faster than AI models and sufficient for 80% of use cases — only use AI enhancement when RNNoise/DeepFilterNet don't deliver. For recommended models, see [Llama 3.3 70B](/models/llama-3-3-70b). Deploy locally with [Ollama](/tools/ollama) on [RTX 4090](/hardware/rtx-4090).

For speech enhancement (podcast cleanup, meeting recording fix, interview restoration), install Resemble Enhance via pip: `pip install resemble-enhance`. It's a Python package with CLI: `resemble-enhance input.wav output.wav`. No GPU required — runs on CPU, 5-20× real-time depending on CPU (a 30-minute recording processes in 1.5-6 minutes on a modern laptop). If you have a GPU, add `--use_gpu` for 2-5× speedup. Your most common use case: "I recorded an interview on my phone and the audio sounds distant and echoey." Resemble Enhance will: (1) reduce room reverb, (2) boost the voice presence (bring it "closer" to the mic), (3) reduce broadband background noise (traffic, HVAC, crowd). It won't fix: clipped/distorted audio (overdriven mic, digital clipping), severe noise where SNR <0dB (noise is louder than speech), or multiple overlapping speakers. For your first test: record yourself speaking from 3+ meters away in a noisy room (kitchen with fridge hum, distant traffic). Run Resemble Enhance. Compare before/after. The improvement is dramatic on this type of audio — distant + noisy → close + clean. The model excels at exactly this use case. If you have music or mixed audio (not just speech): install AudioSR (`pip install audiosr`). Upsample low-bitrate MP3s, recover high frequencies from compressed audio, or enhance old recordings. AudioSR requires GPU (4 GB+ VRAM). Processing speed: 10-30× real-time on [RTX 3060 12GB](/hardware/rtx-3060-12gb). For pure speech, use Resemble Enhance — faster and fewer artifacts on voice. For real-time noise suppression during calls/streaming: install RNNoise or DeepFilterNet via OBS Studio plugins (obs-rnnoise, obs-deepfilter). These run on CPU at <1ms latency, suppress background noise during live recording/streaming, and require no GPU. Not as clean as offline AI processing but real-time capable — the only option for live enhancement. For recommended models, see [Llama 3.3 70B](/models/llama-3-3-70b).

Production audio enhancement requires choosing between real-time (low-latency, lower-quality) and offline (batch, higher-quality) processing, and building quality-gated pipelines for high-volume audio. **Offline enhancement pipeline**: (1) Audio ingest → quality audit (SNR measurement, clipping detection, silence ratio, bandwidth check via spectral analysis) → (2) route by problem type: noise → Resemble Enhance denoiser, low bandwidth → Resemble Enhance super-resolution, reverb → Demucs source separation (isolate direct sound from reverb), compression artifacts → AudioSR → (3) quality gate (PESQ >3.0, STOI >0.85, or MOS prediction >3.5) → (4) loudness normalization (ITU-R BS.1770, -16 LUFS for podcast, -14 for streaming) → (5) delivery. Single [RTX 4090 24GB](/hardware/rtx-4090): ~100-200 hours of speech audio/month (Resemble Enhance at 10-20× real-time). **When AI enhancement vs classical DSP**: Use RNNoise/DeepFilterNet for real-time applications (live streaming, VoIP calls, video conferencing) — sub-1ms latency, CPU-friendly, good enough at SNR >10dB. Use Resemble Enhance for offline improvement of archival content, podcast production, interview cleanup where latency doesn't matter and quality ceiling is higher. Use AudioSR for upsampling old audio libraries (phone recordings at 8kHz → 48kHz restoration), music enhancement, and bandwidth extension. **Quality gates**: (1) PESQ: compare original to enhanced via full-reference quality metric. PESQ <2.5 = enhancement likely degraded audio (algorithm made it worse). Flag for manual review. (2) MOS predictor (DNSMOS or similar): no-reference quality estimation. MOS <2.5 = poor quality, consider re-recording instead of enhancing. (3) Spectral flatness: enhanced audio should have non-flat spectrum (flat = white noise, bad; structured = signal, good). (4) Artifact detection: train a classifier on known AI enhancement artifacts (metallic ringing, warbling, musical noise from spectral subtraction) — auto-reject frames with >80% artifact probability. **Batch processing architecture**: Audio ingest from S3/local storage → problem classifier (SNR, bandwidth, reverb metrics per file) → route to appropriate enhancer (Resemble Enhance for speech-noise, AudioSR for low-bandwidth, classical DSP for real-time files) → parallel GPU workers per file → quality gate → normalized output → storage. Use Celery + Redis for job distribution. 4× [RTX 4070 12GB](/hardware/rtx-4070) handles 400-800 hours/month. **Cost**: Resemble Enhance self-hosted = $0.001-0.003/audio minute (GPU amortized). Adobe Podcast Enhance = free but cloud-only (data leaves your system). Commercial audio restoration services = $0.10-1.00/audio minute. Self-hosting breaks even at ~500+ audio minutes/month vs commercial, with privacy as the primary driver, cost as secondary.

- **Output quality degradation under load.** Symptom: response quality drops under high concurrency — outputs become shorter and less detailed. Cause: [vLLM](/tools/vllm) under GPU memory pressure reduces effective context or switches generation parameters. Mitigation: monitor per-request quality scores, set max concurrency based on memory headroom, scale horizontally when metrics degrade. - **Confidence hallucination.** Symptom: outputs are confident and well-structured but factually wrong. Cause: LLMs generate by probability not factual verification. Confident-sounding outputs are harder to detect as errors. Mitigation: implement deterministic validation on outputs (numeric bounds, schema checks). Flag for human review when validation fails. Never trust LLM self-reported confidence. - **Edge case collapse.** Symptom: model performs well on common Audio Enhancement tasks (90%+) but fails catastrophically on rare edge cases. Cause: training distribution under-represents edge cases; model defaults to nearest common pattern. Mitigation: build regression test suite of edge cases, run before every model update, track edge case pass rate separately from general accuracy. - **Context pollution in long sessions.** Symptom: after 20-30 exchanges the model mixes information from earlier conversation parts, producing contradictions. Cause: long contexts dilute attention — relevant context competes with noise. Mitigation: periodically summarize and reset context every 15-20 exchanges, store key findings externally, trigger summarization at 80% context window capacity. For models see [Llama 3.3 70B](/models/llama-3-3-70b).

**CPU-only ($0)**: Resemble Enhance runs CPU-only on any modern CPU. Processing: 5-20× real-time (a 30-min recording in 1.5-6 min on Apple M4 / Ryzen 9 / Core Ultra 9). AudioSR requires GPU — CPU processing is impractical (50-200× real-time, a 3-min song takes 2.5-10 hours). RNNoise/DeepFilterNet run on any CPU at <1ms per frame — real-time capable on 15-year-old hardware. **Entry GPU (4-8 GB)**: Any GPU with 4 GB+. [RTX 3060 12GB](/hardware/rtx-3060-12gb) accelerates Resemble Enhance to 20-50× real-time (30-min recording in 36-90s). AudioSR runs at 10-30× real-time on 8 GB+. [Apple M2 Max](/hardware/apple-m2-max) via MPS: Resemble Enhance at 10-20× real-time. **SMB/production (8-16 GB)**: [RTX 4060 Ti 16GB](/hardware/rtx-4060-ti-16gb) or [RTX 4070 12GB](/hardware/rtx-4070). Resemble Enhance at 30-60× real-time. AudioSR at 20-50× real-time. Supports concurrent processing of 2-4 audio files on one GPU. **Enterprise throughput (16-24 GB)**: [RTX 4090 24GB](/hardware/rtx-4090) or [RTX 3090 24GB](/hardware/rtx-3090). Resemble Enhance at 50-100× real-time. AudioSR at 30-60× real-time. 4-8 concurrent files. [NVIDIA L40S](/hardware/nvidia-l40s) 48 GB for 8-16 concurrent files — audio processing farm for archival restoration (converting decades of low-quality audio to modern quality). **Memory scaling**: Resemble Enhance: 1-3 GB VRAM. AudioSR: 4-8 GB VRAM. DeepFilterNet: negligible (runs on CPU). Audio enhancement is among the most hardware-accessible AI tasks — even entry-level GPUs deliver excellent throughput because models are small (Resemble Enhance ~100M params, AudioSR ~600M). Past 8 GB VRAM, benefits are purely throughput (more concurrent files), not per-file quality improvement.

**If enhancing speech recordings (podcasts, interviews, calls)** → Resemble Enhance Python CLI or Python API. CLI: `resemble-enhance input.wav output.wav --use_gpu`. Python API: `from resemble_enhance import enhance; enhance(input_path, output_path, solver="ddim", nfe=50, tau=1.0, use_gpu=True)`. Key parameters: `solver` (ddim for speed, dpm++ for quality), `nfe` (number of function evaluations — 25 for draft, 50 for standard, 100 for max quality), `tau` (denoising strength — 0.5 gentle, 1.0 standard, 1.5 aggressive). **If upsampling general audio (music, mixed content)** → AudioSR Python API. `from audiosr import build_model, super_resolution; model = build_model('basic'); super_resolution(model, input_path, output_path, seed=42, guidance_scale=3.5, ddim_steps=50)`. Key parameters: `guidance_scale` (3-7, higher = stronger conditioning on input, lower = more creative high-frequency hallucination), `ddim_steps` (25-100, more steps = better quality, slower). AudioSR works on any audio type but expect speech artifacts — for pure speech, use Resemble Enhance instead. **If you need real-time noise suppression for streaming/calls** → RNNoise (OBS plugin or standalone binary) or DeepFilterNet (OBS plugin or standalone). Both are VST3/AU audio plugins usable in OBS, DaVinci Resolve Fairlight, Audacity. RNNoise: better latency (0.5ms), DeepFilterNet: better quality on non-stationary noise. For critical live broadcasts: chain RNNoise (suppress broadband noise) → DeepFilterNet (remove residual tonal noise) → Compressor (tame dynamics). Real-time DSP chain, no AI post-processing. **If building an automated podcast cleanup pipeline** → Resemble Enhance for noise reduction → FFmpeg for loudness normalization (`loudnorm=I=-16:LRA=11:TP=-1.5`) → Auphonic (commercial, API for final mastering with leveling/autoEQ). Integration: Resemble Enhance Python → FFmpeg loudnorm → upload to Auphonic API for final polish. The AI step handles noise/reverb; traditional tools handle loudness and EQ for consistency. For recommended models, see [Llama 3.3 70B](/models/llama-3-3-70b). Deploy locally with [Ollama](/tools/ollama) on [RTX 4090](/hardware/rtx-4090).