13. Streaming Video

Streaming video processing introduces latency constraints that batch processing architectures cannot satisfy. Real-time video pipelines require careful orchestration of frame ingestion, model inference, and output rendering. This chapter covers the engineering fundamentals of building video streaming systems that maintain consistent frame rates while executing multimodal inference.

The basic streaming architecture uses a producer-consumer pattern where video frames enter a queue at the capture rate and inference workers consume frames for processing. Python's queue.Queue works for development, but production systems require lock-free ring buffers implemented in C++ or CUDA for minimal latency overhead.

import numpy as np
from collections import deque
import threading

class VideoStreamBuffer:
    def __init__(self, max_frames=30):
        self.buffer = deque(maxlen=max_frames)
        self.lock = threading.Lock()
    
    def push(self, frame):
        with self.lock:
            self.buffer.append(frame.copy())
    
    def get_latest(self, n=1):
        with self.lock:
            if len(self.buffer) < n:
                return None
            return [self.buffer[i] for i in range(-n, 0)]

Frame dropping becomes necessary when inference time exceeds the frame budget. A naive approach drops every nth frame, but adaptive strategies monitor queue depth and increase drop rate when backlog grows. The critical failure mode occurs when model inference time has high variance—processing 10 frames at 100ms each followed by 10 frames at 20ms each creates temporal aliasing artifacts.

Zero-copy frame passing between pipeline stages eliminates memory bandwidth bottlenecks. Using CUDA Unified Memory with cudaMemcpyAsync transfers frame data directly to GPU memory without staging through CPU RAM. FFmpeg's libavcodec provides hardware-accelerated decode that can output directly to CUDA surfaces.

Local verification checkpoint

Run the smallest example from this chapter in a local workspace and record the package version, runtime, data path, and observed output. If the result depends on model size, vector count, CPU/GPU backend, or available memory, note that constraint beside the exercise so the lesson remains reproducible.