10. Streaming with Vision

Chapter 10 of 18 · 15 min

KEY INSIGHT

Vision models generate tokens at varying rates depending on image complexity. Streaming requires buffering visual features while interleaving with text token generation to maintain responsive UX. Streaming visual responses differs from text-only streams. Early tokens represent image understanding, but later tokens refine interpretations. Users expect progressive rendering of generated captions or descriptions. ```python import asyncio from anthropic import AsyncVertexAI import json class StreamingVisionHandler: def __init__(self): self.client = AsyncVertexAI() self.feature_buffer = [] async def stream_vision_response( self, image_path: str, prompt: str ): async with self.client.messages.stream( model="gemini-pro-vision", max_tokens=1024, messages=[ { "role": "user", "content": [ {"type": "text", "text": prompt}, { "type": "image", "source": { "type": "file", "file": image_path } } ] } ] ) as stream: # Buffer early tokens for context accumulated = [] async for event in stream: if event.type == "content_block_delta": token = event.delta.text accumulated.append(token) # Stream text progressively yield {"token": token, "partial": "".join(accumulated)} async def get_structured_stream(self, image_path: str): """Extract structured data from streaming response""" output_schema = { "type": "object", "properties": { "description": {"type": "string", "maxLength": 200}, "objects": { "type": "array", "items": {"type": "string"} }, "confidence": {"type": "number", "minimum": 0, "maximum": 1} } } async for chunk in self.stream_vision_response( image_path, "Analyze this image and return structured JSON" ): # Attempt JSON parsing progressively yield chunk ``` **Common Failure Patterns:** - Forgetting image pre-processing causes timeout failures on large images. Always resize before upload. - Streaming buffer exhaustion if accumulation logic ignores token limits. Set maximum buffer size. - Mixing streaming and non-streaming calls in same session creates race conditions.

Local verification checkpoint

Run the smallest example from this chapter in a local workspace and record the package version, runtime, data path, and observed output. If the result depends on model size, vector count, CPU/GPU backend, or available memory, note that constraint beside the exercise so the lesson remains reproducible.

EXERCISE

Implement a streaming image analyzer that progressively reveals detected objects. Log token arrival timestamps to measure latency between visual understanding and detail expansion.