06. Chart and Diagram Understanding

Any content violation in: NONE Chart and diagram understanding requires extracting structured information from visual representations. Multi-modal models can describe charts, identify trends, and answer specific data questions, though accuracy varies by chart complexity.

Charts present unique challenges for vision models. They combine visual layout, text annotations, colors encoding information, and implicit data patterns. Effective chart understanding requires both accurate transcription and reasoning about the data.

def analyze_chart(model, processor, image_path, question=None):
    image = Image.open(image_path).convert("RGB")
    
    # Structured prompt for chart analysis
    analysis_prompt = f"""Analyze this chart carefully. Provide:
1. Chart type and title
2. X and Y axis labels with units
3. Key data points or trends visible
4. Any notable patterns or anomalies"""

    if question:
        analysis_prompt += f"\n\nBased on this chart, {question}"
    
    conversation = [
        {
            "role": "user",
            "content": [
                {"type": "image"},
                {"type": "text", "text": analysis_prompt}
            ]
        }
    ]
    
    prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
    inputs = processor(
        images=image,
        text=prompt,
        return_tensors="pt"
    ).to(model.device)
    
    with torch.no_grad():
        output = model.generate(**inputs, max_new_tokens=300)
    
    return processor.batch_decode(output, skip_special_tokens=True)[0]

Extract quantitative data from charts:

def extract_chart_data(model, processor, image_path):
    """Attempt to extract structured data from chart visualization."""
    image = Image.open(image_path).convert("RGB")
    
    prompt = "Extract the numeric data from this chart as a structured list. For each data point, provide x and y values. If values cannot be determined exactly, estimate based on visual position."
    
    conversation = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": prompt}]}]
    
    prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
    inputs = processor(images=image, text=prompt, return_tensors="pt").to(model.device)
    
    with torch.no_grad():
        output = model.generate(**inputs, max_new_tokens=500)
    
    result = processor.batch_decode(output, skip_special_tokens=True)[0]
    return result

Bar charts vs. line charts present different challenges:

Failure modes when analyzing charts:

• Scale misinterpretation: Models may miss logarithmic scales
• Small text: Axis labels often skipped
• 3D effects: Distorted charts lead to inaccurate readings