01. Code Generation Landscape

Chapter 1 of 18 · 15 min

KEY INSIGHT

Local code generation replaces cloud API dependencies with models running on your own hardware, giving you full control over data, latency, and costs. The code generation ecosystem splits into two distinct approach: streaming autocomplete and conversational chat. Autocomplete systems predict the next tokens as you type, requiring predictions within 50-100ms to feel responsive. Chat systems generate responses on demand, tolerating higher latency since users actively wait for results. Autocomplete implementations typically use fill-in-the-middle (FIM) architectures trained to predict code between prefix and suffix. These models process the visible cursor context and suggest what belongs at that position. Starcoder, Codestral, and Deepseek-Coder families support FIM inference. Autocomplete models are generally smaller (7-15B parameters) because speed matters more than breadth. Chat-based code assistants handle more complex tasks: explaining code, generating new functions, refactoring, and debugging. These systems work with full file context, project-wide knowledge, and multi-turn conversations. Models like CodeQwen, WizardCoder, and Deepseek-Coder-instruct excel at conversational coding tasks but require more compute for acceptable generation speeds. Context management differs significantly between approach. Autocomplete typically receives only the current file and cursor positionΓÇötypically 4K-8K tokens of context. Chat systems can incorporate entire repositories, documentation, and conversation historyΓÇödemanding models that support 32K-128K token contexts. This distinction drives hardware requirements: autocomplete needs fast token generation (50+ tokens/second), while chat prioritizes context window size. Latency targets vary by use case. Research shows developers tolerate autocomplete delays up to 150ms before perceiving sluggishness. Chat responses of 10-20 tokens/second feel responsive for most tasks, though complex refactoring may justify waiting longer. Local hardware determines which model classes are viable: | Use Case | Model Size | Latency Target | VRAM | |----------|-----------|----------------|------| | Autocomplete | 7-15B | <100ms/token | 8-16GB | | Chat | 7-33B | 10-30 tok/sec | 16-48GB | Security and privacy considerations drive many teams to local deployment. Code processed through external APIs leaves your infrastructure, potentially violating compliance requirements. Local models keep proprietary code, internal APIs, and architecture patterns within your network boundary. The tradeoff is managing model updates, infrastructure maintenance, and ensuring quality matches cloud alternatives. This course uses Continue as the primary interface because it provides both autocomplete and chat capabilities through a unified plugin architecture. Alternative editors have similar plugins (Cursor, Cline, Roo Code), but Continue's open-source nature and configuration flexibility make it ideal for learning the underlying concepts that transfer across tools.

EXERCISE

Run nvidia-smi and note your VRAM capacity. Review your typical project sizes by running find . -name "*.py" -o -name "*.js" | xargs wc -l | tail -1 to understand your context requirements. This determines which model tiers are realistic for your hardware.