07. Low-RAM Optimization

Memory constraints define the feasible model architecture for most African deployment contexts. Devices with 1-2GB total RAM, with the operating system consuming 500-800MB, leave limited headroom for model loading and inference. Optimization strategies span model architecture, inference implementation, and runtime memory management.

Model architecture choices significantly impact memory requirements. Transformer models scale with attention complexity—reducing sequence length from 512 to 128 tokens provides substantial savings. Smaller embedding dimensions reduce parameter count. Non-transformer architectures like LSTMs or state-space models may achieve comparable task performance with lower memory footprint for certain sequence tasks.

Quantization provides systematic memory reduction. INT8 quantization typically achieves 4x reduction with minimal accuracy loss for most tasks. INT4 quantization enables 8x reduction but requires careful calibration on representative data. The quantization process itself demands memory, so calibration must run on a machine with sufficient resources, with the resulting quantized model distributed to constrained devices.

# Memory-optimized inference with dynamic batching
import gc
import numpy as np
from typing import Generator

class LowRAMInference:
    """Optimized inference for constrained memory environments."""
    
    def __init__(self, model, max_memory_mb: int = 512):
        self.model = model
        self.max_memory_bytes = max_memory_mb * 1024 * 1024
        
        # Pre-allocate buffers based on available memory
        self._configure_memory()
    
    def _configure_memory(self):
        """Calculate safe batch sizes and buffer limits."""
        # Get available memory estimate
        import sys
        # This is a simplified estimate - production code needs 
        # platform-specific memory queries
        estimated_free = self.max_memory_bytes * 0.4  # Reserve 60%
        
        # Calculate safe batch size
        sample_input = self.model.sample_input()
        element_size = sample_input.nbytes
        self.safe_batch_size = max(1, int(estimated_free / (element_size * 3)))
    
    def process_stream(self, input_generator: Generator) -> Generator:
        """Process inputs in memory-safe batches."""
        batch = []
        
        for item in input_generator:
            batch.append(item)
            
            if len(batch) >= self.safe_batch_size:
                yield from self._process_batch(batch)
                batch = []
                gc.collect()  # Force garbage collection
        
        # Process remaining items
        if batch:
            yield from self._process_batch(batch)
    
    def _process_batch(self, batch: list) -> list:
        """Process a single batch with memory management."""
        # Convert to batched tensor
        inputs = np.stack(batch)
        
        # Run inference
        outputs = self.model.run(inputs)
        
        # Clear input buffer before freeing outputs
        del inputs
        
        # Convert to list and free tensor
        results = outputs.tolist()
        del outputs
        
        return results

# Memory monitoring helper
def get_memory_usage_mb() -> float:
    """Cross-platform memory usage query."""
    import sys
    try:
        # Unix systems
        import resource
        usage = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
        if sys.platform != 'darwin':
            return usage / 1024  # Linux reports in KB
        else:
            return usage / (1024 * 1024)  # macOS reports in bytes
    except (ImportError, AttributeError):
        # Windows or fallback
        try:
            import psutil
            process = psutil.Process()
            return process.memory_info().rss / (1024 * 1024)
        except ImportError:
            return 0.0

Memory management during inference matters as much as model size. Paging in model weights creates allocation pressure that can trigger OOM conditions. Careful tensor lifecycle management—deleting intermediate results immediately after use—keeps peak memory low. Streaming large inputs rather than loading complete documents prevents memory spikes.

Hardware-specific optimization takes advantage of processor features. NEON SIMD instructions on ARM processors accelerate matrix operations. GPU memory management differs fundamentally from CPU inference. Specialized accelerators like NPUs in MediaTek and Qualcomm chipsets provide additional options when available, though driver and runtime support varies.

Benchmarking must occur on target hardware rather than development machines. Emulators and cross-compilation cannot capture memory management quirks, cache behavior, or thermal throttling. Budget for device acquisition—testing across five to ten representative devices from different manufacturers reveals issues that simulation misses.