Foundation Model

A foundation model is a large neural network trained on broad data at scale, designed to be adapted for a wide range of downstream tasks rather than a single task. In local AI, operators encounter foundation models as the base checkpoint (e.g., Llama 3.1, Mistral, Qwen) that they then quantize, fine-tune, or run inference on. These models are typically too large to fit in consumer VRAM at full precision, so operators rely on quantized versions (e.g., Q4_K_M) to run them locally. The term distinguishes general-purpose models from smaller, task-specific models.

Foundation models are trained on internet-scale datasets using self-supervised learning (e.g., next-token prediction for LLMs). This pre-training phase builds broad knowledge, which can then be specialized via fine-tuning or in-context learning. For local operators, the key implications are: (1) model size — foundation models range from 1B to over 400B parameters, with consumer hardware typically handling 1B–13B at usable speeds; (2) quantization — running a foundation model locally almost always requires quantizing weights to 4-bit or 8-bit to fit VRAM; (3) licensing — many foundation models (e.g., Llama 3.1, Mistral) have permissive licenses that allow local use and fine-tuning. The term gained prominence with BERT and GPT-3, but now refers to any large pre-trained model that serves as a starting point for further adaptation.

A 7B-parameter foundation model like Mistral 7B at 16-bit precision requires ~14 GB of VRAM. On an RTX 3060 12 GB, that won't fit. Quantizing to 4-bit (Q4_K_M) reduces the size to ~4.5 GB, fitting comfortably and allowing ~40 tok/s inference. A 70B foundation model like Llama 3.1 70B at Q4 requires ~40 GB, exceeding a single RTX 4090 24 GB, so operators must offload layers to system RAM or use multi-GPU setups.

When you run ollama pull llama3.1:8b, Ollama downloads the 8B foundation model weights (quantized to Q4 by default) and stores them in ~/.ollama/models/blobs. The runtime then loads the model into VRAM. If you want to fine-tune a foundation model, you might use Hugging Face Transformers: from transformers import AutoModelForCausalLM; model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1") and then apply LoRA to adapt it to your data.