Linear Regression

Linear regression is a statistical method that models the relationship between an input variable (feature) and an output variable (target) by fitting a straight line: y = wx + b. In machine learning, it's the simplest form of a neural network (a single linear layer). Operators encounter it as a baseline for regression tasks or as the final layer in many models (e.g., predicting a single number from embeddings). Training adjusts w and b to minimize prediction error, typically via mean squared error.

When using Hugging Face Transformers for a regression task like predicting house prices from text descriptions, you might replace the default classification head with a linear layer that outputs a single float. The model's final hidden state (e.g., from BERT) is fed into a linear layer with one output neuron. Training uses mean squared error loss.

In a custom training script with PyTorch, you'd define nn.Linear(input_dim, 1) as the regression head. During inference, the model outputs a raw number. If you're using Ollama or llama.cpp for text generation, linear regression isn't directly used, but the final unembedding layer in LLMs is a linear transformation from hidden states to vocabulary logits.