Gradient Boosting

Gradient boosting is an ensemble machine learning technique that builds a strong predictive model by sequentially adding weak models—typically decision trees—each one trained to correct the errors of its predecessor. In practice, each new tree fits the negative gradient of the loss function (e.g., mean squared error or log loss) with respect to the current ensemble's predictions. The final prediction is the sum of all trees' outputs, scaled by a learning rate. Operators encounter gradient boosting in classical ML tasks like tabular data classification or regression, often via XGBoost, LightGBM, or CatBoost. While not used directly in local LLM inference, gradient boosting models can run on CPU or GPU and are small enough to fit in system RAM, making them practical for local deployment.

Gradient boosting works by initializing the model with a constant prediction (e.g., the mean of the target). At each iteration, it computes the residuals (negative gradient) of the current model, then fits a new decision tree to those residuals. The tree's output is added to the ensemble, scaled by a learning rate (typically 0.01–0.3). This process repeats for a user-specified number of trees (n_estimators). Regularization techniques like tree depth limits, subsampling, and shrinkage help prevent overfitting. Popular implementations: XGBoost (efficient, handles missing values), LightGBM (faster training with histogram-based splits), and CatBoost (handles categorical features natively). For local operators, gradient boosting models are often trained on CPU and can be exported to ONNX for inference on GPU. They are not related to neural networks or transformers, but are a staple for structured data tasks.

A local operator training a house price predictor on a CSV with 50 features might use XGBoost. With a dataset of 100k rows, training 500 trees of depth 6 takes ~2 minutes on a Ryzen 9 CPU and uses ~500 MB RAM. The resulting model file is ~50 MB, easily loaded in Python with xgboost.Booster(model_file='model.json'). Inference on a single row takes <1 ms on CPU.

In a typical workflow, the operator loads data with pandas, splits into train/test, then runs import xgboost as xgb; model = xgb.XGBRegressor(n_estimators=500, learning_rate=0.05, max_depth=6); model.fit(X_train, y_train). After training, they save the model with model.save_model('house_price.json'). For inference, they load the model and call model.predict(X_test). No GPU is required, though XGBoost supports GPU training with tree_method='gpu_hist' if a CUDA-capable card is available.