Feature Selection

Feature selection is the process of identifying and retaining only the most relevant input variables (features) for a machine learning model while discarding redundant or irrelevant ones. In local AI, this matters because fewer features mean smaller model inputs, which reduces VRAM usage and inference latency. For example, a text classifier trained on 10,000 word tokens can be pruned to 2,000 high-information tokens, shrinking the embedding table and speeding up token processing without significant accuracy loss. Operators encounter feature selection when tuning data preprocessing pipelines or optimizing models for limited hardware.

Feature selection methods fall into three categories: filter (statistical tests like chi-squared or mutual information), wrapper (trial-and-error using model performance, e.g., recursive feature elimination), and embedded (built into model training, like L1 regularization in linear models or attention weights in transformers). For local AI, embedded methods are especially useful because they don't require separate training runs. In practice, operators might use feature importance scores from a random forest or gradient-boosted tree to select the top-k features before feeding data into a neural network. This reduces the input dimension, which directly lowers the number of parameters in the first layer and thus VRAM consumption. However, aggressive feature selection can discard nuanced interactions that deep learning models exploit, so a balance is needed. Tools like scikit-learn's SelectKBest or SelectFromModel are common for this task.

An operator fine-tuning a BERT-based sentiment model on a 6 GB VRAM GPU might start with 50,000 token vocabulary. By applying mutual information feature selection to keep only the 10,000 most informative tokens, the embedding layer shrinks from ~50M to ~10M parameters, freeing ~160 MB VRAM. This allows fitting a larger batch size or longer sequence length.

In a Hugging Face Transformers pipeline, feature selection often happens during tokenization. The operator sets tokenizer.model_max_length and uses tokenizer.encode with max_length and truncation=True to limit input length. For custom features, they might preprocess data with sklearn.feature_selection.SelectKBest before training, then save the selected feature indices to apply during inference.