Recommender Systems

Recommender systems are machine learning models that predict user preferences for items (movies, products, content) based on historical interactions. Operators encounter them when deploying models like collaborative filtering or neural recommendation architectures (e.g., YouTube's DNN) that rank candidates. These systems typically require embedding tables and scoring layers, which consume significant VRAM—a 10M-user × 100-dim embedding matrix alone uses ~4 GB at float32. Latency matters: real-time inference must stay under 100 ms for user-facing recommendations, often forcing quantization or model pruning.

Recommender systems fall into two main families: collaborative filtering (user-item interactions) and content-based (item features). Modern deep learning approaches (e.g., Neural Collaborative Filtering, Two-Tower models) learn embeddings for users and items, then compute similarity scores. Training requires large interaction datasets; inference involves candidate generation (retrieving a subset of items) followed by ranking (scoring each candidate). For local AI operators, the key challenge is memory: embedding tables scale with user/item count. A 1M-item catalog with 128-dim embeddings at float16 uses ~256 MB, but a 100M-user table uses ~25 GB—often exceeding consumer VRAM. Techniques like quantization (to int8) or hashing (e.g., Bloom filters) reduce memory at the cost of accuracy. Latency constraints also drive model compression: pruning 50% of weights can double inference speed with minimal recall loss.

A movie recommender with 100k users and 50k movies, using 64-dim embeddings at float32, needs ~40 MB for user embeddings and ~20 MB for movie embeddings—easily fitting on any GPU. But a production-scale system with 10M users and 1M items at float32 would need ~2.5 GB for embeddings alone, plus scoring layers. On an RTX 3060 (12 GB VRAM), this fits, but adding context features (e.g., time, device) could push memory over budget, requiring quantization to float16 or pruning.

When deploying a recommender with Hugging Face Transformers, operators load a model like 'bert-base-uncased' for content-based recommendations. The workflow: tokenize user history, run inference to get embeddings, then compute cosine similarity with item embeddings stored in a vector database (e.g., FAISS). In LM Studio, operators can load a small collaborative filtering model (e.g., SVD from Surprise) and serve predictions via HTTP. VRAM usage is visible in the UI; if the model exceeds available memory, the runtime falls back to CPU, increasing latency from ~10 ms to ~500 ms per prediction.