NLTK

NLTK (Natural Language Toolkit) is a Python library for classical NLP tasks like tokenization, stemming, tagging, and parsing. It is not a deep learning framework; it works with rule-based and statistical methods. Operators encounter NLTK when preprocessing text for local LLMs—e.g., splitting text into sentences or words before feeding it to a model. NLTK is CPU-bound and lightweight, so it runs on any hardware without GPU acceleration. It is often used alongside Hugging Face Transformers for tokenization or data cleaning pipelines.

An operator running a local RAG pipeline might use NLTK's sent_tokenize to split a 10-page PDF into sentences before embedding them. This runs in seconds on a laptop CPU, unlike the LLM inference that requires a GPU. NLTK's word_tokenize is also commonly used for counting tokens or building simple keyword filters.

In a Python script, an operator imports nltk and downloads tokenizer data (nltk.download('punkt')). Then they call nltk.sent_tokenize(text) to split text into sentences before passing each sentence to a local LLM via Hugging Face Transformers or Ollama. NLTK runs on the CPU while the LLM uses VRAM.