Airflow

Airflow is a workflow orchestration tool for scheduling, monitoring, and managing complex data pipelines as directed acyclic graphs (DAGs). Operators running local AI might encounter Airflow when automating multi-step inference pipelines—e.g., fetching data, running a local LLM via a script, and storing results—especially in environments where cron jobs are insufficient for handling dependencies, retries, or logging. Airflow itself does not run models; it triggers external processes (e.g., Python scripts calling llama.cpp or Ollama) on a schedule or event. For local AI, Airflow is overkill for single-model inference but useful when chaining multiple models or preprocessing/postprocessing steps with reliability requirements.

Apache Airflow defines workflows as DAGs: each node is a task (e.g., a PythonOperator that runs a script), and edges define dependencies. The scheduler continuously checks for DAGs that are due to run based on a cron-like schedule or data triggers. Tasks are executed by workers, which can be local processes or distributed across machines. For local AI operators, Airflow is typically used in a development or research setting where reproducibility and logging matter—e.g., running nightly batch inference on a local dataset, then pushing results to a database. Airflow's web UI provides visibility into task status, logs, and retries. However, its overhead (Python environment, metadata database, scheduler process) makes it impractical for real-time or low-latency inference. Tools like Prefect or Dagster are lighter alternatives, but Airflow remains the most widely adopted in enterprise data stacks. Operators should consider Airflow only when they need to orchestrate multiple steps with complex dependencies, not for single-model serving.

An operator runs a nightly batch summarization pipeline on a local RTX 4090. The DAG has three tasks: (1) fetch new articles from a local database, (2) run a Python script that loads Llama 3.1 8B via llama.cpp and generates summaries, (3) insert summaries back into the database. Airflow schedules this DAG at 2 AM daily. If task 2 fails due to VRAM exhaustion (e.g., another process using the GPU), Airflow retries up to 3 times with a 5-minute delay. The operator monitors progress via the Airflow web UI at http://localhost:8080.

To set up Airflow for local AI, the operator installs Airflow via pip, initializes the metadata database with airflow db init, and creates a DAG file in the dags/ folder. The DAG uses a PythonOperator that calls subprocess.run(["llama-cli", "-m", "model.gguf", "-p", prompt]). The operator starts the scheduler with airflow scheduler and the web server with airflow webserver. They then trigger the DAG manually from the UI or wait for the scheduled time. Logs from each task appear in the Airflow UI, showing stdout/stderr from the llama-cli process.