server
Open source
free (OSS, AGPL-3.0)

TabbyAPI

OpenAI-API frontend for ExLlamaV2. Wraps the EXL2 inference engine in a clean HTTP API, adds streaming, batching, and OAI-compatible chat templates. The default front-of-house when you've already committed to the EXL2 quant format and want to expose it to clients that speak OpenAI.

By Eruo Fredoline·Last verified Oct 7, 2026·1,500 GitHub stars

Overview

OpenAI-API frontend for ExLlamaV2. Wraps the EXL2 inference engine in a clean HTTP API, adds streaming, batching, and OAI-compatible chat templates. The default front-of-house when you've already committed to the EXL2 quant format and want to expose it to clients that speak OpenAI.

Stack & relationships

How TabbyAPI relates to other entries in the catalog — recommended pairings, alternatives, dependencies, and edges to avoid. Each edge carries a one-line operator note from our editorial team.

TabbyAPI ↔ ecosystem

Recommended stack

  • Pairs with
    ExLlamaV2

    The canonical pairing for production-ish ExLlamaV2 serving. ExLlamaV2 is the engine; TabbyAPI is the front of house.

Alternatives

  • Alternative to
    Ollama

    Both expose OpenAI-compatible APIs locally. TabbyAPI wins on raw single-card EXL2 speed for advanced users; Ollama wins on ergonomics and breadth of quant formats. Pick by quant commitment.

Depends on

  • Depends on
    ExLlamaV2

    TabbyAPI is purely a frontend — it wraps ExLlamaV2 in an OpenAI-compatible HTTP API. No TabbyAPI without ExLlamaV2 installed underneath.

Pros

  • Cleanest production wrapper for ExLlamaV2
  • Streaming + batching + tool-call support
  • Minimal operational footprint (single process)

Cons

  • NVIDIA only (it's bound to ExLlamaV2)
  • EXL2 quant only (no GGUF, no AWQ)
  • Smaller community than vLLM / llama.cpp server mode

Compatibility

Operating systems
Linux
Windows
macOS
GPU backends
NVIDIA CUDA
LicenseOpen source · free (OSS, AGPL-3.0)

Runtime health

Operator-grade signals on how actively TabbyAPI is being maintained, how fresh its measurements are, and what failure classes operators have flagged. Every label below is anchored to a real date or count — we never infer maintainer activity we can't show.

Release cadence

Derived from the most recent editorial signal on this row.

Active
Updated Oct 7, 2026

6 days since last refresh · source: lastUpdated

Benchmark freshness

How recent the editorial measurements on this runtime are.

0editorial benchmarks

No editorial benchmarks for this runtime yet.

Community reproduction

Submissions that match an editorial measurement on similar hardware.

0reproduced reports

No community reproductions on file yet.

Get TabbyAPI

Official site
https://github.com/theroyallab/tabbyAPI
GitHub
https://github.com/theroyallab/tabbyAPI

Frequently asked

Is TabbyAPI free?

Yes — TabbyAPI is free to use and open-source.

What operating systems does TabbyAPI support?

TabbyAPI supports Linux, Windows, macOS.

Which GPUs work with TabbyAPI?

TabbyAPI supports NVIDIA CUDA. CPU-only operation is also possible but typically slower.

Reviewed by RunLocalAI Editorial. See our editorial policy for how we evaluate tools.

Related — keep moving

Compare hardware
Buyer guides
When it doesn't work
Recommended hardware
Alternatives
Text Generation Inference (TGI)ChromaQdrantWeaviateLanceDBMilvusRedis (vector search)Neo4j GraphRAG
Before you buy

Verify TabbyAPI runs on your specific hardware before committing money.

Will it run on my hardware? →Custom hardware comparison →GPU recommender (4 questions) →