RUNLOCALAIv38
->Will it run?Best GPUCompareTroubleshootStartLearnPulseModelsHardwareToolsBench
Run check
RUNLOCALAI

Independently operated catalog for local-AI hardware and software. Hand-written verdicts. Source-cited claims. Reproducible commands when we have them.

OP·Fredoline Eruo
DIR
  • Models
  • Hardware
  • Tools
  • Benchmarks
TOOLS
  • Will it run?
  • Compare hardware
  • Cost vs cloud
  • Choose my GPU
  • Prompting kits
  • Quick answers
REF
  • All buyer guides
  • Learn local AI
  • Methodology
  • Glossary
  • Errors KB
  • Trust
EDITOR
  • About
  • Author
  • How we make money
  • Editorial policy
  • Contact
LEGAL
  • Privacy
  • Terms
  • Sitemap
MAIL · MONTHLY DIGEST
Get monthly local AI changes
Monthly recap. No spam.
DISCLOSURE

Some links on this site are affiliate links (Amazon Associates and other first-class retailers). When you buy through them, we earn a small commission at no extra cost to you. Affiliate links do not influence our verdicts — there are cards we rate highly that we don't have affiliate relationships with, and cards that sell well that we refuse to recommend. Read more →

© 2026 runlocalai.coIndependently operated
RUNLOCALAI · v38
  1. >
  2. Home
  3. /Tools
  4. /TabbyAPI
server
Open source
free (OSS, AGPL-3.0)

TabbyAPI

OpenAI-API frontend for ExLlamaV2. Wraps the EXL2 inference engine in a clean HTTP API, adds streaming, batching, and OAI-compatible chat templates. The default front-of-house when you've already committed to the EXL2 quant format and want to expose it to clients that speak OpenAI.

By Fredoline Eruo·Last verified Jun 12, 2026·1,500 GitHub stars

Overview

OpenAI-API frontend for ExLlamaV2. Wraps the EXL2 inference engine in a clean HTTP API, adds streaming, batching, and OAI-compatible chat templates. The default front-of-house when you've already committed to the EXL2 quant format and want to expose it to clients that speak OpenAI.

Stack & relationships

How TabbyAPI relates to other entries in the catalog — recommended pairings, alternatives, dependencies, and edges to avoid. Each edge carries a one-line operator note from our editorial team.

TabbyAPI ↔ ecosystem

Recommended stack

  • Pairs with
    ExLlamaV2

    The canonical pairing for production-ish ExLlamaV2 serving. ExLlamaV2 is the engine; TabbyAPI is the front of house.

Alternatives

  • Alternative to
    Ollama

    Both expose OpenAI-compatible APIs locally. TabbyAPI wins on raw single-card EXL2 speed for advanced users; Ollama wins on ergonomics and breadth of quant formats. Pick by quant commitment.

Depends on

  • Depends on
    ExLlamaV2

    TabbyAPI is purely a frontend — it wraps ExLlamaV2 in an OpenAI-compatible HTTP API. No TabbyAPI without ExLlamaV2 installed underneath.

Pros

  • Cleanest production wrapper for ExLlamaV2
  • Streaming + batching + tool-call support
  • Minimal operational footprint (single process)

Cons

  • NVIDIA only (it's bound to ExLlamaV2)
  • EXL2 quant only (no GGUF, no AWQ)
  • Smaller community than vLLM / llama.cpp server mode

Compatibility

Operating systems
Linux
Windows
macOS
GPU backends
NVIDIA CUDA
LicenseOpen source · free (OSS, AGPL-3.0)

Runtime health

Operator-grade signals on how actively TabbyAPI is being maintained, how fresh its measurements are, and what failure classes operators have flagged. Every label below is anchored to a real date or count — we never infer maintainer activity we can't show.

Release cadence

Derived from the most recent editorial signal on this row.

Active
Updated Jun 12, 2026

8 days since last refresh · source: lastUpdated

Benchmark freshness

How recent the editorial measurements on this runtime are.

0editorial benchmarks

No editorial benchmarks for this runtime yet.

Community reproduction

Submissions that match an editorial measurement on similar hardware.

0reproduced reports

No community reproductions on file yet.

Get TabbyAPI

Official site
https://github.com/theroyallab/tabbyAPI
GitHub
https://github.com/theroyallab/tabbyAPI

Frequently asked

Is TabbyAPI free?

Yes — TabbyAPI is free to use and open-source.

What operating systems does TabbyAPI support?

TabbyAPI supports Linux, Windows, macOS.

Which GPUs work with TabbyAPI?

TabbyAPI supports NVIDIA CUDA. CPU-only operation is also possible but typically slower.
See something off?Report outdated·Suggest a correctionWe read every submission. Editorial review takes 1-7 days.

Reviewed by RunLocalAI Editorial. See our editorial policy for how we evaluate tools.

Related — keep moving

Compare hardware
  • RTX 4090 vs RTX 5090 →
  • Dual 3090 vs RTX 5090 (tensor-parallel) →
  • RTX 5090 vs H100 →
Buyer guides
  • Best GPU for local AI →
  • Best AI PC build under $2,000 →
When it doesn't work
  • vLLM CUDA version mismatch →
  • Tensor parallelism crash →
  • CUDA driver too old →
  • CUDA out of memory →
Recommended hardware
  • RTX 4090 (24 GB) →
  • RTX 5090 (32 GB) →
  • H100 PCIe (datacenter) →
Alternatives
SGLangText Generation Inference (TGI)vLLMQdrantWeaviateGraphiti (Zep)LanceDBRedis (vector search)
Before you buy

Verify TabbyAPI runs on your specific hardware before committing money.

Will it run on my hardware? →Custom hardware comparison →GPU recommender (4 questions) →