Petals

What it is and how it works

Petals is a decentralized inference system for large language models, built out of the BigScience workshop (the same research collective behind BLOOM) as a way to run models far larger than any single participant's hardware can hold. The core idea is borrowed directly from BitTorrent: instead of one machine loading an entire model, the model's transformer blocks are sharded across many machines — a swarm — and each machine hosts only a subset of layers. A client that wants to run inference keeps the embedding/input layer and the output head locally, then streams intermediate activations out to whichever swarm nodes hold the next block of transformer layers, gets back the result, and continues the forward pass block by block until it reaches the output layer again.

This is architecturally very different from the two dominant local-inference patterns most people are used to: full in-memory loading (llama.cpp, Ollama, vLLM) and CPU/GPU offloading (also llama.cpp with --n-gpu-layers, or DeepSpeed-style paging). Offloading keeps everything on one machine and pays a swap-in/swap-out penalty every time a layer that isn't resident in VRAM needs to run. Petals instead distributes the state itself across a network, so no single machine ever needs to hold more than a slice of the model. Each peer runs the standard Petals server process, announces which blocks it's serving to a DHT (distributed hash table, again a BitTorrent-derived mechanism) so clients can discover routes through the swarm, and the client library stitches together a full forward pass by pipelining requests to a sequence of peers.

The tradeoff for this is obvious and the project is honest about it in its own materials: every activation tensor for every token leaves your machine and transits the network to strangers' hardware. There is no meaningful way to make that private without running your own closed swarm end to end (see below). The other real cost is that inference now depends on network latency and the health of round-trip hops between peers rather than pure compute throughput, which is why per-token throughput on the public swarm is modest and variable — usable for chat-style interaction on models otherwise completely out of reach on consumer hardware, not usable for latency-sensitive or high-throughput production serving.

Deployment patterns

The default and most common deployment is simply connecting as a client to the public swarm at petals.dev — no install of a server component required, just the Python client library, and you get inference on models like Llama-2 70B or Falcon 180B that would otherwise require multiple A100/H100-class GPUs to hold in VRAM. This is the "I have a laptop and want to talk to a 70B model" use case, and it's the scenario Petals was actually designed to unlock. Throughput here is whatever the volunteer swarm happens to be able to provide at that moment — it rises and falls with how many hosts are online and serving which blocks, so it behaves more like a shared public resource than a dedicated inference endpoint.

The second pattern, and the one that actually matters for anyone who cares about data handling, is standing up a private swarm. This means running the Petals server component on your own set of machines (a homelab cluster, a handful of team workstations, or on-prem servers) and pointing the client at your private DHT/rendezvous point instead of the public one. This is genuinely the only responsible way to use Petals for anything beyond public/toy data, since it keeps activations inside infrastructure you control — but it also means you now need enough aggregate hardware across your own nodes to shard the whole model, which is a smaller version of the same GPU-cluster problem Petals exists to route around. It's most sensible for a homelab or small team that has several under-utilized GPU machines (a few 3090s or 4090s scattered across desks, say) and wants to pool them into something that can serve a 70B+ model none of the individual machines could load alone.

There isn't really a "solo laptop, no network" deployment mode for Petals in any useful sense — a laptop with no swarm to talk to has nothing to shard against, which puts Petals in a fundamentally different bucket from single-node runners.

How it compares

Against llama.cpp (and its GGUF ecosystem) with CPU/GPU offloading, Petals wins decisively on raw ceiling: you can reach 70B-180B-class models with no individual high-end GPU at all, whereas offloading on a single consumer machine hits severe throughput cliffs once you're paging significant portions of a large model through system RAM or disk. But llama.cpp keeps everything local — no data leaves your machine — and has far broader model/architecture support, whereas Petals' block-sharding approach only works for the specific architectures its server implementation supports (Llama family, Mixtral, Falcon, BLOOM), so a novel or niche architecture likely isn't served at all.

Against vLLM or TGI (Text Generation Inference) run on a dedicated multi-GPU box, Petals is solving a different problem: those are built for high-throughput, low-latency serving when you already own or rent the GPUs to hold the model. Petals exists precisely for the case where you don't have that hardware and are willing to trade throughput and privacy for reach. If you can afford a cloud GPU instance or already have a multi-GPU server, vLLM/TGI will outperform a Petals swarm by a wide margin on both latency and reliability.

Against pure API usage (hosted inference from a model provider), Petals is the open, self-hostable, no-recurring-cost alternative, but it inherits none of a hosted API's guarantees around data handling, uptime, or consistent latency — the public swarm is best-effort infrastructure run by volunteers.

Best use cases and honest limitations

Petals is the right tool when the constraint is "I cannot fit this model anywhere and cannot afford the GPUs to do so," and the workload is not sensitive — research experimentation, hobbyist exploration of frontier-scale open models, education, or prototyping against a model architecture you couldn't otherwise touch. The 3-25x latency improvement over offloading at comparable hardware tiers (per the project's own comparisons) is a real and meaningful win for that narrow case, and setting up a private swarm across a small cluster of GPUs you already own is a legitimate way to pool resources without buying a single monster machine.

It is the wrong tool, unambiguously, for anything involving private, regulated, proprietary, or otherwise sensitive data on the public swarm, since activations transit through volunteer hosts you don't control or trust. It's also not the right choice if you need predictable throughput or low tail latency — public swarm performance depends entirely on who else is online — or if your target model architecture isn't among the small set Petals' server actually supports. Anyone needing dependable production-grade serving should look at vLLM, TGI, or a hosted API instead; anyone who just needs a 7B-13B model to run well on their own box is better served by llama.cpp or Ollama, which are simpler, fully local, and don't require trusting a swarm at all.