Milvus

What it is and how it works

Milvus is a purpose-built vector database designed from the outset for distributed, large-scale similarity search rather than as a vector index bolted onto an existing datastore. Its defining architectural decision is compute-storage separation: data is persisted to an object store (S3, MinIO, or compatible blob storage) and a log broker (Pulsar or Kafka), while a fleet of stateless compute nodes — query nodes, data nodes, index nodes — handle search, ingestion, and index building independently. This means you can scale query throughput and indexing capacity separately from storage capacity, which is the same decoupling pattern used by systems like Snowflake or modern data lakehouses, just applied to vector search instead of SQL.

Under the hood, Milvus organizes data into collections, which are further split into shards and segments. Segments are the unit of indexing: as data streams in, it accumulates in growing segments that get periodically sealed and compacted, at which point an index (HNSW, IVF variants, DiskANN, or GPU-accelerated indexes like CAGRA) gets built over them. This segment-based model is what lets Milvus handle continuous ingestion at scale without stalling query traffic — new data is searchable via brute-force or a temporary index on the growing segment while the sealed segments serve indexed queries. A coordinator layer (or, in newer versions, a simplified "Milvus Standalone" mode that folds coordination into fewer processes) manages metadata, schema, and segment placement via etcd.

The multi-tenancy story is genuine rather than bolted on: Milvus supports partitioning at the collection and partition-key level, resource groups for isolating compute between tenants, and role-based access control, which matters if you're building a platform that serves vector search to many customers or teams from one cluster rather than standing up a database per tenant.

Deployment patterns

Milvus has two operationally distinct personalities, and picking the right one matters more than with most databases in this category. The first is Milvus Standalone: a single-binary or single-container deployment (with embedded or attached MinIO and etcd) that runs comfortably on one Linux box or in a single Docker Compose file. This is the right entry point for prototyping, for a homelab RAG pipeline, or for a team server holding tens of millions of vectors — it gets you the same query API and index types as the distributed version without needing Kubernetes.

The second personality is Milvus Cluster: the fully decomposed microservice architecture with independently scaled coordinators, query nodes, data nodes, index nodes, and a message queue, deployed via the official Helm chart onto Kubernetes. This is where the "operationally heavy" cost from the cons list becomes real — you're now running and monitoring etcd, Pulsar or Kafka, MinIO or S3, and half a dozen Milvus microservice types, which is a legitimate platform-engineering commitment, not a weekend project. Teams that reach for cluster mode are usually doing so because they've hit hundreds of millions to billions of vectors, need GPU-accelerated index builds (CAGRA/IVF on NVIDIA CUDA) to keep re-indexing latency sane at that scale, or need hard multi-tenant isolation for a production SaaS product. For teams that want the distributed architecture without operating it themselves, Zilliz Cloud (built by the same team behind Milvus) offers a managed version — useful context since RunLocalAI readers evaluating self-hosting will want to know that escape hatch exists.

How it compares

Against Qdrant, the comparison is fairly clean: Qdrant is a single-binary Rust database that's dramatically easier to run solo or on a small team server, with a simpler mental model and lighter resource footprint, but it doesn't natively decompose into independently scalable microservices the way Milvus does — scaling Qdrant means clustering full nodes, not scaling query/index/data planes separately. Milvus is the tool you reach for after you've outgrown that model, typically north of 100M vectors or when GPU-accelerated index builds become a bottleneck.

Against Weaviate, both are full-featured vector databases with schema, hybrid search, and multi-tenancy support, but Weaviate leans harder into being a general-purpose "AI-native database" with built-in vectorization modules and a more opinionated GraphQL-ish query layer, while Milvus stays closer to being a raw high-performance ANN engine with fewer built-in conveniences and a sharper focus on raw scale and index-type flexibility (it supports the widest range of index algorithms of the major open-source options, including GPU indexes).

Against pgvector, the comparison isn't really apples-to-apples: pgvector is what you use when you already run Postgres and want vector search without adding a new system, and it's genuinely simpler for small-to-mid datasets. Milvus is a dedicated system built to scale past what a single Postgres instance (even with pgvector's newer indexing) can comfortably handle, at the cost of introducing an entirely separate piece of infrastructure.

Best use cases and honest limitations

Milvus earns its keep when you have a genuine billion-scale (or fast-approaching-billion-scale) vector workload, need GPU-accelerated index builds to keep re-indexing tractable, or are building a multi-tenant platform where isolation between customers' vector data is a hard requirement rather than a nice-to-have. It's also a reasonable choice in standalone mode for teams that want headroom to grow into the distributed architecture later without a migration.

It's the wrong choice, per its own cons, for anything under roughly 100M vectors — the operational surface area of even Standalone mode (etcd, object storage, and now a full vector database's worth of index and query tuning) is more than a solo developer or small team needs when Qdrant or pgvector would do the job with a fraction of the moving parts. And if you go to Cluster mode, budget for real Kubernetes expertise: this is not a "docker-compose up and forget" system at scale, it's a distributed system with all the monitoring, upgrade coordination, and failure-mode complexity that implies. Teams without existing Kubernetes operational maturity should think hard before adopting cluster-mode Milvus, or should offload that burden to Zilliz Cloud instead of self-hosting it.