Local-first vector search: embeddings and ANN retrieval on your own devices

The problem: semantic search usually means shipping your data away

The default way to add semantic search to an application is to send your text to a hosted vector database. You embed each chunk, push the vectors to a cloud service, and query it over the network. It works, but it makes two quiet assumptions: that your data is allowed to leave your machine, and that you always have a connection to the service. For private corpora, regulated data, or agents that run on a laptop on a plane, both assumptions break.

Local-first vector search removes them. The embeddings, the index, and the query all stay on the device. Nothing is sent to a third party, and search keeps working with the network unplugged.

Embeddings and vector similarity

An embedding is a fixed-length vector — typically a few hundred to a couple thousand floats — that an encoder model assigns to a piece of text so that semantically similar text lands at nearby points. “Reset my password” and “I forgot my login” map to vectors close together even though they share no words. That is what makes the search semantic rather than lexical.

Closeness is measured two common ways. Cosine similarity scores the angle between two vectors, ignoring magnitude — cos(θ) = (a·b) / (‖a‖‖b‖). Dot product keeps magnitude in play and is cheaper. When vectors are L2-normalized the two rank identically, so many systems normalize on insert and then use a plain dot product at query time. Either way, retrieval reduces to: find the vectors with the highest similarity to the query vector.

Why exact search doesn’t scale, and what ANN does instead

The honest way to find the top-k most similar vectors is to compare the query against all of them — exact nearest neighbor. It is correct and trivial to implement, but it is O(n) per query and touches every vector, which falls over once the corpus is large.

Approximate nearest neighbor (ANN) search trades a sliver of recall for a large drop in latency. Instead of scanning everything, it organizes the vectors so a query only inspects a small, promising subset and returns the top-k that are almost always the true nearest neighbors. The approximation is tunable: you choose where to sit on the recall/latency curve.

HNSW and IVF: two ways to index

Two index families dominate on-device work.

• HNSW (Hierarchical Navigable Small World) builds a layered proximity graph. A query enters at a sparse top layer, greedily hops toward closer neighbors, and descends into denser layers to refine. Lookups are fast and recall is high; the cost is memory — the graph stores neighbor links per vector — and slower inserts. The efSearch knob trades recall for speed at query time.
• IVF (Inverted File) clusters vectors around centroids and, at query time, probes only the few clusters nearest the query (nprobe). It is lighter on memory and fast to build, with recall that depends on how many clusters you probe. Paired with product quantization (IVF-PQ), it compresses vectors aggressively, which is what makes million-scale indexes fit comfortably on a laptop.

The tradeoff space is three-cornered: recall, latency, and memory. HNSW leans toward recall and latency at the expense of memory; IVF (especially IVF-PQ) leans toward small memory at some recall cost. There is no universally right choice — you pick the corner that matches the device and the corpus size.

The local-first architecture

In a local-first design the index is a local artifact. The corpus lives on disk on the device, an on-device ANN index such as HNSW is built over its embeddings, and a query is embedded and resolved entirely in-process. The query never crosses the network; there is no hosted vector service to authenticate to, pay per query for, or depend on for uptime. Offline operation is the default, not a degraded mode.

This composes naturally with a synced corpus. If every device holds a full local copy of the same corpus and reconciles peer-to-peer, then every device can build the same ANN index and answer the same questions — no central vector store mediating between them. The retrieval layer sits on top of the data layer rather than replacing it. See the peer-to-peer sync protocol deep-dive for how copies converge, and the CRDT deep-dive for why concurrent writes to that corpus merge deterministically.

How this fits HiveMind

HiveMind is local-first shared memory: each machine holds a full local copy of an append-mostly corpus that syncs peer-to-peer, and your data stays on your own devices — never sent to a hosted service. That corpus is the single source of truth your agents read and write. An on-device ANN index is the architecture pattern that fits this model: because the searchable copy already lives on every device, semantic retrieval can be built locally over it without standing up a separate cloud vector database.

The contrast with a hosted vector database is exactly the local-first one. As the HiveMind vs Pinecone comparison lays out, a managed service keeps your embeddings on its servers, charges per query, and needs a connection — strong tradeoffs on data residency, latency, and dependency. A local-first ANN index keeps all three on your side of the line: your vectors stay home, the query path is a function call rather than a round trip, and the only thing you depend on is the device in front of you.

Frequently asked

Related