CRDTs for agent state: conflict-free shared memory without a central server
A conflict-free replicated data type (CRDT) is a data structure whose concurrent updates always converge to the same value regardless of the order they arrive — so every node can accept writes locally (multi-master) and sync later without locks, a coordinator, or a central database. Convergence is guaranteed mathematically: the merge function is commutative, associative, and idempotent, which makes the replicated state a join-semilattice. HiveMind models its shared corpus as append-mostly CRDT state so any agent on any device can write while offline, and all replicas reach one identical reality on the next sync.
The problem: many writers, no coordinator
When several agents — possibly on different machines, possibly offline — all read and write one shared memory, you need a way for their writes to combine that never produces two divergent copies. The classic answer is a central database that serializes every write behind a lock or a transaction. That reintroduces exactly the single point of failure and the single point of latency you were trying to avoid by going multi-device in the first place.
Conflict-free replicated data types (CRDTs) solve this without a coordinator. Each replica accepts writes locally and immediately — this is multi-master replication — and replicas reconcile asynchronously. The guarantee is strong eventual convergence: any two replicas that have received the same set of updates are in the same state, no matter what order those updates arrived in or how many times they were duplicated.
Why convergence is guaranteed, not hoped for
The property that makes this work is algebraic. A state-based CRDT defines a merge function ⊔ over its state space, and that function must be:
- commutative —
a ⊔ b = b ⊔ a(order doesn’t matter), - associative —
(a ⊔ b) ⊔ c = a ⊔ (b ⊔ c)(grouping doesn’t matter), - idempotent —
a ⊔ a = a(re-delivery doesn’t matter).
A set with those three properties is a join-semilattice, and merging always moves “upward” toward a least upper bound. Because the network can reorder, drop-and-retry, and duplicate messages freely, those exact three properties are what let an unreliable transport still produce a deterministic result. You don’t need exactly-once delivery; you need eventually-everything delivery, which is far cheaper to build.
The families you actually use
- G-Counter / PN-Counter — grow-only and positive-negative counters; merge takes the per-node max.
- G-Set — a grow-only set; merge is union. Add-only, so it can never lose a member.
- 2P-Set / OR-Set — sets that support removal. The observed-remove set (OR-Set) tags each add with a unique token so a concurrent add wins over a remove instead of being silently lost.
- LWW-Register / MV-Register — single-value registers; last-write-wins picks by timestamp, multi-value keeps concurrent writes for the application to resolve.
- RGA / sequence CRDTs — ordered lists for collaborative text and logs.
For an institutional memory, the safe default is add-wins / grow-mostly structures: every observation an agent records is preserved, and “removal” is modeled as another fact (a correction or a tombstone) rather than a destructive delete. That’s why HiveMind’s corpus is append-mostly — see the append-only log deep-dive and the tombstones and compaction deep-dive for how removal and reclamation are handled without violating the lattice.
Ordering: logical clocks, not wall clocks
To merge concurrent writes correctly you need to know which writes happened-before which, and wall-clock timestamps lie — clocks skew, and two events can be genuinely concurrent. CRDTs lean on logical clocks: Lamport timestamps for a total-ish order, or vector clocks when you need to detect true concurrency (the case where neither write happened-before the other). Detecting concurrency is what lets an OR-Set or MV-Register do the right thing instead of clobbering. The companion deep-dive on causal consistency and vector clocks goes deeper on this.
How HiveMind applies it
HiveMind keeps a full copy of the shared corpus on each of your machines and syncs them peer-to-peer, so there is no central server to bottleneck or lose. Modeling that corpus as append-mostly CRDT state is what makes the “single source of truth” claim hold without a single point of failure: an agent can write while a laptop is offline on a plane, another agent can write on a desktop at home, and when the two devices next reach each other the merge is deterministic — both observations survive, in the same order, on both machines. Logically there is one reality; physically it lives in many places. Your data never leaves your devices to get there.
That convergence is the floor, not the ceiling. CRDTs guarantee every replica agrees on what was written; deciding which of two conflicting claims to trust is a separate layer built on corroboration and confidence.
Frequently asked
What's the difference between a state-based and an operation-based CRDT?
A state-based CRDT (CvRDT) ships its whole state and merges via a join function that is commutative, associative, and idempotent; it tolerates duplicate and out-of-order delivery but the payload can be large. An operation-based CRDT (CmRDT) ships only the operations and applies them, which is compact but requires the transport to deliver each op exactly once (or to make ops idempotent). Delta-state CRDTs are the practical middle ground: ship small deltas with CvRDT-style merge semantics.
Do CRDTs guarantee that the final state is the 'correct' one?
They guarantee convergence, not human correctness. Every replica that has seen the same set of updates computes the same value — there is no split-brain. But if two agents make genuinely conflicting decisions, a CRDT records both deterministically (e.g. a multi-value register keeps concurrent writes); it doesn't pick a winner for you. Resolving the meaning of a conflict is an application concern, which is why corroboration and confidence sit on top.
Why not just use a database with last-write-wins?
Last-write-wins (LWW) is itself a CRDT, but it silently discards a concurrent update when two clocks tie or drift, so writes are lost without anyone noticing. For agent memory that is dangerous — a dropped observation looks identical to one that never happened. Grow-only and add-wins structures preserve every write, which is why an append-mostly model fits institutional memory better than LWW.
Related
Take yourself out of the loop.
Let your agents do the lifting while you keep the judgment.
Get the Playbook