Who May Write? Placement, Leases, and Fencing

HydraDB can let many query nodes read a shared durable graph, but a mutation asks a stricter question: which node may open the writer for one cell right now? This chapter follows HydraDB's answer through three separate layers—liveness and placement, an object-store lease, and SlateDB's writer epoch. The distinction matters because a node can be the expected contender without yet being admitted, and a node can hold an old local handle without retaining the authority to commit.

The problem is not solved by choosing a node once and storing that assignment forever. Nodes restart, heartbeats expire, network views fail, and a process that used to write may still have an open handle after another process has taken over. HydraDB therefore treats ownership as a sequence of decisions with different authorities. The sequence makes normal routing efficient while leaving SlateDB a final way to reject stale writers.

Liveness comes from a shared store

HydraDB's placement view begins with heartbeats. A ready graph node publishes one heartbeat object under a shared object-store prefix and refreshes it periodically. The object name carries the node identity, while the object's LastModified timestamp supplies the freshness signal. Placement obtains the names and timestamps with one LIST operation; it does not fetch every heartbeat body to decide who is alive.

That choice is more than a request optimization. If each caller probed each node separately, two callers could construct different live sets at nearly the same time. A node might appear healthy to one routing calculation and expired to another. HydraDB instead makes the object store the common observation point. Every node sees the same timestamped records and applies the same liveness rules.

The live set is the intersection of two facts: configured membership and fresh heartbeats. A leftover heartbeat from a decommissioned node does not make that node a candidate if it is no longer part of the configured fleet. A configured node that has stopped publishing does not remain live forever merely because it appears in static membership. If a node briefly has more than one heartbeat object, the youngest entry for that node is selected before freshness filtering, so an old leftover cannot speak for a newer live record.

HydraDB also handles the cases where evidence is incomplete. A node counts itself as live, because a stale view of its own heartbeat is not reliable proof that it should abandon a cell. During startup, a configured peer that has never published a heartbeat is temporarily treated as live; absence is not yet evidence of death. By contrast, a heartbeat that exists but has aged past the freshness boundary is positive evidence that the publisher stopped refreshing. An empty successful LIST means that peers have withdrawn and the fleet is known to be empty. A failed LIST means that the fleet is unknown.

Those distinctions prevent a dangerous shortcut: treating every absence of a visible owner as permission to promote. HydraDB gives a failed LIST a bounded grace period, during which the node serves the cached live view and keeps retrying. If the failure lasts beyond that grace, the view becomes Shed. The node becomes unready, withdraws its own heartbeat, and refuses promotion without an owner hint. Withdrawing is essential.