How Indexes Catch Up Without Becoming Truth
HydraDB's graph indexer exists to make traversal faster without joining the canonical write path. This chapter follows one indexer's lifecycle: discovering scopes, finding dirty topology, building an immutable generation, publishing it safely, and cleaning up older material. It also explains when an incremental patch is genuinely safe and when a full rebuild is the correct fallback, so readers can judge what index lag changes and what it does not.
The indexer begins with a constraint inherited from the storage model. The graph's canonical records belong to the cell and its SlateDB lifecycle. An indexer may read those records and write a derived traversal representation, but it must not become a second writer of the graph. If its Pod disappears, the graph must remain readable and writable. If two indexers perform the same work, their race must affect efficiency or freshness, not silently replace a newer generation with an older one.
A background role with a durable boundary
graph-indexer has no public query listener and never opens graph writers. It operates as a read-only materialization worker. Query nodes and clients continue to own reads and canonical mutations; the indexer turns durable adjacency into immutable compressed sparse column, or CSC, generations that readers may later use.
That separation is what makes indexer failure a different class from writer failure. A failed writer can threaten the mutation authority of a cell and therefore needs leases and SlateDB fencing. A failed indexer leaves canonical graph records intact. Queries may have to use an older generation, reconcile a larger WAL tail, or fall back to canonical snapshot adjacency, but the indexer is not allowed to block the graph's basic correctness path.
The indexer discovers scopes through immutable scope markers rather than a writable catalog in the graph data path. It can then open bounded read-only handles for scopes that contain data. Retaining some handles avoids replaying the same durable state every time a scheduler revisits a scope, but the handles remain a cache. Evicting one closes local reader state; it does not delete the scope or its graph.
The scheduling problem is fairness. A registry may contain many tenant graphs, and a process that always begins at the first scope would repeatedly favor the same part of the fleet. HydraDB sorts the discovered scopes, rotates the starting point after the durable cursor, and processes a bounded slice in batches with bounded concurrency. After a completed fair batch, it advances a scope cursor with object-store CAS. If two indexers race, a stale cursor update loses rather than overwrites the progress recorded by the winner.
The cursor is a progress aid, not graph truth. It tells the scheduler where to resume its rotation. It does not say that a scope has been fully indexed, that its generation is current, or that an index is safe for a particular query sequence. Those questions are answered by the cell's durable sequence, dirty markers, generation metadata, and the query-side snapshot rules.
One indexing cycle
Within a selected scope, the indexer refreshes its reader sequence before inspecting work. It discovers dirty edge types—topology categories whose canonical adjacency has changed—and checks the current generation for each one. If the current generation already covers the dirty sequence, the edge type can be skipped as already current.