From a Document to a Graph Without Losing the Edges
In Utopia, the document-to-graph pipeline is not a single conversion step. It is a sequence of boundaries that lets a source become searchable quickly while its more uncertain interpretation is built in the background. This chapter follows that sequence—from source identity and versioning through parsing, embedding, extraction, and review—so the reader can see why Utopia preserves an observation even when it cannot yet give the observation a reliable type or relation.
Readiness and interpretation are different states
A source item begins as material with an identity, not as a graph fact. Utopia’s pull and push interfaces both rely on a stable identifier within the source. If the same identifier arrives with new content, the system updates the same document, keeps the earlier version, re-indexes the content, and sends the new material through extraction again. A deletion signal marks the document as no longer present in its source; it does not immediately erase the document or all the knowledge that once cited it. That choice matters because a later reviewer may need to distinguish “the source withdrew this item” from “we never had evidence for it.”
The identity rule is more important than a filename. A file may be renamed while remaining the same source item, and a feed or business system may use an external identifier that is unrelated to the displayed title. Utopia’s push interface therefore treats an external identifier as the stable key when one is supplied, with the filename serving as a fallback. The pull interface likewise asks the source to return stable IDs and a full current body rather than a diff. Repeating an unchanged item is harmless because content hashing can recognize it; omitting an item is not treated as deletion unless the source explicitly returns a deletion signal. These decisions keep synchronization incremental without allowing an incomplete response to erase history.
The document then takes two related but deliberately asynchronous paths. First, Utopia parses it and divides the text into overlapping chunks—roughly 1,200 characters with 150 characters of overlap—then creates embeddings. Once this work is complete, the document is ready for full-text and semantic search, and it can support question answering. The graph does not have to wait for every extraction call. A long report can therefore be findable within seconds even if turning its contents into entities and relations takes much longer.
Second, extraction is queued as background work, with an LLM call for each chunk. The extractor is not asked to summarize the document into one undifferentiated answer. It identifies entities, attributes, and relations, associates them with the source passage, and tries to place them within the ontology available to that knowledge base. The output is consequently a set of observations that can enter different states. Some become ordinary facts. Some remain incomplete because the vocabulary does not contain the right term. Some are held for review or rejected at a particular boundary.
The separation solves a practical problem without pretending that it solves an epistemic one. Search readiness means “the text and its vector representation can be retrieved.” It does not mean “every graph interpretation is complete or correct.” If Utopia blocked search until extraction finished, the system would make the slowest and least certain stage control the most basic use case. If it treated search readiness as graph completeness, it would hide the difference between a passage that can be read and a claim that can safely be reused.
This two-speed design also clarifies who owns what. The source interface owns the identity of the item and its versions. The search path owns retrieval of text. The extraction path proposes structure. The graph accepts only the observations that pass its write rules, while Review records what needs a person’s attention. A failure in one path should not be silently reinterpreted as success in another.
There is a consequential ordering here. Embedding is a prerequisite for the ready state, but it is not evidence that the model has understood the text. Extraction needs the chunk and its surrounding evidence, but its output is not automatically a new piece of ontology. Graph insertion comes after validation, and review is not a replacement for retaining the source passage. Each transition narrows what the system is willing to claim. That is why the pipeline can be responsive without making responsiveness a proxy for correctness.
Every extraction outcome stays legible
The important question is not whether an LLM can produce a triple. It is what Utopia does when the triple is incomplete, ambiguous, or inconsistent with the available ontology. The pipeline records failure at the narrowest meaningful point instead of treating the whole document as either perfect or useless.
A truncated model response can invalidate the chunk because there is no reliable way to know what the omitted portion would have contained. A malformed item can be dropped without discarding other valid items from the same chunk. A model may return an entity name that is actually a sentence, or a relation whose subject was never declared among the entities in that chunk. A literal may fail datatype conversion. A relation may be missing its object. These are different failures because they call for different remedies, and Utopia records them as different extraction-drop reasons rather than one generic “processing error.”
That granularity also protects downstream interpretation. If a whole chunk were marked simply “failed,” a reviewer would not know whether to retry the model, add an ontology term, correct a document, or inspect an entity. If every malformed item caused the document to fail, one bad response would hide useful observations. Conversely, if every partial response were marked complete, the graph would look more certain than the evidence justifies. The drop report does not solve the underlying problem, but it preserves the information needed to choose a repair.
The treatment of unknown vocabulary is especially revealing. Suppose a sentence says that a product is “available on” a particular platform, but the ontology has no accepted predicate for that wording. The system does not need to invent a relation called related_to. That fallback would look like a normal edge and would assert that the relationship is known to be “related,” even though the only conclusion supported by the pipeline is that an edge was observed but not named within the current vocabulary. Utopia can leave the predicate empty while retaining the subject, object, time, evidence, and the source’s original wording. The wording remains available for display and for the ontology-growth process described later.
The same distinction applies to a literal. A number, date, or phrase that is an attribute value should not become an entity merely because the extractor could not map its predicate. Keeping the value on the observation preserves what the sentence actually supplied. It also prevents a value such as “2015” or “$4 billion” from becoming a shared node that appears to be a thing in the world. What is not yet understood remains attached to the evidence instead of being promoted into a misleading category.
Ontology signatures create a different kind of boundary. A relation may be recognized, but its endpoints may be reversed. If the declared domain and range say that an employee relation runs from an organization to a person, and the extracted subject/object order violates that signature while the reverse order fits, Utopia swaps them and records that the direction was corrected. The system treats argument order as an encoding contract, not as a detail the model may ignore. If neither direction is legal, it removes the predicate but retains the observation and evidence. In both cases, a questionable declaration does not silently destroy the sentence, and a correction is not hidden as if it had come from the source.
This is a subtle but important separation of responsibilities. The text can support the existence of a relationship even when it does not establish the ontology’s preferred orientation. The ontology can reject an endpoint pairing without proving that the underlying sentence is false. The write path therefore repairs the representation when a safe repair is available and falls back to an unnamed observation when it is not. It avoids both extremes: blindly trusting the model’s ordering and treating a schema mismatch as permission to discard the source.
This is why confidence is only one signal. A model can report high confidence for a relation that the ontology rejects, while a low-confidence extraction may still preserve a useful passage for a person to inspect. Utopia’s pipeline exposes the distinction between “the model produced this,” “the write path accepted this structure,” and “a person has reviewed this.” The graph can then keep moving without converting a probability estimate into authority.
The visible drop trail makes the distinction operational. Reasons such as a truncated reply, low confidence, undeclared subject, datatype failure, or domain mismatch are recorded in an extraction-drop report associated with the document. A user can see that facts did not land and why, rather than seeing a green completion state that implies completeness. The report is not an accuracy score: it does not prove that the facts which did land are true, nor does it tell a person that every omitted statement has been found. It is a completeness and diagnosis surface, allowing the next action—re-extraction, ontology refinement, type review, or source inspection—to be chosen with more information.
Versions and document time make that diagnosis durable. A connector that returns the same stable item with changed content does not create an unrelated document and lose the relationship between versions. The updated material can add, correct, or supersede extracted knowledge while the earlier evidence remains attributable to the version that supplied it. When a source supplies a real publication or effective date, that date can inform the world-time position of facts; when it supplies no such date, the system must not pretend that ingestion time is the date asserted by the document. The full temporal interpretation is more involved, but its foundation is already present here: evidence is not only a quote, it is a quote from a particular source state.
This also explains why a source update is not equivalent to a graph rewrite. The new document version may lead extraction to a different predicate, a changed entity, or a corrected interval. Those are later knowledge transitions, not reasons to delete the old observation before anyone can inspect it. Conversely, a source’s absence from an incremental response is not enough to conclude that the item was deleted. Utopia requires an explicit signal because silence in a synchronization response may mean only that the source returned changes rather than a full snapshot. The system preserves a distinction between “nothing was returned” and “the source says this item is gone.”
A document can therefore be useful while still being unfinished. Search can surface its text. Extraction can add several valid facts. Other items can wait because the ontology lacks a term, the entity is uncertain, or a write constraint has exposed a contradiction. These states are not noise around the “real” graph. They are the information that tells a person how much of the document has become structured knowledge and where the system’s understanding stops.
The next problem follows naturally. Preserving an unknown relation is safer than inventing one, but a system still needs a vocabulary if observations are to become comparable across documents. The ontology must decide which terms are available, how large a prompt can be, and when a recurring phrase has earned a place in the model—without mistaking its own categories for reality.