From Fragmented Sources to a Working Knowledge Graph
For Semantica, the difficult part is not accepting one more document format; it is preserving useful meaning while very different kinds of data pass through the same AI system. This chapter follows that source-to-graph boundary, from files and records to entities, relationships, events, and graph-ready assertions. The practical result is a way to see where Semantica provides a common contract and where the application must still make decisions about shaping, quality, and review.
A useful source pipeline has two jobs that are easy to confuse. First, it must move information from wherever it lives into a form the system can process. Second, it must decide what that information means in a structure that later retrieval and reasoning can use. A PDF page, a database row, and a message on a stream may all describe the same customer, vulnerability, or policy. They do not arrive in the same shape, and they do not carry the same clues about relationships or time.
One ingestion contract across very different sources
Semantica's ingestion layer presents a family of source-specific connectors behind a common downstream pattern. Files and web pages generally produce objects with text. API and database connectors produce structured records. Feeds produce items with fields such as title, description, and publication time. Stream connectors produce messages. Git and enterprise-platform connectors add their own structured forms. The important unifying step is not that every connector returns an identical object; it is that each result can be transformed into content that AgentContext can store, index, and optionally turn into graph state.
That distinction matters most for structured data. A database row has columns, identifiers, and values, but an embedding model does not automatically know which combination expresses the business fact that matters. A record such as a vulnerability ID, score, affected product, and publication date is more useful to retrieval when it becomes a readable statement: the vulnerability, its severity, the product it affects, and when it was published. The same principle applies to a support ticket, a regulatory rule, or a transaction. The application chooses which fields to project, how to phrase them, and which rows deserve inclusion.
This shaping step is not cosmetic. If an application sends a raw delimiter-separated row into semantic indexing, the resulting vector may reflect the presence of tokens without capturing their relationship. If it turns every column into an enormous sentence, it may add noise and inflate the context later supplied to a model. Good ingestion therefore begins with a focused query or source filter, then produces a compact representation that preserves the facts a reader would use to identify and connect the record.
AgentContext makes a related distinction between memory and documents. A single string is stored as a memory item. A list of strings or document dictionaries is handled as a document collection; when a knowledge graph is attached and extraction is enabled, that collection can also populate graph state. This is a small interface rule with a large architectural consequence. A conversational observation and a source corpus may both be text, but they have different lifecycles and different reasons to become graph nodes. Treating them as interchangeable can create surprising graph content or leave the intended graph empty.
The ingestion guide also treats scale as part of correctness. REST endpoints may paginate, and a one-page call is not the same as a complete collection. Database reads should select only the rows and columns needed for the use case. Stream and scheduled workflows need failure reporting rather than a silent assumption that every source was available. Semantica supplies retry and pagination primitives in relevant connectors, but the application still sets limits, chooses filters, and decides whether partial input is acceptable.
The pipeline builder extends this idea from one call to a directed workflow. A step receives upstream data and configuration, and the execution engine orders dependent steps topologically. Independent branches can run in parallel when they are explicitly marked safe and return compatible data. Retries can address transient failures such as a connection timeout, while an extraction-model failure may deserve a different policy. A delta-mode step can carry a base and target version to its handler, but the handler remains responsible for using those values to select changed input. The framework coordinates transitions; it does not know what “changed” means for every source.
That division of ownership is healthy. A generic platform cannot infer whether a regulatory feed should be complete before publication, whether a missing page should block a threat report, or whether a duplicate event is safe to ignore. It can make those decisions visible as configuration and execution results. The domain workflow must supply the judgment.
The diagram's review branch is deliberate. Extraction can be automatic, but acceptance is not necessarily automatic. A pipeline that moves every candidate directly into a shared graph turns a recognition error into a relationship that later retrieval and inference may treat as established context.
From prose to graph-ready assertions
Once content has a usable representation, Semantica's semantic extraction layer converts language into structured candidates. Named entity recognition identifies meaningful spans and assigns labels such as person, organization, product, threat actor, or vulnerability. The result can retain the text, label, character offsets, confidence, and method metadata. That information is more useful than a bare string because downstream components can distinguish what was recognized, where it appeared, and how strongly the extractor supported it.
Relation extraction adds the missing verbs. A report might mention an actor, a tool, and a target, but the graph needs more than their co-occurrence. It needs a statement such as actor uses tool or tool targets organization. Relation objects retain a subject, predicate, object, confidence, and surrounding context. The surrounding sentence is especially important when an analyst needs to inspect why a connection was proposed rather than merely accept the edge because two names appeared nearby.
Events add another dimension. A static relation says that one entity affects another; an event records that something happened, with participants and, where available, a time or location. Coreference resolution addresses a different problem: “the group,” “they,” and a previously named actor may be the same entity. Without that step, a document can create separate nodes or relations for each wording. Triplet extraction and validation then provide a compact subject-predicate-object representation that can be serialized into RDF or passed into graph construction.
These operations form a chain, but they should not be imagined as a single act of understanding. Each stage can introduce or preserve uncertainty. A relation extractor may identify the correct entities but assign the wrong predicate. A coreference resolver may connect a pronoun to the wrong antecedent. A triplet can be syntactically valid and still be semantically wrong. The useful mental model is that extraction produces assertions with evidence and confidence, not facts that have already earned trust.
Semantica offers several extraction methods because the input and the latency budget matter. Pattern or regular-expression extraction is a strong choice for narrow, predictable forms such as CVE identifiers, IP addresses, hashes, and UUIDs. These patterns do not require a model to infer a context that is already encoded in a format. ML or spaCy-style extraction can provide an offline general-purpose path. Domain-tuned models can recognize vocabulary that a general model may miss. LLM-backed extraction is positioned for implicit entities, custom labels, and relationships that require broader language interpretation.
A method chain can fall back from a primary method to another method when the earlier method returns no results. That is useful resilience when the configured primary method produces no candidates. It changes the source of the candidate, not the truth status of the candidate. But fallback is not verification. A pattern match can be reliable for a well-formed identifier and still fail on corrupted text. An LLM can recover an implicit relationship and still hallucinate one. The source quality and the acceptance policy remain decisive.
Confidence thresholds provide a first filter. They can prevent weak candidates from entering later stages, and they make the trade-off between recall and precision explicit. A high threshold may protect a compliance graph from uncertain assertions while missing useful leads. A low threshold may improve discovery while increasing review load and graph noise. The number is not a universal quality level; it is a control that must be calibrated against the domain, method, and consequence of error.
Input quality is equally important. OCR errors, redactions, broken encoding, or very long unsegmented documents make extraction less reliable. Semantica's splitting and normalization stages exist partly to give extraction a better substrate: text can be cleaned, dates and numbers normalized, and chunks formed without unnecessarily separating an entity or relation. The application still has to decide how to handle a document that is too noisy or too long. A system cannot recover a relationship that the input has destroyed.
Entity linking is where local mentions begin to become shared graph identity. If one document says “North Atlantic Treaty Organization” and another says “NATO,” the graph should not automatically assume that two nodes are the intended result. Linking uses the available text and graph context to connect mentions, but aliases, abbreviations, and ambiguous names require later deduplication or review. The important point is that extraction and linking answer different questions: extraction asks what appears in this content; linking asks whether it corresponds to an entity already known to the system.
When documents are stored through AgentContext with a knowledge graph attached, the context layer can pass extracted entities and relationships into graph construction. The graph builder can then create nodes and edges while the vector store keeps searchable content. This dual destination is what allows later retrieval to combine semantic similarity with structural context. It also creates a coupling that deserves attention: a bad entity label or link affects both the graph topology and the text that a later retriever uses to describe it.
A simple illustration makes the handoff clearer. Suppose a source says that a threat actor deployed a tool against a network during a stated period. In the source text, those details are distributed across nouns, a verb, and a time expression. NER identifies the actor, tool, and network. Relation extraction proposes the deployment and targeting edges. Event detection supplies the campaign window. Coreference resolution keeps later references attached to the same actor. Validation rejects malformed triplets. Linking decides whether the actor matches an existing node. The graph can now represent more than a bag of words: it can represent who did what to whom and when the claim was reported to apply.
That representation is still only as good as its inputs and acceptance rules. Semantica's own extraction guidance warns against treating extraction as guaranteed truth, skipping entity resolution, or using an LLM where a deterministic pattern is sufficient. The graph is a multiplier. It can make a correct relationship easier to find across documents, but it can also make an incorrect relationship easier to propagate.
The source-to-graph boundary therefore has a clear success condition: the system has preserved the content needed for a meaningful entity, relation, event, and source link, and has kept uncertainty visible enough for the next control. It also has a clear failure condition: a candidate is silently promoted to shared context despite ambiguous text, inconsistent labels, weak identity, or an unexamined source. “End-to-end” should mean that the interfaces connect, not that judgment has disappeared.
At this point Semantica has done something important but incomplete. It has turned heterogeneous inputs into a working graph and searchable content, with enough structure to ask relationship questions. The next question is how that structure changes retrieval itself: when a query should follow a connection, respect a historical time window, or combine graph context with a vector result instead of returning the nearest paragraph alone.