How a Document Becomes Searchable Context

This chapter examines RAGFlow’s ingestion boundary: the path from an uploaded file to an indexed context unit that retrieval can actually use. The practical judgment is not “Which parser has the most features?” but “Which representation preserves the relationships this application needs?” Following that path through parsing, chunking, enrichment, tokenization, and indexing shows where context quality is created—and where a later model cannot repair what was lost.

Parsing preserves the relationships retrieval will need

The first mistake in a document pipeline is to treat extraction as a neutral conversion. A text file can often be read as a sequence of characters. A scanned PDF, a spreadsheet, or a slide deck cannot. Their meaning depends partly on position, hierarchy, row and column roles, captions, or the relationship between an image and the words around it. When those relationships disappear, the output may remain readable while becoming unreliable as evidence.

RAGFlow’s DeepDoc materials describe three connected capabilities. Optical character recognition, or OCR, turns text inside images into machine-readable text. Layout recognition identifies roles such as titles, ordinary text, figures, captions, tables, headers, footers, references, and equations. Table-structure recognition reconstructs relationships among columns, rows, headers, projected row headers, and spanning cells. The important causal sequence is straightforward: recognition supplies structure; structure guides grouping; grouping gives retrieval a better unit to select.

The parser then carries that structure into a shared representation. For a complex PDF, the output can retain page and position information, table content rendered into language-like sentences, figure text, and captions. That position data is not decorative. It lets a later preview point back toward the source region and helps keep a retrieved fragment connected to the page from which it came. For a spreadsheet, retaining the table as a self-contained unit prevents a row from being separated from the column headings that give its values meaning. For Word or presentation files, preserving headings, paragraphs, and slides supports a different kind of structural boundary.

A useful example is a service table in a product manual. Suppose one column names a model, another names an operating condition, and a third gives the required inspection interval. A flat text extractor might emit all three columns in a sequence that looks plausible but makes row membership ambiguous. RAGFlow’s parser and table handling are intended to preserve the table’s structure and convert it into content that a language model can interpret. That does not mean the parser has proven the table correct. OCR can misread a character, layout recognition can assign a region the wrong role, and a model-based parser can make an incorrect interpretation. It means the pipeline keeps the possibility of checking and retrieving the relationship alive.

Format-specific dispatch is therefore a design boundary. RAGFlow’s parser has separate paths for PDF, office documents, spreadsheets, markup, images, audio, video, email, and EPUB. A PDF may use DeepDoc, a plain-text route, or another configured parser such as MinerU, Docling, or a vision-model path. An image may use OCR or visual description. An email may select fields such as subject and body. These alternatives should not be read as a feature checklist. They are responses to different information geometries. The right question is whether the selected path produces the fields and structure that downstream chunking and retrieval will need.

There is also a version boundary inside the repository. The Python parser contains a broad set of format branches, while the current Go parser documents a narrower live ingestion scope for the families claimed by its ingestor and uses a conservative raw-text fallback for unported families. That distinction matters operationally: a capability visible in documentation or in a retained legacy path is not automatically available in the runtime path handling a particular deployment. The owner of this boundary is the parser configuration and its execution path; the next component can only operate on the representation it receives.

The parser’s output is not yet a retrieval result. It is a structured stream of records: text and non-text items, layout information, media references, and source-location metadata. The chunker owns the next transition. Its job is to decide which records can travel together as one retrievable unit without exceeding the model’s context or losing the meaning that makes the unit useful.

Chunking and enrichment decide what the index can recognize

Chunking is often described as “splitting long text,” but RAGFlow’s implementation makes a stronger distinction. A chunk is a knowledge fragment that later participates in retrieval and Q&A. Its boundary determines both what can match a query and what can be shown to the answering model. A chunk that is too small may retrieve a precise phrase while omitting the condition that qualifies it. A chunk that is too large may preserve the condition but bury the relevant phrase among unrelated material and consume context budget.

RAGFlow exposes two broad ways to manage that tension. Token-oriented chunking uses delimiters, a configured token target, and optional overlap. It tries to split at natural paragraph or sentence boundaries before merging material into units that fit the budget. Title-oriented chunking uses document hierarchy: PDF outlines when available, configured heading patterns, layout hints, and fallback recognition for numbered or bulleted structures. Built-in document methods extend the same idea to source forms such as Q&A pairs, tables, papers, books, manuals, legal documents, and presentations. The selection rule is conditional. A regular narrative document may benefit from general token chunking; a contract or manual may need section boundaries; a table may be best treated as a structured row or table unit rather than a paragraph stream.

The choice becomes clearer when the application’s question is used as a test. Imagine a manual organized as “Installation,” “Maintenance,” and “Troubleshooting,” with several short instructions under each heading. If a user asks for the steps under one section, a title-aware unit can preserve the section’s scope and its conditions. If users ask narrow questions across many short entries, general token chunks may provide more focused candidates. Neither method is inherently better: one preserves a structural boundary, while the other offers a more uniform retrieval granularity. The correct choice follows from how the source expresses meaning and how queries are likely to cut across it.

There is an ownership boundary behind every failure. The parser starts with file bytes and ends with structured records; a parser failure or a lost table relationship cannot be fixed by changing a similarity threshold. The chunker starts with those records and ends with bounded fragments; a poor boundary can omit a qualifying sentence even when parsing succeeded. The enrichment stage creates alternate handles; a hallucinated keyword can make a chunk easier to retrieve for the wrong reason. Tokenization and embedding define the searchable model space; an incompatible model can make a technically successful index semantically incoherent. The indexer makes the result durable and searchable. Naming these owners prevents an operator from asking the wrong stage to repair the problem.

A reusable pipeline makes those ownership decisions repeatable. A dataset can select a saved sequence of parser, transformer, chunker, and indexer components so future files follow the same assumptions. But repeatability is not the same as correctness. A pipeline built for clean text may consistently mishandle scanned pages; a pipeline built for long manuals may add unnecessary model work to short FAQs. The first run should therefore be treated as a representative test: inspect the structured output, check a few chunk boundaries, verify that enrichment remains faithful, and only then apply the same path more broadly.

sequenceDiagram participant U as Uploaded file participant P as Parser participant D as DeepDoc or format parser participant C as Chunker participant X as Extractor or Transformer participant T as Tokenizer and embedding model participant I as Indexer and document engine U->>P: Apply file-family and parser configuration P->>D: Read text, layout, tables, and media D-->>P: Return structured records and source positions P->>C: Pass records to a chunking strategy C-->>X: Emit retrievable chunks with context X-->>T: Add questions, keywords, summaries, or metadata T->>T: Tokenize and resolve the dataset-bound model T->>I: Send searchable fields and vectors I-->>U: Make enriched context available for retrieval

The order in this sequence explains several safeguards. Overlap copies a portion of adjacent text so a fact split at a boundary is less likely to lose its surrounding condition. It also repeats content, so excessive overlap can increase index size and make retrieved context redundant. For a table or image, a configurable context window can pull nearby text into the media fragment. This protects the case where the table contains values but the preceding paragraph defines the measurement or exception. The context is added around the media item rather than pretending the media itself is an ordinary sentence.

Parent and child chunks address a different problem. A fine-grained child can be a good recall unit because its text has a clear semantic focus. A larger parent can be a better answer unit because it restores the section-level context. RAGFlow’s retrieval path can group child hits back toward a parent when that relationship exists. The resulting design separates the unit used to find evidence from the unit used to explain it. That separation is useful, but it creates another lifecycle to maintain: parent identifiers, child availability, and the relationship between them must remain consistent after a document is reparsed or edited.

Hard token ceilings protect the model boundary itself. RAGFlow’s current chunkers enforce a configured ceiling, first trying sentence boundaries and then using a token-based fallback when a single run is still too large. They also preserve or proportionally carry source-position metadata when a text region is split. This is more than defensive programming. If an oversized unit is sent unchanged to embedding or generation, the model or provider may reject it, truncate it silently, or give the unit a vector that represents only its beginning. If it is split without preserving its source coordinates, the application can no longer show the reader where a retrieved claim came from.

After chunk boundaries are chosen, RAGFlow can enrich the fragments with alternative handles. A Transformer or Extractor may create a concise summary, important keywords, questions that the chunk can answer, or custom metadata. These fields address a mismatch between source language and query language. A manual may say “thermal derating,” while a user asks when a device “needs a lower operating limit.” A generated question or keyword can make the passage easier to discover even when the wording differs.

The enrichment is not free and it is not authoritative by default. It requires additional model calls, can increase latency and token cost, and can introduce a wrong keyword or a summary that omits a qualification. RAGFlow’s current extraction path reflects that risk with bounded concurrency, retries, timeouts, context fitting, and per-chunk caches. Its metadata prompt explicitly tells the model to extract only values stated in the content, reject unsupported enum choices, and return an empty object when no evidence exists. Those rules reduce a specific kind of invention, but they cannot establish that every generated field is faithful. The original chunk remains the safer anchor; enrichment is an index aid that should be checked against it.

The final transition is model compatibility. Tokenization prepares lexical fields for full-text search and prepares text for embedding. The embedding model converts a chunk into a vector in a particular mathematical space. RAGFlow resolves that model from the dataset’s configuration rather than allowing an unrelated per-run choice to define the index. When several datasets are searched together, they need compatible embedding models, because a vector produced in one space cannot be meaningfully compared with a vector produced in another merely because both are arrays of numbers.

This constraint explains why changing an embedding model after documents have been parsed is not a cosmetic setting change. Existing vectors represent the old model’s space; new vectors represent the new one. A mixed index can produce a technically valid query and an operationally available search while making the similarity scores difficult to interpret. RAGFlow’s dataset guidance therefore treats the embedding choice as a property of the knowledge base, and the current Go tokenizer keys its embedding cache with the dataset-bound model identity and the actual embedded text. A configuration or model change must not accidentally reuse a stale vector.

RAGFlow packages these transitions into a reusable ingestion graph. The documented core is Parser, Transformer, Chunker, and Indexer, and a saved pipeline can be selected by a dataset so future uploads follow the same configured path. The current Go runtime compiles a canvas DSL, reports component progress, writes chunks in bounded batches, and can use checkpoint and per-chunk cache behavior during a resumable run. This turns preprocessing into an observable dataflow instead of a hidden script, but it also makes the pipeline itself a versioned asset. Its parameters, model choices, source assumptions, and persistence behavior all affect what later retrieval can know.

The completed output is therefore not “a document in a vector database.” It is a set of structured fragments with source metadata, lexical representations, optional semantic enrichments, and vectors tied to a dataset model. That set is the input to retrieval policy. The next question is how RAGFlow combines its signals—exact terms, semantic similarity, metadata, and ranking—when deciding which of these carefully shaped fragments deserve the model’s attention.

References