From Upload to Cited Answer: The Evidence Lifecycle

RAGFlow’s value is determined across a lifecycle, not at the moment an LLM prints a response. This chapter follows a document from upload through parsing, indexing, retrieval testing, revision, and cited use, so you can locate a failure instead of blaming “the AI” as one undifferentiated system. The practical outcome is an acceptance loop that separates a missing fragment, a bad ranking, an unsupported answer, and an infrastructure or state-management problem.

A document moves through owned states

The lifecycle begins when a file enters a dataset, but the file is not yet knowledge that the system can safely retrieve. The first transition is parsing. RAGFlow’s parser reads the file through a format-appropriate path and produces structured records: text, layout or hierarchy information, media items, and, where available, source positions. The owner of this transition is the parser. Its success condition is not merely “some text came out”; the output must preserve enough of the source’s meaning for the next stage to form useful chunks.

Chunking turns those records into knowledge fragments. Token or title-based rules decide what travels together, while tables and images may retain surrounding context. The chunker owns the boundary between a continuous source representation and a collection of retrievable units. If it cuts a condition away from the value it qualifies, retrieval can return a true phrase with a false implication. If it leaves a very large unit intact, an embedding or answer model may be forced to truncate it or spend its context on irrelevant material.

Indexing is the next durable transition. RAGFlow can store lexical fields, semantic vectors, source metadata, availability state, and optional enrichment such as generated questions or keywords. The document engine owns persistence and query access, while the dataset configuration owns important compatibility choices such as the embedding model. At this point the system has a searchable representation, not a validated answer. A successful index write only establishes that the records were stored in a form the engine can query.

stateDiagram-v2 [*] --> Uploaded Uploaded --> Parsing: Start document processing Parsing --> Indexed: Store chunks and search fields Parsing --> Failed: Parser or model error Failed --> Parsing: Retry or resume Indexed --> RetrievalTested: Run representative queries RetrievalTested --> Available: Scope, recall, and ranking acceptable RetrievalTested --> Revised: Missing or unsuitable evidence Revised --> Parsing: Edit, reconfigure, or reparse Indexed --> Disabled: Disable stale or incorrect chunks Disabled --> Indexed: Re-enable or rebuild Available --> Used: Chat, Search, or Agent retrieves context Used --> Cited: Answer links to selected evidence Used --> Revised: Feedback exposes a context defect Cited --> Revised: Source or ranking review

The state transitions show why “indexed” and “available for trusted use” are different states. Retrieval testing is the gate between them. RAGFlow’s documentation recommends using typical questions to check whether the expected chunks are recalled, whether they come from the correct document, whether their content is complete enough to answer the question, and whether highly relevant material appears near the top. A dataset that has finished parsing but has never passed those checks is operationally ready to search, but not yet demonstrated to be fit for its intended questions.

The failure path is part of the lifecycle rather than an exception to it. A parser or model failure can stop the transition from Uploaded to Indexed. A resumable Go ingestion run can persist a checkpoint and use per-chunk caches so later extraction and embedding work need not be paid for again after an interruption.