The Pipeline Behind a Searchable Source

Open Notebook's searchable-source pipeline answers a specific operational question: how can a PDF, web page, recording, or pasted passage become material that an AI can find and use without treating every input as plain text? This chapter follows the two boundaries that matter most: first, whether the original content was extracted faithfully; second, whether that extracted content was indexed and made ready for retrieval. The payoff is practical: a reader can tell whether a poor answer began with inaccessible content, a weak representation, an unfinished background job, or the question itself.

A source is not searchable merely because it appears in a notebook. It has to pass through several transformations, and each one can preserve, compress, or lose meaning.

First, make the source readable

Open Notebook accepts several kinds of research material. A user can paste text, upload documents, add a web link, or bring in audio and video that need transcription. The application stores the source as a research item, but it does not ask the search system to work directly with every original file format. It sends the material through content extraction first.

For a text-native document, extraction may be straightforward. A PDF can yield paragraphs; a Word document can yield its text; an HTML page can be reduced to readable content. Other cases are harder. A scanned PDF may contain only images of letters. A document with two columns or complex tables can be read in the wrong order by a lightweight extractor. A JavaScript-heavy web page may contain little useful text in the initial HTML even though a browser displays a full article. Audio and video need speech-to-text before their contents can be searched.

This is why Open Notebook offers more than one processing route. The default URL path can try a chain of available engines: managed services such as Firecrawl or Jina when configured, a local Crawl4AI browser for pages that need JavaScript rendering, and a simple HTTP parser as a lightweight fallback. Document processing similarly distinguishes a balanced automatic route, a layout-aware Docling route, and a fast simple route. Docling can handle layout, tables, reading order, OCR, and image sources, but it is a heavy optional runtime. Local Crawl4AI also adds a browser and its associated installation weight. Both are kept out of the lean default image and enabled only when needed.

The choice is therefore not a ranking from “bad” to “good.” It is a match between the source's difficulty and the cost of processing it. A clean, text-native batch may benefit from a simple extractor because speed matters more than recovering a complicated layout. A scanned contract may justify layout-aware extraction and OCR because missing a clause is more serious than waiting longer. A web page that looks complete in a browser may require a rendering-capable engine; a page with ordinary HTML may not.

The choice also affects what kind of mistake reaches the researcher. A lightweight parser may preserve the words in a table while losing the relationship between a row and its heading. An OCR pass may recover a paragraph but misread a character in a number or name. A browser-based extractor may capture the visible article while still missing content that appears only after an interaction. A speech-to-text engine may produce a useful transcript whose uncertainty is concentrated around names, jargon, or noisy passages. In each case, the downstream model receives the extracted representation, not an invisible copy of the original layout, page, image, or voice.

That is why “the source was imported” is a weak quality check. A stronger check asks whether the parts that matter to the intended question survived extraction. For a contract, inspect clauses, tables, and page order. For a technical article, check headings, examples, and code-adjacent explanation. For an interview, listen to or sample the transcript around names and decisions. For a web page, compare the captured text with the page as rendered. The application can make an extraction available; the researcher still has to judge whether it is fit for the claim they intend to make.

The application also treats optionality as a real operating condition. Enabling Docling or local Crawl4AI can make the first container start slower because the runtime and its downloads have to be installed and cached. If installation fails, the application is designed to continue running while reporting that the capability is unavailable. If a saved setting names a runtime that is no longer installed after a redeployment, the source-processing graph can fall back to an automatic route rather than failing every source without explanation. That protects availability, but it does not guarantee the fallback will preserve the layout or JavaScript content that motivated the original choice.

Extraction failure has several recognizable forms. An unsupported upload can be rejected before it spends time in a background queue. An inaccessible URL can return no usable content. A YouTube video may have no available transcript and may need a configured speech-to-text model. A scanned document can appear blank when OCR is unavailable or disabled. A page can return a title and shell but not the text rendered by JavaScript. These are not interchangeable errors. Repeating the same AI question will not repair an empty extraction.

The current source-processing graph turns an extraction sentinel or empty output into a failed processing job rather than saving the error message as if it were the source. That distinction is important for recovery. A failed source can be retried or reprocessed; a “successful” source whose body is actually an extraction error can silently contaminate every later step.

Extraction is also a privacy boundary. A local database does not imply that a URL was fetched locally. If a managed scraping service is enabled, the page request and the resulting content may pass through that service. If speech-to-text is hosted, a recording may leave the deployment. The correct question is not “Where is Open Notebook running?” but “Which component handled this particular input?”

Then make the text findable

Once readable content exists, Open Notebook prepares representations for the search modes that depend on it. Long text is divided into chunks. Each chunk can receive an embedding—a numerical representation intended to capture semantic similarity—and the resulting vectors are stored alongside the text. Source documents therefore become multiple searchable pieces rather than one enormous object. Notes and source insights can also receive embeddings so that the knowledge accumulated after ingestion remains discoverable.

The sequence is easier to understand when the ownership of each transition is visible.

sequenceDiagram participant U as Researcher participant A as Source service participant X as Content extractor participant D as Source record participant W as Background worker participant E as Embedding model participant I as Search index U->>A: Add URL, file, or text A->>X: Extract readable content X-->>A: Text or extraction error A->>D: Save source and status A->>W: Queue vectorization W->>E: Embed chunks in batches E-->>W: Return vectors W->>I: Store searchable chunks I-->>U: Ready or failed status

Chunking is not just a storage optimization. It determines the size and shape of the pieces that semantic search can return. Open Notebook uses different splitters for HTML, Markdown, and plain text. Header-aware splitting can preserve the structure of a web page or Markdown document, while a recursive splitter provides a fallback for ordinary text and for oversized sections. The default chunk size is 400 tokens with an overlap of 15 percent, though the setting can be changed for models with larger or smaller embedding windows.

The trade-off is intuitive. Very large chunks carry more surrounding context, but a query may then retrieve a block containing much irrelevant material. Very small chunks can be precise, but they may separate a claim from its definition, qualification, or heading. Overlap helps preserve continuity at boundaries, but it also repeats material and increases the amount that must be embedded and searched. The current defaults are engineering choices designed to fit common embedding constraints, not a guarantee that every domain or language gets optimal retrieval from them.

Imagine a policy document whose key sentence says that an exception applies only to small organizations. If the sentence is separated from the heading that defines “small,” a semantic match may find the claim while losing the condition. If the chunk is made very large, the condition may remain present but be buried among unrelated sections, and the model may have to sift through more context. Header-aware splitting is an attempt to keep structure near content; it is not a proof that every important relationship survives. A researcher who needs a precise answer should still search for the exact term or inspect the source around the returned passage.

Embeddings themselves should be understood as a way to compare meanings, not as a second copy of the document. The system turns a chunk and a query into numerical vectors and looks for similarity. Similar wording or related concepts can therefore meet even when the exact phrase is absent. But a vector does not preserve every number, exception, attribution, or ordering relationship in a form that a human can inspect directly. Semantic retrieval is excellent for finding candidates; the original text remains the place to verify what the candidate actually says.

The system also handles structure imperfectly. A heading-based splitter can produce tiny fragments such as punctuation or isolated markers, which do not make useful semantic units. The chunking utility filters those fragments when other meaningful chunks remain. A source with unusual formatting, mixed languages, tables, or a poor extraction can still produce awkward pieces even after that filter. Inspecting the extracted source is more informative than assuming the index faithfully represents the original.

Embedding creates another separation between the source record and the searchable index. The source record contains the extracted full text and metadata. The index contains chunk text and vectors that support retrieval. Source vectorization is explicitly submitted as a background job; saving a source does not by itself prove that its chunks have been embedded. Notes are saved first and then submit their own embedding job. Insights are created and then submitted for embedding as a separate step.

That separation explains a common status puzzle. A source may be visible in the notebook, may have readable full text, and still return no vector-search result because its embedding job has not finished—or because no embedding model is configured. A note may be visible and editable while its semantic representation is still pending. A worker that is not running can leave the jobs queued indefinitely. The material is present, but one of the representations needed by a particular feature is not ready.

The application batches embedding requests to respect provider payload limits and uses retry behavior for some transient failures. For a long single item, it can embed chunks and combine their normalized vectors through mean pooling. That is a useful way to obtain one representation for a large note or insight, but it is still a lossy summary of many local meanings. Source search retains individual chunk records, which allows a result to point back to a more specific portion of the document.

This design also makes the provider boundary more visible. The extraction path may use a managed scraper, while embeddings are a separate background operation; a speech-to-text model may be needed for some sources. Each choice affects what leaves the deployment. The result is useful flexibility, but privacy must be evaluated stage by stage.

Diagnosing the first bad result

When a source produces a poor answer, start at the earliest uncertain transition. Ask whether the original URL or file was accessible. Check whether the extracted text contains the paragraph, table, caption, or transcript that matters. If the text is missing, change the extraction path, enable the necessary optional runtime, use a different source representation, or supply a transcript manually. If the text is present, check whether processing and embedding have completed. If the source is indexed but the result is still weak, then examine chunk boundaries, query wording, search mode, and the model's interpretation.

A few examples make the order clear. A scanned report with a blank extracted body is an OCR problem before it is a retrieval problem. A dynamic article whose title appears but whose body is absent is a rendering or extraction problem. A clean report that can be read in the source view but cannot be found through semantic search may have a pending or failed embedding job. A result that finds the right paragraph but produces an overstated conclusion is now a context, model, or verification problem.

The API and user interface expose both asynchronous and synchronous source-processing paths. The interface defaults to the asynchronous path so that a slow extraction does not hold the user in a waiting request, and the project's architectural decision favors worker-backed long-running work. The API still retains a synchronous option for callers that need it. That difference matters to integrations: a client should inspect the returned status and not assume that submitting a request means every downstream representation is complete.

The durable result of the pipeline is therefore not a single “processed” label. It is a chain of states: the source has been accepted, content has been extracted, the record has been saved, chunks and embeddings have been generated, and a particular search mode can now use them. A reliable research workflow keeps those states separate. It asks not only whether Open Notebook contains the source, but whether it contains the right text, the right representation, and the right readiness for the question at hand.

References