Inside the dispatch engine: how MarkItDown decides what an input is
MarkItDown's central engineering problem is deciding which interpretation should receive an input before it can produce structure-aware Markdown. This chapter follows that decision from a path, URI, response, or binary stream through metadata enrichment, content detection, converter selection, retries, and errors. The result is a practical model for choosing the right API and for understanding why a successful conversion is a negotiated outcome rather than a simple extension lookup.
From a locator to a stream with identity
The public convert() method accepts several kinds of source. A string may be a local path, a web address, a file: URI, or a data: URI. A Path is treated as a local file, a requests.Response can be handed in after a caller has fetched it, and a binary stream can be converted directly. That variety is convenient because the same object can sit at the center of a script, a command-line workflow, or a larger ingestion system. It also means that the first decision is about access, not format: what resource is the process being asked to read, and who controls that decision?
MarkItDown separates the convenience path from narrower entry points. With convert_local(), the caller has already chosen a local file. With convert_response(), the caller can perform the network request and apply its own fetching policy before handing over the response. With convert_stream(), the caller can open or construct the exact byte stream that should be interpreted. The broad convert() method is useful when those distinctions do not need to be managed separately, but it is deliberately permissive: it can route local resources, network resources, and byte streams through one interface.
For a string source, the scheme determines the first branch. http and https go through URI fetching; file is translated to a local path; data is decoded into an in-memory stream. A string with no supported URI scheme is interpreted as a local path. HTTP responses are read into a seekable buffer before the converter loop begins. A non-seekable stream is likewise copied into memory, because the later selection process may need to inspect it more than once. This design makes the converter contract dependable: every candidate can start from the same position. It does not make input size irrelevant. The source does not state a maximum response or stream size, so the caller still owns limits for large or untrusted inputs.
The access branch also carries identity forward. MarkItDown builds a StreamInfo value that can contain a MIME type, file extension, character set, filename, local path, and source URL. Some of those fields come from the original path or response headers; others are supplied by the caller. A response may provide a content type and charset, a filename through Content-Disposition, or an extension through its URL. A stream may have none of these unless the caller supplies a hint. Rather than treating missing metadata as a terminal problem, MarkItDown uses the available fields as a starting point and then asks the content itself for more evidence.
That metadata is not merely descriptive. It is part of the selection input. A filename ending in .pdf suggests one family of converters, while a MIME type such as text/html suggests another. A caller can provide an extension, MIME type, or charset hint through the command line, which is especially useful when content arrives through standard input and has no filename. The important design choice is that these hints guide selection without becoming an unquestionable declaration of truth.
MarkItDown adds a content-based guess with Magika. If the stream looks like text, it also uses charset-normalizer to infer an encoding. The text sample is bounded rather than requiring a full decode just to form an initial guess, and the implementation takes care not to stop in the middle of a split UTF-8 character at the sample boundary. The point is not that content detection will always win. It is that an input can carry several imperfect clues, and the dispatcher can retain them long enough to compare them.
When the caller's extension or MIME type conflicts with the content prediction, MarkItDown does not simply discard one side. It can keep an enhanced metadata guess and a separate content-based guess, then try both during selection. A mislabeled file therefore has a chance to reach the converter that matches its bytes, while an unusual but valid file is not forced to abandon the caller's explicit context. This is a cautious approach to ambiguity: preserve competing hypotheses, then let converter acceptance and conversion success provide more evidence.
Selection is negotiation, not a guess
Once MarkItDown has one or more StreamInfo guesses, it enters the converter registry. Each converter implements two related operations. accepts() is a quick determination of whether the converter should attempt the stream. convert() performs the actual transformation and returns a DocumentConverterResult, whose required content is the Markdown representation and whose optional title can carry a document-level label.
The distinction between the two operations is important. Acceptance should be cheap enough to run across candidates, but it can sometimes need to inspect the stream. A converter may read enough bytes to recognize a format, then it must restore the original position before returning. The dispatcher records the position before testing candidates and asserts that the position has not changed. If a converter accepts the stream, its conversion attempt begins from that same position. After the attempt, the dispatcher seeks back before trying another candidate.
The registry is ordered by priority. In the built-in arrangement, format-specific converters normally use priority 0.0. Plain text, HTML, and ZIP are more generic and use 10.0, so they are tried later. Lower numbers run first. This prevents a broad text converter from claiming a structured document before the more precise converter has a chance to interpret it. The distinction is not only about speed. It determines which structural story the output will tell. A ZIP-aware converter can preserve member boundaries; a generic fallback might only see an archive as unsupported or as opaque bytes.
Registration order still matters when priorities are equal. MarkItDown inserts new registrations at the front and then performs a stable sort by priority. As a result, a later registration with the same priority is tried before an earlier one. This provides a second control over selection while leaving the numeric priority as the main way to position a converter relative to generic fallbacks. The registry is therefore an active policy surface, not a passive collection of format handlers.
The sequence below shows why the stream position is part of the protocol. Detection and acceptance may look like read-only questions, but they happen against a mutable file-like object. Rewinding is what allows a later candidate to see the original bytes rather than the tail left by an earlier inspection.
The loop is tolerant of imperfect candidates. If a converter accepts a stream but raises an exception during conversion, MarkItDown records the failed attempt and continues looking. This matters for optional dependencies and for ambiguous content. A converter can recognize that an input looks like its format while still discovering that a required library is absent or that the document is malformed. Another suitable converter may still succeed.
The final error distinguishes two situations. UnsupportedFormatException means that no suitable converter attempted a conversion. FileConversionException means that one or more suitable converters did attempt conversion but failed; the exception can retain the list of failed attempts and their exception types and messages. A missing optional dependency belongs to this second path when the corresponding converter is selected. The distinction gives the caller a more useful diagnosis than a single “conversion failed” result: the input may be outside the registry, or the registry may recognize it but lack a working route.
This fallback behavior has a boundary. It is not a guarantee that trying more converters will produce a better interpretation. A generic converter may succeed after a specialized converter fails, but the resulting Markdown may contain less structure. Conversely, a converter can accept a mislabeled input and fail only after deeper parsing. The dispatcher can preserve the chance of recovery; it cannot verify semantic correctness from the fact that some candidate returned text.
The same principle applies to content guesses. Magika can make the selection more resilient when a filename is wrong, while an explicit hint can supply context that content inspection cannot know. Neither signal owns the truth in every case. The successful result depends on the interaction among source metadata, byte-level clues, converter acceptance, and the quality of each converter's structural interpretation.
Uniformity has a price
The dispatch model gives MarkItDown an unusually small conceptual interface. A new converter does not need to know whether its bytes came from a path, a response, a data URI, or standard input. It receives a stream and metadata, returns a Markdown result, and participates in the same acceptance and failure rules. That is what allows the library to span unrelated source families without exposing a different top-level API for each one.
The uniformity is purchased with buffering and repeated inspection. A non-seekable stream is loaded into memory so candidates can rewind. A fetched response is also accumulated before conversion. This makes format detection and fallback predictable, but it means the process may hold the input and intermediate data at the same time. The code does not promise bounded memory use or true streaming conversion. An embedding service should therefore impose its own size, time, and concurrency limits rather than infer them from the simplicity of the API.
Installation state is another part of the effective pipeline. The core package keeps a relatively small required dependency set and exposes format-specific extras. A converter can recognize a PDF, Word document, or spreadsheet while still being unable to convert it because the needed optional package is not installed. MarkItDown delays that failure until the selected converter is used, which keeps the base installation lighter and lets another candidate try. The trade-off is operational: the same input can have different outcomes in two environments with different extras installed.
The command-line interface makes some of these choices visible. It can read a named file or standard input, write Markdown to standard output or an output file, and accept extension, MIME-type, and charset hints. A caller piping bytes without a filename is not forced to guess how the dispatcher should interpret them; the hints become part of StreamInfo. But a hint is still only one participant in the negotiation. If the bytes and hint disagree, the resulting path depends on the competing guesses and the registered converters.
The narrow APIs are consequently more than conveniences for advanced users. They are ways to move ownership of an earlier boundary outward. If the application already has a vetted local path, convert_local() avoids asking MarkItDown to interpret a string as either a path or a URI. If it has already fetched a permitted URL, convert_response() keeps the request policy outside the reader. If it has enforced its own limits while opening a stream, convert_stream() lets the application hand over only the bytes it intends to expose. These methods do not change the structural conversion itself; they reduce what the conversion layer is asked to decide.
The dispatch engine therefore answers the first architectural question: MarkItDown does not know what an input is from one label alone. It assembles clues, preserves ambiguity long enough to test it, gives specific converters priority over catch-alls, and rewinds the stream so competing interpretations remain possible. That is a strong foundation for a common Markdown boundary, but it stops before the harder question of what a winning converter can actually recover. Once selection is complete, the source's own structure—semantic markup, application objects, or page geometry—determines what “preserved” will mean.