Preserving Signal in JSON, Logs, Search, and Code
Headroom’s content-specific compressors answer a practical question that a token counter cannot: what must remain visible for the model’s next action? This chapter examines the project’s JSON, search, log, diff, prose, and code paths, with SmartCrusher as the clearest example of the choice between compact representation and selective omission. The goal is to let you decide whether a compressed block is suitable for reasoning, for patching, or only when a retrieval path remains available.
A smaller block is not automatically a better block. A useful transformation preserves the structure and signal that the consuming agent needs, and it knows when exact bytes matter more than a large reduction. Headroom’s router therefore treats compression as a family of content contracts rather than one universal operation.
SmartCrusher starts by preserving rows
SmartCrusher handles JSON arrays, especially the arrays that tools return from searches, APIs, databases, and diagnostic services. Its current default is not simply “keep the first few records.” It begins by asking whether a consistently shaped array can be represented more compactly while retaining every row. When a lossless tabular representation clears its savings condition, the array is re-encoded into a compact schema form. The model receives less repeated syntax, but the records themselves are not discarded.
That first step is important because repetition often lives in the representation. If every row repeats the same field names, punctuation, and predictable formatting, the system can reduce those costs without guessing which row matters. This is a different operation from summarization. The result may look unfamiliar compared with the original JSON, but it still carries the complete table’s data contract. It is a structural answer to structural redundancy.
Only when the array is not suitable for that representation, or the lossless form is not worth adopting, does SmartCrusher use its scored keep/drop path. That path selects a bounded subset using several signals rather than a fixed positional slice:
- first items, which can establish schema, ordering, or pagination context;
- last items, which can carry recency or final-state information;
- errors and warnings, which may be rare but operationally decisive;
- statistical anomalies, including unusual numeric values or string lengths;
- query relevance, using the user’s question and, where available, tool-call arguments;
- change points, where the pattern of values shifts sharply.
The result is intended to be a representative view, not a claim that every row is equally important. Identical items can be deduplicated, and nested arrays can be handled within configured shape and depth limits. The selection budget can be exceeded for critical signals: an error or anomaly is not discarded merely because the ordinary subset is full.
This makes SmartCrusher more than a row counter, but it does not make it omniscient. A business-critical row that contains no detectable error, anomaly, boundary, or query match may not be selected. The repository includes an audit-safe option for callers that provide protected patterns; matched rows are then kept verbatim rather than left to statistical sampling. That safeguard illustrates a general rule: statistical importance and business or legal importance are not the same thing unless the caller supplies the missing knowledge.
Relevance changes what “representative” means
A subset selected only from position and variance answers a generic question: which records describe the overall shape of this array? An agent usually has a more specific question. It may be looking for payment failures, a particular service, or a path connected to an authentication bug. SmartCrusher can use query context so selection is not blind to that information need.
The context can come from recent user messages and, in tool-aware paths, the arguments that triggered the result. A search pattern or requested path can therefore be a sharper clue than a generic keyword list. The same array may produce a different visible subset under a different question, even though the input records are identical. That is useful, but it also creates a dependency: a caller that supplies weak or missing context should not expect query-conditioned selection to recover the reader’s intent.
The project’s TOIN, or Tool Output Intelligence Network, extends this feedback over repeated calls. It records local observations about tool signatures, item counts, rough token counts, strategies, query context, and small row samples. Those observations can supply field-level hints to later compression decisions. A new tool begins with statistical heuristics; repeated use can reveal which fields or patterns tend to carry signal for that tool.
TOIN is not a shared intelligence service in the evidence available here. It is a local observation loop, and its confidence grows from the workload it has actually seen. That makes it potentially useful for a stable tool ecosystem, but it also means a learned preference should not be treated as a universal truth about a newly changed tool or a different team’s data.
Different formats need different evidence
JSON is only one form of tool output. Headroom’s other routes encode different ideas of relevance because their inputs support different actions.
Search results often repeat file paths, line prefixes, and surrounding context. The search compressor keeps exact matches, high-relevance lines, diversity across files, and first or last matches. Its purpose is not to invent a summary of the codebase; it is to reduce repeated search framing while retaining a useful spread of evidence.
Build and test logs contain a great deal of success noise. A log compressor can preserve errors, failures, warnings, stack traces, section headers, and final summaries while collapsing repeated “passed” output. The distinction is operational: a successful test line usually confirms routine progress, while a failure line can determine the next debugging action.
Diffs have another constraint. The changed lines are the subject, and surrounding context helps explain them. A diff compressor can remove unchanged bulk, but Headroom deliberately avoids running a generic lossy text pass over diff hunks. A paraphrase that reads well to a model could still change the patch’s meaning or undermine later application of the diff.
HTML can contain a small article surrounded by navigation, scripts, and advertising. Extraction targets the main content rather than treating page chrome as equally valuable prose. Tables and structured configuration benefit from preserving their shape and relationships. Plain text can reach an ML token-selection path, but its savings vary with redundancy and runtime conditions; it is not a fixed percentage operation.
These routes share an idea but not a preservation guarantee. Search output is useful when matches remain identifiable. A log is useful when failure evidence and sequence remain legible. A diff is useful when the changed hunk remains trustworthy. A document is useful when relevant passages and organization survive. The right question is always tied to what the model will do next.
Code has two jobs: reason and edit
Source code exposes the clearest difference between structural usefulness and byte identity. Headroom includes an AST-aware CodeAwareCompressor powered by tree-sitter. Instead of cutting a file at an arbitrary character boundary, it can preserve imports, function and method signatures, class definitions, type annotations, decorators, and syntax while compressing parts of function bodies and verbose comments or docstrings. The intended result gives a model the shape and interfaces of an implementation without requiring every implementation line to remain in view.
That can be valuable when the model is surveying a large codebase or reasoning about relationships among functions. It is not the same as preserving a file for an edit. An agent that must copy an exact old string into a patch needs the original spelling, indentation, and line breaks. Even a data-preserving fold that changes path headings or whitespace can make that edit fail. The current coding posture therefore protects file reads from lossy compression, while allowing other tool output—such as logs or searches—to remain compressible.
Protection is also conditioned on intent and recency. Recent code and code in a conversation where the user is asking to analyze, review, explain, fix, or debug can be kept intact. The proxy can enable AST-aware routing, while the default library/router configuration does not necessarily send detected code to that compressor. This is a subtle but important deployment boundary: the project’s capability, the path’s default, and the action the model must perform are three separate questions.
The distinction applies beyond code. User and system messages can carry instructions rather than disposable observations. Cache-marked blocks can already be part of a provider’s cache key. A retrieval result may be the original content recovered from an earlier compression and should not be compressed again into a marker the model cannot redeem. In each case, the router protects a relationship between the bytes and the next operation.
When the safest transformation is no transformation
Headroom’s safety gates create deliberate passthrough cases. Small content may not contain enough redundancy to justify routing and storage. Malformed JSON cannot safely enter a JSON-specific path. A failed AST parse returns the original rather than a damaged outline. A candidate that expands the input, or fails to clear its acceptance condition, is rejected. Already-compressed content is pinned so repeated turns do not keep changing its representation.
These gates make the result less visually dramatic, but they protect the system’s actual contract. A compressor that transforms every block would be easy to advertise and difficult to trust. A compressor that refuses a short, instruction-bearing, cache-marked, malformed, or patch-sensitive block is making a more informative decision: the expected benefit does not justify the risk under the current conditions.
The same restraint should govern how readers interpret the published ranges. The project’s tables describe directional or workload-specific savings, and the larger gains tend to occur where repetition is high. They do not establish that every search result, code file, log, or prose document will shrink by the listed amount. They also do not establish that a selected subset will answer every future question. A no-op is not evidence that the whole system failed; a large reduction is not evidence that the model retained everything it might later need.
The useful mental model is a ladder of fidelity. A structural fold may preserve the data while changing its textual form. An AST outline may preserve syntax and interfaces while omitting implementation detail. A lossy route may preserve the most likely signals and rely on CCR when detail is needed. A protected passthrough preserves the source bytes. Headroom’s decision is sound only when the chosen rung matches the action that follows.
From content type to practical judgment
The deeper lesson of these compressors is that context optimization is a problem of matching representation to use. SmartCrusher reduces repeated JSON structure before it considers selecting rows. Search and log paths preserve the evidence that explains a query or failure. Diff handling protects patch semantics. Code handling separates a useful structural view from exact edit material. Plain-text compression remains more dependent on model behavior and workload shape.
That is why the next judgment cannot be made from the visible token count alone. Ask what was removed, what stayed visible, whether the form is byte-exact or merely information-preserving, and whether an omitted detail can be recovered. The next chapter connects those questions to the recovery mechanism itself: how Headroom stores original content, identifies it with markers, and gives the model a way to ask for more when the compact view is not enough.