How to Know the CLI Worked

CLI-Anything treats testing as the evidence pipeline for the boundary between an agent command and a real software artifact. This chapter asks what each test layer can establish, why an installed-command test is different from an imported function, and how domain-specific output checks turn “the process succeeded” into a narrower, defensible claim. The reader should leave able to match a test to the failure it can catch—and to recognize the evidence a test cannot provide.

Each test layer answers a different question

A harness can fail at several distances from the final artifact. Its core function may mishandle a state transition. Its project writer may produce malformed XML, JSON, or another native structure. The target application may reject the project or render it differently from the harness's assumptions. The installed command may not be discoverable from PATH even though the source module works. CLI-Anything's testing method separates these failure surfaces rather than compressing them into one pass/fail number.

The first layer is unit testing over core logic and synthetic data. It is the fast layer: a function can be tested with a small in-memory project, a parsed timecode, or a deliberately invalid parameter. This can establish that validation rules, state updates, and error paths behave consistently. It cannot establish that a real renderer understands the generated project, that a media file has the expected properties, or that an installed entry point is wired correctly. Its strength is isolation; its boundary is that it has removed the external software on purpose.

The second layer tests native or intermediate project files. A harness can create a real file and inspect whether its XML is well-formed, its JSON contains the required relationships, or its archive has the expected structure. This catches errors that an in-memory unit test may miss, such as a serialization omission or an invalid package layout. It still stops before the target application's full semantics. A valid project file is evidence that the representation is structurally plausible, not evidence that the real application will render every requested effect.

The third layer invokes the real backend and checks the final output. This is where the rendering gap becomes testable. A document conversion can be checked for a valid file signature or package structure. A video or image workflow can be checked through frames, brightness, color, duration, or other properties that correspond to the requested operation. An audio workflow can examine levels or spectral behavior. The exact measurement belongs to the domain, but the logic is consistent: inspect the artifact that matters rather than infer success from the absence of an exception.

The fourth layer invokes the installed command as a user or agent would. CLI-Anything's subprocess pattern resolves the installed console command and can force tests to use that installed path rather than silently falling back to a source-module invocation.