How Video-Use Makes Video Legible to an Agent
video-use is an open-source, conversation-driven editing skill built around a deceptively difficult question: how can an agent make useful decisions about moving images without being forced to ingest a continuous stream of video? This series examines video-use's answer—turn footage into a compact, audio-first evidence surface, inspect selected visual moments, and surround the agent's judgment with explicit rendering and verification rules. The goal is not merely to name the project's tools, but to understand what each boundary makes possible and what it still leaves to an editor.
A different starting point for video editing
The project's promise is easy to picture. A person places raw footage in a folder, tells an agent what they want—perhaps a launch video, a tutorial, or a coherent cut from several takes—and receives an output in an edit/ directory. That sounds like a single act of “AI editing.” The repository describes something more disciplined: the agent inventories the material, proposes a strategy in ordinary language, waits for approval, produces an edit decision list (EDL), renders the selected ranges, checks the result, and keeps a record for the next session.
An EDL is simply a structured account of what to keep and where it comes from. It connects a source clip to a start and end time, and can also carry the beat, quote, reason, grade, overlays, and subtitle choices that make the decision intelligible. The distinction matters because video-use does not ask one opaque model to leap from pixels to a final file. It gives different jobs to different stages: evidence describes the material, editorial judgment chooses among possibilities, media helpers execute the choice, and inspection tests the artifact that was actually produced.
That division also clarifies the project's relationships. User footage is the material being shaped. A shell-capable agent—Claude Code or another compatible host—is the conversational decision-maker. ElevenLabs Scribe supplies the timing-rich transcript. Python helpers and ffmpeg transform ranges into media. Optional animation tools can create overlays, while the central renderer still decides when those overlays enter the composition. The result is a system whose behavior emerges from interfaces between a person, an agent, external transcription, and deterministic media operations.
Consider what this means for a familiar editing problem: several takes of one spoken sentence, each with slightly different pacing. The agent does not need to compare every frame of every take before it can identify the useful material. Word timings can reveal a clean opening, a false start, or a pause that separates one thought from the next. That narrows the search. But the narrowed search is not the verdict. A visual check may show that the cleanest sentence begins with the speaker already moving, that a cut removes a meaningful reaction, or that two adjacent takes have incompatible lighting. The architecture is valuable because it lets each kind of evidence answer the question it is suited to answer.
This also explains why “understanding the video” is too broad a description. A transcript can establish that a phrase was spoken and approximately where it belongs. It cannot establish that the speaker's expression matches the sentence, that a background object disappears at a cut, or that a subtitle will be readable above a platform's interface. Conversely, a filmstrip can reveal a visual jump without explaining whether the spoken wording is the best version. video-use does not erase these differences; it preserves them as separate inputs to judgment. That separation makes a decision easier to inspect because a reader can ask not only what the agent chose, but which evidence justified the choice.
Read the structure before touching the frames
The central design choice is that the language model does not watch the entire video. The repository makes the transcript the always-loaded view and treats visual inspection as an on-demand supplement. This is not a claim that pictures are unimportant. It is a claim about representation: a frame stream contains enormous amounts of information that may not help answer an editorial question, while speech has a useful built-in structure of words, pauses, speakers, and audible events.
The first layer therefore records speech with word-level timestamps, speaker labels, and events such as laughter or applause. The second layer can render a bounded timeline view containing a filmstrip, waveform, word labels, and shaded silence gaps. The agent can reason over the text most of the time, then ask for a visual range when a pause, retake, or boundary is ambiguous. In the repository's own terms, this resembles giving an agent a structured representation of a webpage instead of making it interpret a screenshot at every step. The analogy is useful only up to a point: video-use still relies on sampled images when continuity or performance cannot be inferred from words.
This arrangement solves two opposing problems. A transcript alone can tell the editor that a speaker stopped between two phrases, but not whether the camera jumped, a gesture was cut off, or a reaction should remain. A naive visual approach can expose thousands of frames while making the relevant moment harder to locate. video-use places the compact representation on the normal path and spends visual attention at decision points. The important resource being managed is not only compute or tokens; it is the editor's ability to connect a proposed cut to evidence.
The word “surface” is important here. The transcript is not the final truth about the footage, and the packed markdown view is not a replacement for the raw timestamps. They are interfaces with different purposes. The compact view lets an agent scan several takes, compare wording, and locate pauses in a form that fits ordinary reasoning. The underlying word records keep the precision needed when a boundary is finally chosen or when captions are placed on the output timeline. In other words, video-use compresses the material for thought without asking the renderer to trust a compression that discarded the timing it needs.
That design also makes disagreement visible. If the text suggests a clean cut but the timeline view shows an awkward gesture, the problem is not hidden inside a mysterious model score; it becomes a reason to inspect or revise the EDL. If the rendered result has a flash or an audio pop even though the selected words were correct, the defect belongs to the execution stage and can be checked there. The project therefore treats an edit as a chain of claims about media, each of which should have an evidence source and a place where it can fail.
The flow is deliberately not a straight arrow from “video” to “final.” The visual timeline can feed back into the decision, and a rendered-output issue can send the edit back for revision. That feedback structure is the project's most important practical implication: the evidence used to choose a cut and the artifact used to judge the result are related, but they are not the same thing.
The human boundary is part of the architecture
video-use also treats conversation as a control boundary. Before execution, the agent is expected to inspect the material, describe what it sees in plain language, ask questions shaped by the footage, and propose a strategy. The strategy might address the content type, pacing, target format, must-keep moments, grade, animation, or subtitle treatment. The agent should not silently decide all of those choices and then present an irreversible result as if the user's intent had been obvious.
This is especially important because many editing decisions are not reducible to a clean technical rule. Removing a filler word may improve pace, but preserving a laugh may be essential to the speaker's performance. A short silence may be dead air in one launch video and a meaningful pause in a documentary. The repository supplies hard production rules for problems such as cutting inside a word, misaligning an overlay, or hiding subtitles, while leaving rhythm and aesthetic direction open to judgment. Confirmation is where the person authorizes that judgment before the media is changed.
The helpers themselves do not magically enforce the whole conversation. The renderer accepts an EDL; it does not know whether the user approved the strategy. Likewise, the documented self-evaluation loop depends on an agent or person interpreting diagnostic views. That distinction is a strength when stated clearly: the project combines procedural obligations with mechanical operations instead of pretending that every desired behavior is encoded in one executable component.
Confirmation changes the meaning of the same evidence. A long pause might be marked for removal in a fast launch video, but kept to let an interview answer breathe. A speaker's laugh might be treated as noise in a tightly scripted tutorial, or as the moment that makes a testimonial credible. The transcript can expose both possibilities; it cannot select between them without a purpose. By asking for a strategy first, video-use gives the person a chance to state what “better” means before the agent optimizes for speed, coherence, or polish.
The same principle protects against a quieter failure: a technically valid edit can still be the wrong edit. The renderer may produce a clean file from perfectly timed ranges, yet the chosen sequence may bury the main point or flatten the speaker's delivery. Conversely, an editor may approve a deliberately abrupt cut for energy, provided they understand that it is an aesthetic choice rather than a rendering defect. Keeping authorization and execution separate allows the system to preserve that distinction instead of labeling every difference from a default as an error.
A model that includes memory and limits
The output contract reinforces the same boundary. Session artifacts—transcripts, packed evidence, EDLs, rendered clips, subtitles, verification images, and final output—belong beside the user's footage in edit/, not inside the skill repository. A project.md file records strategy, decisions, reasoning, and outstanding items. Transcript caching means a later session can reuse the evidence for unchanged sources rather than paying to transcribe them again. This is modest compared with a full project database, but it changes the interaction from a disposable command into a resumable working relationship.
Persistence is not just convenience. It gives later decisions a memory of why a particular take, grade, or unresolved issue was chosen, while keeping derived work separate from the source repository. That separation makes it easier to inspect or replace an artifact without confusing it with the original footage. It also exposes the system's actual state: if a transcript is cached, the next run can reuse it; if a visual check or final render is missing, the workflow has a visible gap rather than an implied success.
The architecture is useful precisely because it is not a promise of unrestricted visual understanding. It depends on Scribe for transcription and on ffmpeg and the host environment for media processing. The visual tool samples selected ranges rather than continuously classifying every shot. The repository documents a full-project timeline mode as not yet implemented, and its focused tests cover particular orientation and frame-rate hazards rather than proving every subtitle, audio, or continuity case. Current helper behavior can also be more authoritative than a worked example in the prose: for instance, the renderer's current subtitle safe-zone setting differs from an older example in the skill guide. A careful reader should therefore distinguish implemented defaults, design guidance, and unmeasured outcomes.
That is the standard this series will apply to video-use. It will follow the transcript from source audio into a readable editing surface, show how timestamps become word-safe cut decisions, examine why visual checkpoints are selective, and trace how the renderer protects the final file from silent failures. It will also keep asking where the repository's contracts end: at provider accuracy, at visual interpretation, at creative taste, or at the need for a person to approve and inspect the result.
The first answer lies in the transcript. For video-use, transcription is not a preliminary summary that can be thrown away once the agent has “understood” the footage. It is the timing substrate on which phrase boundaries, speaker changes, audio events, and later caption alignment depend. Understanding what that substrate preserves—and what can go wrong before an edit is even proposed—is the next step in understanding the project itself.