When Detection Becomes a Timeline: Masks, Tracking, and Scene Cuts
For Jasna, the first quality decision happens before BasicVSR++ or LTX restores anything: the system must decide which pixels count as a mosaic in each frame. This chapter examines how a detector choice becomes a time-aware track, how short gaps and false positives are handled, and why a hard scene cut ends a track even when the target seems to remain in the same place. The practical goal is to distinguish a missed detection, a false positive, a broken track, and a legitimate shot boundary before changing the restoration model.
A video gives Jasna a new detection problem on every frame. A single frame can suggest a region, but a stable restoration requires a coherent answer across time. If the selected area flickers in and out, the restorer receives broken clips. If two shots are treated as one continuous scene, information from one shot can influence another. Detection and tracking therefore do more than mark rectangles: they define the evidence and the temporal boundaries of the work that follows.
Detection is the first quality gate
Jasna's default detector is rfdetr-v6, described by the project as fast and suitable for most videos. That default is a practical starting point, not a declaration that one detector is best for every source. A larger RF-DETR model may help with 4K content at higher cost. A VR-specific detector is recommended for VR180 material. Lada YOLO models may be useful for 2D animation, and the project notes that lada-yolo-v4 can be faster on AMD when detection speed matters more than detection quality.
These choices are meaningful because the detector determines what enters the rest of the pipeline. Choosing a detector is not like choosing a color filter after the important work is complete. It changes the set of regions that can become tracks, crops, and restored patches. A detector that is fast but misses small or unusual mosaics may leave the video untouched in the places that matter. A detector that is more sensitive may identify more candidates, but it can also create false work or demand more processing.
The detection score threshold makes that trade visible. Jasna gives each model a recommended threshold and lets the user adjust it. Lowering the threshold can help when real mosaics are missed; raising it can reduce false detections when normal regions are being selected. Neither direction is a universal improvement. The right threshold depends on the content, the detector, the hardware backend, and what kind of error is more costly for the job.
Consider two simple cases. In the first, a mosaic is present for many frames but the detector misses it in a few of them. If those misses are treated as definitive, one continuous subject can be split into several shorter pieces. In the second, a texture or ordinary object briefly resembles a mosaic. If that one-frame hit is accepted without qualification, Jasna may create a restoration request that has no valid target. The threshold changes how often these cases begin, but it does not by itself decide how a sequence should behave after the first detection.
That is why detector recommendations should be read conditionally. The repository's guidance supports choices for common content types, 4K, VR180, animation, and AMD speed, but it does not provide a head-to-head accuracy guarantee for every video. A useful selection process starts with the source's geometry and content, then uses a short scan or preview to see whether the chosen detector is missing targets or creating false ones. The question is not “which model wins in the abstract?” It is “which detector gives this pipeline usable evidence at an acceptable cost?”
From boxes and masks to a track
A detection in one frame is not yet a video object. To create a track, Jasna compares the new detection with regions it was already following. The public tracker uses bounding-box overlap—often described as intersection over union, or IoU—to decide whether a new box plausibly continues an existing one. A box that overlaps strongly enough can be attached to the active track; a box without a suitable match can begin a new track.
The box provides a compact description of location. The mask provides a more detailed description of the pixels inside that location. Jasna carries both through the active clip. This distinction matters because a rectangle can include ordinary pixels around a mosaic, while the mask helps the later blending stage restrict the generated change to the detected region. Tracking the box answers “which region is this over time?” Carrying the mask answers “which part of the region should influence the frame when restoration returns?”
The matching process is intentionally local and sequential. It compares current detections with the most recent boxes rather than proving a permanent identity for an object. That is enough for the job's purpose—assembling a coherent clip—but it does not guarantee perfect identity under every motion, occlusion, shape change, or crowded scene. The public code supports the mechanism of overlap-based association; it does not establish that every unusual sequence will track correctly.
When the match succeeds, the active clip grows. When the detector produces no matching box, Jasna has to choose between treating the region as temporarily absent and ending the track. That decision is controlled by a bounded detection gap. During a short gap, the tracker can coast by reusing the last box and mask. If the mosaic returns at the same location quickly enough, the clip can remain continuous instead of being split at the missing frame.
The word “bounded” is essential. Reusing the last location for too long would turn uncertainty into invented certainty. The target may have moved, disappeared, or been replaced by another object. A long coast could cause a generated patch to be placed over ordinary content. Jasna's design therefore treats gap filling as a repair for brief detector instability, not as permission to ignore missing evidence indefinitely. The documented default for the maximum detection gap is two frames, and the user can turn the feature off with zero.
A second control, minimum detection duration, addresses the opposite problem. A very short detection may be a false positive rather than a real mosaic. Jasna can discard detections shorter than the configured duration, leaving those frames unchanged. The documented default is also two frames. But a high minimum would erase genuine short mosaics, so this control is not a general “make detection more reliable” switch. It chooses which kind of error to tolerate: unnecessary restoration work or missed brief targets.
Together, the two controls describe a small temporal decision system. Gap filling says, “a short absence may not mean the region ended.” Minimum duration says, “a short presence may not mean the region was real.” They solve different problems and can pull in opposite directions. A reader who sees a one-frame artifact should not automatically increase the gap, and a reader who sees a missing short target should not automatically increase the minimum duration. The visible symptom has to be connected to the relevant side of the decision.
A scene cut is a different kind of absence
A hard scene cut creates a more fundamental boundary than a missed detector result. Before the cut, a tracked box belongs to one shot; after the cut, a visually similar box may belong to a different shot with different content, lighting, scale, and motion. Continuing the same clip across that transition would allow the temporal restorer to mix unrelated evidence. Jasna therefore uses scene-cut detection to end tracked clips at hard cuts.
The implementation looks for a sudden change in frame content rather than trying to identify the semantic subject of the shot. It downsamples frames to roughly 192 pixels wide, converts them to luma—the brightness component of the image—and compares the mean absolute difference between consecutive frames. A cut is reported when the score rises above both a minimum content threshold and an adaptive multiple of the recent median score. The recent median gives the detector a moving baseline: ordinary motion can raise the scores without automatically becoming a cut, while a sharp spike can stand out.
This is a sensible mechanism for the specific boundary it targets. It is inexpensive to run on the GPU, and it gives the tracker a clear event at which to flush active clips. It also has a clear limit. A hard-cut detector is not a universal editor or a semantic understanding system. Gradual dissolves, flashes, fast camera motion, and other unusual transitions can produce behavior that differs from a clean hard cut. The correct claim is that scene detection protects clips at the supported kind of boundary, not that it perfectly understands every transition in a video.
The same screen location can make this distinction difficult to see. Imagine a cut from one shot to another where the mosaic appears in the same corner. A location-only rule might treat that as continuity. Scene detection supplies an independent reason to end the old track, because spatial coincidence does not establish that the temporal evidence is related. In a restoration system, refusing to merge two uncertain contexts can be safer than producing a longer clip that looks smooth but combines incompatible frames.
The state diagram makes an important distinction visible: “not detected” is not always the same as “the clip ended.” A short gap can preserve continuity, a short presence can be discarded, a long clip can split while continuing through overlap, and a scene cut can terminate the track immediately. The later restoration stage inherits the result of these decisions; it does not get to reconstruct the missing temporal logic afterward.
Read detector behavior as evidence, not as truth
A detector score is evidence that a region may be a mosaic. It is not a final statement about what the region is. Jasna's architecture turns that evidence into a sequence of decisions: thresholding, matching, gap handling, duration filtering, and scene boundaries. Each step can improve usability by suppressing noise or preserving continuity, but each can also create a different error if its assumptions do not fit the source.
This is why changing the restoration model is often the wrong first response to a detection problem. If the target is absent from the track, BasicVSR++ and LTX are being asked different versions of the wrong question. Conversely, if a false track selects ordinary content, a more detailed model may make the unwanted change more convincing rather than more correct. The detector and tracker are not clerical preparation; they define the model's evidence.
A disciplined adjustment starts with the symptom. An untouched mosaic suggests a missed detection, an unsuitable detector, or a threshold that is too high. A one-frame or short-lived unwanted patch suggests a false positive or a minimum-duration setting that is too permissive. A visible gap in an otherwise continuous target suggests a short dropout, but increasing the allowed gap is safe only when the region is likely to remain in place. A seam at a shot change suggests a scene-boundary issue, while a seam where a long target was divided points toward clip construction and overlap—the next part of the system.
The general rule is simple but consequential: use detector recommendations as starting points, then test them against the actual source. The project documents useful defaults and content-specific alternatives, but it does not establish a universal threshold or detector ranking. A good choice is the one that supplies reliable enough regional evidence for the rest of Jasna's pipeline without spending more time or creating more false work than the job can justify.
The next question follows directly from a successful track. Once Jasna knows which frames belong together, it still has to decide how many to process in one clip and how much neighboring context to share between clips. That boundary determines whether temporal continuity improves smoothly or becomes a trade between seams, repeated work, and VRAM.