How the Queries Learn: Matching, Losses, Masks, and Keypoints
RF-DETR’s query decoder produces a fixed set of hypotheses, while a real image may contain no objects, one object, or many objects of different sizes. The learning problem is therefore not “which class belongs to query 17?” but “which predictions should be paired with which truths, and how should the remaining predictions be discouraged from claiming an object?” The useful way to read RF-DETR is to follow that assignment and supervision contract from boxes to masks and keypoints. It explains both the model’s unusual output semantics and the practical consequences of choosing a task head.
Assignment makes set prediction learnable
A conventional detector often begins with a dense collection of candidate locations whose roles are tied to a grid or anchor design. RF-DETR begins with object queries. They are useful because each query can gather evidence from the whole set of spatial features, but they create an immediate bookkeeping problem: the image does not come with a fixed correspondence between truth objects and query slots.
Suppose an image contains three annotated objects and the detector has hundreds of query slots. There are many plausible ways to label the slots. One query might describe the first object well but another might describe it slightly better; assigning both to the same truth would reward duplicate predictions and leave another object unsupervised. RF-DETR addresses this with Hungarian one-to-one assignment. The matcher considers the available predictions and truths together, chooses a minimum-cost pairing, and leaves the other queries unmatched for that image. The query’s identity is learned through this repeated competition; it is not a permanent semantic label stored in the slot number.
The matching cost reflects the same three requirements the detector must satisfy at inference. A prediction should assign a plausible class, place its box near the target, and overlap the target with a good geometric shape. RF-DETR combines a focal classification cost with an L1 box cost and a generalized-IoU cost. The weights of those terms determine what “best pairing” means during training. A query with a confident class but a badly placed box is not equivalent to one with a slightly weaker class score and a much better localization. Assignment makes that tension explicit before the loss is computed.
The matcher is not itself the final objective. Once it has produced pairs, the criterion supervises the matched class and box predictions. Classification uses sigmoid/focal-style supervision, while box regression uses L1 and generalized-IoU losses. Unmatched queries receive no positive class assignment, so the system learns to keep their scores low rather than forcing every query to explain an object. A cardinality error is also calculated as a diagnostic: it compares the number of confident non-empty predictions with the number of targets, but it does not propagate gradients as a training loss.
This separation between assignment and loss is central to the model’s behaviour. The matcher decides which prediction is responsible for which target for the current image; the criterion then improves the responsible prediction and suppresses unsupported alternatives. The same process is repeated for auxiliary decoder outputs and, in the two-stage configuration, encoder outputs. Those intermediate predictions are not normally exposed as the deployed result. They provide additional learning signals at earlier states, helping the query and box representations become useful before the final decoder state is reached.
The distinction also explains why a small mismatch can have large consequences. If class IDs are wrong, the matcher pairs queries against the wrong semantic columns. If box coordinates use a different convention, the geometric costs no longer measure the intended error. If a custom head changes the number of classes or keypoint slots without corresponding targets, the model may still produce tensors but the assignment problem is no longer the one the training code expects. Set prediction is flexible, but it is not indifferent to the structure of the labels.
One query can contribute more than one class candidate
At inference, RF-DETR does not interpret each query by taking one class argmax and then treating the query as a single indivisible detection. Its regular path applies a sigmoid to class logits, flattens the query/class scores, and selects the highest-scoring pairs globally. This is a different contract from a softmax classifier in which each query must choose exactly one class.
The difference matters in crowded or ambiguous scenes. A query can score above threshold for more than one class, and global query/class selection can preserve more than one such candidate when the scores warrant it. That behaviour is why a decoder that performs a per-query argmax may silently drop predictions that RF-DETR intended to retain. It is also why the exported graph’s raw logits cannot be treated as already-decoded detections.
The top-k step gives the system a bounded output budget. RF-DETR’s public detection path uses a default query count of 300, and its postprocessor selects a bounded number of query/class pairs. The limit is not merely a display preference: it controls how much candidate computation, matching, and postprocessing the system performs. At the same time, it is not accurate to describe every selected pair as one query-owned object in the conventional sense, because the selection is over the flattened pair space.
This design avoids relying on an additional conventional non-maximum-suppression story to explain the output. The model learns a set through assignment, then the postprocessor ranks the candidate pairs and converts their normalized center-format boxes into pixel-space boxes. The output is consequently the product of two stages: learned responsibility during training and score-based selection during decoding. Changing either stage changes what the reader should expect from the model.
The sequence is deliberately asymmetric. The queries generate candidate predictions, while the targets do not become query states; they only constrain which predictions are rewarded. The matcher is the bridge between a variable-size set of truths and a fixed-size set of model outputs. The same bridge is reused when task-specific costs are added.
Masks require spatial-query interaction
Instance segmentation does not replace the box task with a completely separate model. It adds a mask head to the shared RF-DETR path. The decoder still produces a query representation and a box/class prediction, but each selected query also needs a spatial mask describing the pixels belonging to that instance.
RF-DETR’s segmentation head combines two kinds of information. It takes spatial features from the backbone side and query features from decoder layers, projects them into a compatible interaction space, and combines them to produce per-query mask logits. In effect, the spatial map supplies “where image evidence is available,” while the query supplies “which instance’s mask is being requested.” The query-conditioned construction is what keeps two nearby objects from being forced to share one class-agnostic foreground map.
Training those masks at every pixel of every possible query would be expensive, especially when most query slots are unmatched. RF-DETR therefore uses a sparse training path and point-based mask supervision. It samples points in the predicted masks, gives priority to uncertain locations, and compares the sampled predictions with sampled ground-truth masks using sigmoid cross-entropy and Dice terms. Boundary points are particularly informative because a mask can have a plausible interior while still placing the instance edge incorrectly. Concentrating work around uncertainty is a computation choice that also reflects the structure of the error.
The mask costs participate in assignment when segmentation is active. A pairing is judged not only by class and box geometry, but also by how well the predicted mask agrees with the target at sampled points. This prevents a query with a good box but the wrong instance shape from looking equivalent to a query that explains the full segmentation task. The shared matcher thus changes with the head: the meaning of a “good prediction” is task-dependent.
Because masks are spatial outputs, segmentation has a task-specific computation and evaluation surface beyond a box and class score. Its query-conditioned head, mask-aware assignment costs, and point-sampled BCE/Dice supervision make mask quality part of what a “good prediction” means. A segmentation result should therefore be compared on its own mask metric and runtime profile, not treated as box detection with an extra label.
Keypoints extend the query from an object to its geometry
The keypoint preview uses the same broad idea—an object query gathers evidence, then a task-specific head predicts structured outputs—but the structure is different from a mask. A person can have a bounding box and also a set of joints, each with a location and visibility state. RF-DETR’s GroupPose-style path conditions keypoint queries on the detection representation and processes them with keypoint-specific attention and feed-forward layers.
A keypoint prediction slot carries more than an x-coordinate and y-coordinate. The implementation records location, a findability logit for whether the point can be identified, a visibility logit for whether it is fully visible, precision or uncertainty parameters, and a class contribution that can be aggregated into detection-class logits. This lets the head express not only where a joint is, but how confidently it can be found and how uncertain its localization is.
That uncertainty is not discarded after the head makes its prediction. RF-DETR can fuse keypoint localization uncertainty into the object-level score during postprocessing. The resulting detection confidence is therefore not simply a raw class score: the reliability of the predicted geometry can influence whether the instance clears a threshold. This gives the head a richer contract than a box-only detector, but it also means that a threshold must be interpreted together with the task and its uncertainty model.
The schema is part of the model state. Custom keypoint training can use a dataset-defined keypoint count and class schema from COCO keypoint JSON or Ultralytics YOLO pose data. A one-class pose model and a multi-class pose dataset do not share the same slot layout automatically. The training and checkpoint loaders preserve schema information so that a model can reconstruct the output it was trained to produce.
This feature remains deliberately bounded. The keypoint model is an early-access preview, is available through the rfdetr package, and is not yet exposed through the external inference package. The architecture shows how RF-DETR’s query contract can support structured geometry, but its maturity and API boundaries should remain visible when deciding whether to use it.
What the training objective reveals about the design
The training objective makes RF-DETR’s real-time architecture easier to interpret. A fixed query set supplies bounded candidate capacity. Hungarian assignment turns that capacity into variable object responsibility. Focal class and geometric box losses shape the main detection output, while auxiliary losses strengthen intermediate states. Segmentation and keypoint heads reuse the object-conditioned query representation but add task-specific spatial or geometric supervision. The result is a single family with a shared operating pattern, not one identical output tensor for every task.
There are corresponding failure modes. A model can be accurate on class assignment but poor at localization; a segmentation model can preserve boxes while losing mask boundaries; a keypoint model can retain an instance while expressing high uncertainty for particular joints. A single confidence threshold cannot be assumed to have the same meaning across these outputs. Similarly, increasing the number of queries or selected outputs changes the computation budget and may affect checkpoint compatibility; it is not a free way to solve every crowded-scene problem.
The most important practical consequence is that RF-DETR’s outputs must be read as a contract. During training, targets determine responsibility through assignment. During inference, sigmoid scores and global query/class selection determine which candidates survive. During segmentation or keypoint decoding, additional heads introduce their own geometry, schema, and uncertainty. A reader who understands that contract can compare variants more intelligently: the next question is no longer only how many milliseconds a model takes, but what kind of prediction that time buys, and how the model’s size and runtime move it along the measured frontier.