Inside the Detector: From Image Patches to Object Queries
For RF-DETR, the architectural question is how to preserve enough visual context for accurate localization while avoiding the cost of treating every image token as equally related to every other token at every layer. The answer is a staged path: a DINOv2-derived backbone produces spatial features, a projector puts selected stages into a decoder-ready form, and a limited set of object queries looks into those features through deformable attention. Tracing that path lets the reader judge which parts of the real-time claim come from representation, which come from attention locality, and which come from the decoder’s fixed computation budget.
The important unit is not a named module in isolation. It is the interface between modules: the backbone must return features that still have usable image geometry; the projector must align their scales and channels; the decoder must know where each query is looking; and the final heads must turn the resulting query states into boxes and class scores. RF-DETR’s speed-oriented design emerges from those interfaces working together.
Why the backbone returns more than one view
A classification model can often compress an image toward one global decision. Detection cannot afford to lose the arrangement of evidence. A small object may be recognized by a local texture, while its class or disambiguation depends on broader context. A large object may need coarse spatial coverage as well as fine boundaries. RF-DETR therefore uses the DINOv2 backbone not just as a final feature vector, but as a source of selected intermediate feature maps.
The first step is patch embedding. The input image is divided into non-overlapping patches, and each patch becomes a token for the vision transformer. The patch size is an architectural parameter, not an arbitrary resize setting. Once the tokens pass through the encoder, RF-DETR selects specified intermediate stages and restores them to spatial feature maps. In conceptual terms, the model preserves several snapshots of the visual representation as it becomes deeper: earlier views retain more local geometry, while later views carry more abstract semantic information.
Those views do not arrive at the decoder ready to be mixed. They may have different spatial resolutions, and their channels must be made compatible with the decoder’s hidden dimension. RF-DETR’s multi-scale projector performs that adaptation. It can upsample or downsample a selected feature, concatenate compatible contributions, fuse them through convolutional blocks, and normalize the result. The output follows feature-pyramid logic: each configured level represents the image on a particular spatial scale, while the decoder sees a common channel width.
This is why “DINOv2 backbone” is an incomplete description of the detector. A backbone trained to represent images is being used as a spatial feature generator. The selected stages and the projector determine what information crosses the boundary into detection. If only a final global representation were passed forward, the decoder would have less direct access to the geometry needed for box regression. If every intermediate tensor were carried forward without a scale policy, the decoder would inherit unnecessary memory and alignment costs.
The current RF-DETR configurations make that policy explicit. They select several encoder stages, then specify which projected pyramid scales feed the decoder. The named variant determines the exact resolution, patch size, window count, and stage configuration. These are not implementation trivia: they define the spatial lattice on which the rest of the computation operates.
How locality and global mixing share the budget
Self-attention is attractive because any token can, in principle, exchange information with any other token. Its cost grows quickly with the number of tokens, however. Increasing image resolution increases the patch grid, and the cost of dense attention rises with that grid. A real-time detector therefore needs a way to preserve useful context without paying for global interaction everywhere.
RF-DETR’s windowed DINOv2 path divides patch tokens into spatial windows for selected encoder blocks. Within a window, attention still lets the local tokens interact, but the block avoids a full image-wide attention matrix. Other blocks use full attention. In the implementation, the windowed blocks are identified separately from the blocks whose outputs are selected for detection; when the default routing is derived, the selected feature-producing positions are treated as the complement of the windowed set. The result is a pattern of local processing interrupted by global mixing, rather than a pure choice between “all local” and “all global.”
That pattern has a systems consequence. Local windows reduce the cost of many encoder layers, while full-attention points allow information to move across distant image regions before the decoder consumes the features. The design does not promise that locality is free: the number of windows, the patch size, the input resolution, and the chosen blocks all affect memory and runtime. It says instead that the budget can be spent selectively. Global context is valuable, but not every layer must purchase it at full spatial scale.
The repository also supports explicit window-block overrides in the lower-level backbone construction. That is useful when an exact architecture needs to be pinned, but it is a reason to avoid overinterpreting a configuration name. The implementation notes that its derived schedule preserves historical checkpoint behaviour and is not guaranteed to match every paper-level layer schedule. What can be stated confidently is the operating principle visible in the code: windowed and full attention are routed as distinct modes, and their arrangement is part of the architecture.
The diagram is best read as a flow of responsibilities, not as two independent encoder branches. Windowed and full-attention blocks are alternative operations applied at different points in the same backbone. Their outputs are collected at selected stages, projected into the representation consumed by the decoder, and then queried by the detection mechanism.
Queries that look where objects might be
After the backbone has produced spatial memory, RF-DETR does not create a prediction at every pixel and then rely on a separate suppression stage to remove duplicates. It maintains a limited set of learned object queries. Each query carries a feature representation and a reference point or reference box that tells the decoder where to begin looking. The query is not a preassigned object label; it is a slot that can become responsible for whichever object the learned matching process makes it represent.
The decoder first lets the queries interact with one another through self-attention. This gives the query set an opportunity to coordinate before each query reads the image. It then performs cross-attention against the flattened spatial feature levels. RF-DETR uses multi-scale deformable attention for this step. Instead of forming a dense relationship between one query and every spatial token, the query predicts offsets and weights for a small collection of sampling locations around its reference point, across the configured feature levels.
The distinction is subtle but important. Deformable attention is not the same as cropping a fixed patch and ignoring the rest of the image. The reference point can move, and the learned offsets can select evidence at different scales. A query looking for a small object may need a concentrated pattern on a fine feature level; a query refining a large object may draw from wider or coarser evidence. The attention weights determine how the sampled values contribute to the updated query state.
This mechanism shifts computation from “compare everything with everything” to “learn where useful evidence is.” That is the architectural reason the projector and reference geometry matter. The decoder can only sample intelligently if the feature levels preserve spatial correspondence and if the reference coordinates are expressed in the same normalized geometry. A mismatch at that interface can look like a model-quality problem even though the failure is really a coordinate or shape contract violation.
The decoder repeats this process across layers. After a layer updates the query state, a box head predicts a new reference adjustment. The next layer can therefore attend around a more informed location. Iteration matters because localization is rarely solved in one step: a coarse hypothesis can first identify a region, then refine its center and extent using the features sampled around that region. The model is not simply making independent guesses; its architecture allows successive states to improve the spatial hypothesis.
The number of queries is another deliberate budget. RF-DETR’s detector uses a default query count of 300 for the public detection path, while grouped query embeddings expand the training-time set to accelerate learning. In evaluation and inference, the model uses one query group rather than carrying every training duplicate forward. This separation lets training obtain more assignment opportunities without making the deployed prediction path pay for all of them. It also means that “300 queries” describes a real capacity and selection boundary: the model cannot represent an unlimited number of independent object slots merely because the input image contains more objects.
Why two-stage proposals change the starting point
RF-DETR’s two-stage configuration adds a proposal step before the regular decoder refinement. The encoder memory is converted into candidate spatial proposals at the available feature-map levels. An encoder-side class head scores those positions, and the highest-scoring candidates are selected. Their memory features and box proposals become the starting material for the decoder’s reference points.
This changes the decoder’s initial question. With only learned references, a query must learn how to move from a general starting point toward an object. With two-stage proposals, some starting references already come from image-conditioned encoder evidence. The decoder still has to classify, localize, and refine them, but it begins with a set of candidates ranked by the encoder. The proposal stage is therefore not a replacement for the decoder; it is a way to connect the decoder’s limited query budget to the spatial evidence the encoder has already organized.
The implementation is careful about where the expensive box transformation happens. It ranks encoder positions first, gathers the selected memory rows, and applies the pointwise box head to those survivors rather than transforming every encoder position only to discard most of them. That optimization follows from the head’s local structure: each candidate’s box delta depends on its own feature, not on a cross-token interaction. The result preserves the same selection logic while avoiding work that cannot affect the chosen references.
Reference refinement then continues inside the decoder. RF-DETR represents box centers and sizes in a normalized coordinate system, and the decoder’s deformable attention uses those references to determine sampling locations. Each layer can update the reference through a predicted delta. The process couples classification and localization: a query’s feature state changes as it samples image evidence, and its new box determines where the next layer looks. The final box is thus the outcome of a short feedback loop between query representation and spatial hypothesis.
This also explains why a change in input resolution is not merely a preprocessing preference. A different resolution changes the patch grid, the position-encoding grid, the feature-map geometry, and potentially the valid window partition. RF-DETR can interpolate positional embeddings in supported situations, but the model configuration still has to describe a coherent geometry. If a custom patch size or positional grid no longer matches the native DINOv2 setup, the code warns that the original DINOv2 backbone weights cannot be loaded as-is. Fine-tuning can still be intentional, but it is no longer the same pretrained architecture in the strong sense.
Architectural boundaries are part of the interface
The first hard boundary is spatial divisibility. The height and width supplied to the backbone must be divisible by the product of the patch size and the number of attention windows. This ensures that the patch grid can be partitioned into windows and later reconstructed into spatial feature maps. A resolution that looks harmless numerically can be invalid for a particular variant if it breaks that product.
The second boundary is query capacity. The decoder has a fixed number of query slots, and the postprocessor selects a bounded number of high-scoring query/class pairs. That does not mean every image must contain fewer objects than the query count, because class scoring and selection have their own semantics; it does mean the design is intentionally bounded. Increasing the query budget changes parameter shapes, memory, matching work, and often checkpoint compatibility. It is an architectural change, not a threshold adjustment.
The third boundary is the relationship between configuration and pretrained weights. RF-DETR warns when overrides affect encoder shape, hidden dimensions, decoder depth, attention heads, feature stages, projector scales, patch size, or other architecture-defining fields. A model can sometimes be constructed after such an override, but a partial load leaves substantial regions randomly initialized and typically lowers accuracy. The fact that a Python object exists is not proof that the intended pretrained detector survived the change.
Taken together, these boundaries clarify RF-DETR’s real-time design. The system reduces cost by selecting where to preserve global context, where to sample features, how many queries to maintain, and how much work to repeat during refinement. Those choices are coupled: more resolution supplies more detail but increases tokens; more queries increase capacity but increase decoder and matching work; a different window arrangement changes the backbone’s computation and checkpoint assumptions. The speed claim is therefore a property of a coherent configuration, not a sticker that remains unchanged after arbitrary modifications.
The forward pass now has a complete shape: image patches become spatial DINOv2 features; selected stages become decoder-ready levels; queries carry reference geometry; deformable attention samples evidence around those references; two-stage proposals can improve the starting candidates; and iterative updates produce boxes and class scores. The next question is how those unordered predictions learn which object each one should represent—and how the same query machinery becomes a mask or a body pose rather than only a box.