From Pretrained Checkpoint to Custom Model

The most consequential RF-DETR choice after selecting a model variant is how a run is initialized and what the training system believes the data means. A custom detector is not created by pointing at a folder and waiting for a loss curve. The folder structure selects a dataset loader, the annotations select a task schema, the model head must match the class space, and the checkpoint type determines what training state persists. Understanding those transitions lets the reader plan a first fine-tuning run and distinguish a genuine model problem from a data, configuration, or checkpoint mismatch.

The dataset is a schema, not just a folder

RF-DETR’s high-level training path starts with model.train(dataset_dir=...). The data layer looks at the directory structure to identify a supported format. A COCO-style dataset is recognized through split annotations such as train/_annotations.coco.json; a YOLO-style dataset uses a data.yaml or data.yml together with the expected image and label directories. If neither structure is present, the system fails rather than guessing.

That choice determines how labels cross into the model. COCO annotations link images, categories, boxes, and annotations in JSON. YOLO labels store a class index and normalized box coordinates per image. The two formats express similar detection information, but the class mapping is not an incidental serialization detail: it determines which output class a target supervises. A dataset can be syntactically valid and still be semantically wrong if its IDs, names, or coordinate conventions do not match the loader’s expectation.

The task head adds another layer to the schema. Detection needs boxes and class labels. Instance segmentation needs mask polygons or equivalent mask annotations, because a box cannot supervise the pixels belonging to one instance. Keypoint preview training needs keypoint-bearing COCO JSON or Ultralytics YOLO pose labels, including a keypoint schema. The same directory can therefore be structurally recognizable while still being unsuitable for a selected model head. A detection-only YOLO dataset is not automatically a pose dataset merely because it contains the word “person.”

This is why RF-DETR’s common API should not be mistaken for a common data contract. The call looks similar across detection, segmentation, and keypoints, but the annotations and outputs differ. For custom keypoint data, the model must know how many keypoints belong to each class and how horizontal flips should map paired joints. For segmentation, the mask geometry must survive resizing and cropping. For detection, the class list and box coordinates must remain aligned across splits. The model family is unified at the interface level; the meaning of each target remains task-specific.

Once the dataset is identified, the high-level call constructs the same underlying PyTorch Lightning pieces that the custom API exposes directly: a data module, a model module, and a trainer. The data module owns the train, validation, and test loaders. The model module owns the neural network, criterion, postprocessor, and optimizer/scheduler configuration. The trainer owns the execution strategy, accumulation, precision, callbacks, and distributed runtime. This separation is useful when the default path is not enough, but it also clarifies ownership: changing a loader setting does not change the model head, and changing a model configuration does not repair an invalid annotation schema.