feat: Add DETR, Deformable DETR, and RF-DETR object detection models#1165
Open
ahmedemam576 wants to merge 18 commits into
Open
feat: Add DETR, Deformable DETR, and RF-DETR object detection models#1165ahmedemam576 wants to merge 18 commits into
ahmedemam576 wants to merge 18 commits into
Conversation
Copy verbatim reference implementations for provenance tracking: - detr.py, transformer.py, matcher.py, position_encoding.py from facebook/detr - deformable_detr.py, deformable_transformer.py from fundamentalvision/Deformable-DETR Signed-off-by: josiah.smith <josiahsmithphd@gmail.com>
Modify the verbatim reference files from facebook/detr and
fundamentalvision/Deformable-DETR for use with TerraTorch's
ObjectDetectionModelFactory:
- Remove external dependencies (NestedTensor, util.box_ops,
util.misc, segmentation, backbone modules) and replace with
torchvision.ops equivalents
- Add terratorch_detr.py with TerraTorchDETR and
TerraTorchDeformableDETR wrapper classes that compose reference
components with BackboneWrapper
- Update __init__.py to re-export wrapper classes as DETR and
DeformableDETR
- Integrate with ObjectDetectionModelFactory ('detr' and
'deformable-detr' framework options)
- Add tests and example YAML config
- Ruff lint and format compliance
Signed-off-by: josiah.smith <josiahsmithphd@gmail.com>
… DETR and Deformable DETR Signed-off-by: josiah.smith <josiahsmithphd@gmail.com>
Copy verbatim model files from https://github.com/roboflow/rf-detr (commit 82ad5d1) under Apache-2.0 License for adaptation into TerraTorch's ObjectDetectionModelFactory. Signed-off-by: josiah.smith <josiahsmithphd@gmail.com>
Verbatim copy of facebook/detr models/segmentation.py containing mask prediction heads (MHAttentionMap, MaskHeadSmallConv), losses (dice_loss, sigmoid_focal_loss), and post-processing (PostProcessSegm, PostProcessPanoptic). Will be adapted for TerraTorch in a subsequent commit. Signed-off-by: josiah.smith <josiahsmithphd@gmail.com>
Adapt reference segmentation.py for TerraTorch (remove DETRsegm wrapper, PostProcessPanoptic, NestedTensor; keep MHAttentionMap, MaskHeadSmallConv, dice_loss, sigmoid_focal_loss, PostProcessSegm). Add loss_masks to both SetCriterion classes with masks skipped in aux loss loop. Integrate mask head into TerraTorchDETR and TerraTorchDeformableDETR via masks=True parameter. Update ObjectDetectionTask for DETR segmentation metrics/masks. Add component, criterion, and end-to-end segmentation tests. Signed-off-by: josiah.smith <josiahsmithphd@gmail.com>
Integrate RF-DETR (Roboflow Detection Transformer) into TerraTorch's ObjectDetectionModelFactory alongside existing DETR and Deformable DETR. - Add TerraTorchRFDETR wrapper with decoder-only LW-DETR transformer, two-stage encoder proposals, bbox reparameterization, and pure-PyTorch multi-scale deformable attention (no CUDA extension required) - Register 'rf-detr' framework in ObjectDetectionModelFactory - Remove vendored DINOv2 backbone (replaced by TerraTorch BackboneWrapper) - Adapt all RF-DETR components for ruff compliance (F→f_nn, assert→raise, Optional→union syntax, naming conventions, etc.) - Add comprehensive parity tests proving numerical identity with reference: gen_encoder_output_proposals, full LWDETR forward (no two-stage, two-stage+bbox_reparam, two-stage without), SetCriterion with ia_bce_loss and varifocal_loss including aux/enc outputs, position encoding, matcher, PostProcess, and SegmentationHead - Add functional tests for factory build, train/eval forward, aux/enc loss, and individual component validation Signed-off-by: josiah.smith <josiahsmithphd@gmail.com>
…andards Signed-off-by: josiah.smith <josiahsmithphd@gmail.com>
…ultidimensional tensor indexing in SetCriterion (ia_bce / varifocal) in lwdetr.py Signed-off-by: josiah.smith <josiahsmithphd@gmail.com>
- Fix PyTorch 2.9 deprecation: use tuple indexing instead of list for multidimensional tensor indexing in SetCriterion (ia_bce / varifocal) - Add end-to-end Lightning training-loop smoke tests for DETR, Deformable DETR, and RF-DETR (train + val + predict on CPU with timm_resnet18 backbone and synthetic data) Signed-off-by: josiah.smith <josiahsmithphd@gmail.com>
…acement in tests Supersedes torchgeo#1113 by josiahwsmith10 (add DETR, Deformable DETR, RF-DETR object detection models). Cannot push to original branch as it belongs to a different contributor's fork. The original PR conflicted with main on two files: - terratorch/models/object_detection_model_factory.py - terratorch/tasks/object_detection_task.py Conflicts were due to ruff/style formatting divergence (quote style, import ordering, line wrapping) — no semantic logic differences. Resolved by merging terrastackai:main into josiahwsmith10:add_DETR_models. Additional fix (tests/test_vit_adapter_modules.py): _make_vit_blocks() now accepts a device parameter and moves TimmBlocks to the correct device, preventing CPU/CUDA mismatch in 3 test cases.
ahmedemam576
force-pushed
the
fix/pr-1113-add-detr-models
branch
from
4db3678 to
566baf1
Compare
Collaborator
|
Collaborator
Author
|
Collaborator
Author
|
@romeokienzler I ran the unit tests locally and all of the tests are passed. some tests for deformable Detr are skipped because we need to install a PyTorch Wrapper for CUDA Functions of Multi-Scale Deformable Attention from the original Deformable Detr repo. I added the installation steps for this wrapper in the readme file |
Collaborator
Author
|
@romeokienzler all the integeration test are passing |
Signed-off-by: ahmedemam576 <ahmedemam576@gmail.com>
|
Hi there! 🦝 I noticed that this PR copies the model code directly into the repository instead of adding our package as a dependency. Could you share a bit more about why you prefer this approach? We've made a point to keep our dependencies trimmed down, so adding the package shouldn't bloat your environment. Using the maintained package would also save you from having to manually sync future bug fixes, optimizations, or new features. Just wanted to understand your use case and see if there's anything we can do on our end to make the package work better for TerraTorch. Thanks! |
Collaborator
Author
|
Hi @Borda Thanks for pointing this out and for the detailed explanation — really appreciate the note. |
- use upstream rfdetr modules in TerraTorchRFDETR - add optional extra and runtime import guidance - gate RF-DETR tests when dependency is absent - document install flow and add dedicated CI job
…dependency Replace the copied terratorch/models/detr/rfdetr/ directory with the official rfdetr PyPI package (rfdetr>=1.6.5.post0,<2), already listed as an optional dependency in pyproject.toml under [project.optional-dependencies]. Changes: - Delete entire terratorch/models/detr/rfdetr/ directory (14 files, ~1900 lines) - Update 7 imports in tests/test_detr.py TestRFDETRComponents to use rfdetr.* package paths instead of the deleted terratorch.models.detr.rfdetr.* paths - Fix test_rfdetr_position_encoding_deterministic to wrap tensor in NestedTensor - Update terratorch/models/detr/README.md to reflect RF-DETR is now a dependency All 47 tests pass (17 Deformable DETR tests skipped -- no CUDA extension). Why the same migration cannot be done for original DETR and Deformable DETR: Original DETR (facebookresearch/detr): the PyPI package (detr==0.1.4) is broken (bare top-level imports only work from the project root); repo archived 2024. Deformable DETR (fundamentalvision/Deformable-DETR): abandoned, no PyPI package, no releases. Kept as vendored code intentionally.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR supersedes #1113 by @josiahwsmith10.
The original PR branch could not be updated directly because it belongs to a different contributor fork. This PR preserves the intended changes and includes only minimal conflict resolution plus one targeted test fix.
What changed from #1113:
terratorch/models/object_detection_model_factory.pyandterratorch/tasks/object_detection_task.py(formatting/style divergence from main, with no intended behavior change).tests/test_vit_adapter_modules.py:_make_vit_blocks()now movesTimmBlocks to the target device to prevent CPU/CUDA mismatch in the affected tests.