[Refactor] Refactor anchor head and base head with boxlist #8625

jbwang1997 · 2022-08-23T14:04:00Z

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

Please describe the motivation of this PR and the goal you want to achieve through this PR.

Modification

Refactor anchor head and base head with boxlist.

BC-breaking (Optional)

Does the modification introduce changes that break the backward-compatibility of the downstream repos?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.

Use cases (Optional)

If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.

Checklist

Pre-commit or other linting tools are used to fix the potential lint issues.
The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
If the modification has potential influence on downstream projects, this PR should be tested with downstream projects, like MMDet or MMCls.
The documentation has been modified accordingly, like docstring or example tutorials.

BIGWangYuDong · 2022-08-24T01:49:46Z

mmdet/models/dense_heads/anchor_head.py

@@ -254,8 +257,10 @@ def _get_targets_single(self,
                                              gt_instances)

        num_valid_anchors = anchors.shape[0]
-        bbox_targets = torch.zeros_like(anchors)
-        bbox_weights = torch.zeros_like(anchors)
+        target_dim = gt_instances.bboxes.size(-1) if self.reg_decoded_bbox \


if anchors is BoxLIst here, can we use such as zeros_like here?
What's the difference between use or not use reg_decode_box?

The size of the last dimension of anchors and decoded targets may be different.

BIGWangYuDong · 2022-08-24T01:50:51Z

mmdet/models/dense_heads/anchor_head.py

@@ -271,6 +276,8 @@ def _get_targets_single(self,
                    sampling_result.pos_priors, sampling_result.pos_gt_bboxes)
            else:
                pos_bbox_targets = sampling_result.pos_gt_bboxes
+                if isinstance(pos_bbox_targets, BaseBoxes):


We should give users a document to tell them from where to where we will use boxlist.

mmdet/models/dense_heads/anchor_head.py

BIGWangYuDong · 2022-08-24T02:12:18Z

mmdet/models/detectors/base.py

@@ -149,4 +150,5 @@ def convert_to_datasample(self, results_list: InstanceList) -> SampleList:
            result = DetDataSample()
            result.pred_instances = results_list[i]
            out_results_list.append(result)
+        samplelist_boxlist2tensor(out_results_list)


Same question, we should check where converting boxlist to tensor is more reasonable

The boxlist results will be converted to tensor at the end of the model.

mmdet/models/task_modules/coders/bucketing_bbox_coder.py

mmdet/models/task_modules/coders/delta_xywh_bbox_coder.py

mmdet/models/task_modules/coders/bucketing_bbox_coder.py

mmdet/models/task_modules/coders/delta_xywh_bbox_coder.py

mmdet/structures/bbox/horizontal_boxes.py

zytx121 · 2022-08-31T01:35:59Z

retina_head.py should be refactored together.

mmdet/models/task_modules/coders/base_bbox_coder.py

RangiLyu · 2022-09-05T07:06:53Z

mmdet/models/task_modules/prior_generators/anchor_generator.py

@@ -68,7 +69,8 @@ def __init__(self,
                 octave_base_scale=None,
                 scales_per_octave=None,
                 centers=None,
-                 center_offset=0.):
+                 center_offset=0.,
+                 with_boxlist=False):


liuyanyi · 2022-09-05T08:22:34Z

Hi, when i'm test the rotated retinanet in MMRotate 1.x, i got an error here.

mmdetection/mmdet/models/task_modules/samplers/sampling_result.py

Lines 110 to 113 in f96d514

    
           if gt_bboxes.numel() == 0: 
        
               # hack for index error case 
        
               assert self.pos_assigned_gt_inds.numel() == 0 
        
               self.pos_gt_bboxes = torch.empty_like(gt_bboxes).view(-1, box_dim)

It happened when gt_bboxes is empty with RotatedBox structure, torch.empty_like doesn't support that input. Could you take a look at this, thanks.

zytx121

LGTM

mmdet/models/dense_heads/anchor_head.py

mmdet/models/task_modules/coders/base_bbox_coder.py

RangiLyu

LGTM

ZwwWayne · 2022-09-09T11:41:08Z

mmdet/models/task_modules/prior_generators/utils.py

-            (flat_anchors[:, 1] >= -allowed_border) & \
-            (flat_anchors[:, 2] < img_w + allowed_border) & \
-            (flat_anchors[:, 3] < img_h + allowed_border)
+        if isinstance(flat_anchors, BaseBoxes):


also implement a function like box_inside(flat_anchors, h, w, all_inside, allowed_border)

* Refactor anchor head * Update * Update * Update * Add a series of boxes tools * Fix box type to support n x box_dim boxes * revert box type changes * Add docstring * refactor retina_head * Update * Update * Fix comments * modify docstring of coder and ioucalculator * Replace with_boxlist with use_box_type

* [Fix] Fix UT to be compatible with pytorch 1.6 (#8707) * Update * Update * force reinstall pycocotools * Fix build_cuda * docker install git * Update * comment other job to speedup process * update * uncomment * Update * Update * Add comments for --force-reinstall * [Refactor] Refactor anchor head and base head with boxlist (#8625) * Refactor anchor head * Update * Update * Update * Add a series of boxes tools * Fix box type to support n x box_dim boxes * revert box type changes * Add docstring * refactor retina_head * Update * Update * Fix comments * modify docstring of coder and ioucalculator * Replace with_boxlist with use_box_type * fix: fix config of detr-r18 * fix: modified import of MSDeformAttn in PixelDecoder of Mask2Former * feat: add TransformerDetector as the base detector of DETR-like detectors * refactor: refactor modules and configs of DETR * refactor: refactor DETR-related modules in transformer.py * refactor: refactor DETR-related modules in transformer.py * fix: add type comments in detr.py * correct trainloop in detr_r50 config * fix: modify the parent class of DETRHead to BaseModule * refactor: refactor modules and configs of Deformable DETR * fix: modify the usage of num_query * fix: modify the usage of num_query in configs * refactor: replace input_proj of detr with ChannelMapper neck * refactor: delete multi_apply in DETRHead.forward() * Update detr_r18_8xb2-500e_coco.py using channel mapper for r18 * change the name of detection_transfomer.py to base_detr.py * refactor: modify construct binary masks section of forward_pretransformer * refactor: utilize abstractmethod * update ABCmeta to make sure reload class TransformerDetector * some annotation * some annotation * some annotation * refactor: delete _init_transformer in detectors * refactor: modify args of deformable detr * refactor: modify about super().__init__() * Update detr_head.py Remove the multi feat lvl in function 'predict_by_feat' * Update detr.py update init_weights * some annotation for head * to make sure the head args the same as detector * to make sure the head args the same as detector * some bug * fix: fix bugs of num_pred in DeformableDETRHead * add kwargs to transformer * support MLP and sineembed position * detele positional encodeing * delete useless postnorm * Revert "add kwargs to transformer" This reverts commit a265c1a. * Update detr_head.py Update type and shape of args * Update detr_head.py fix args docstring in predict_by_feat * Update base_detr.py Update docstring for forward_pretransformer * Update deformable_detr.py Fix docstring * to support conditional detr with reload forward_transformer * fix: update config files of Two-stage and Box-refine * replace all bs with batch_size in detr-related files * update deformable.py and transformer.py * update docstring in base_detr * update docstring in base_detr, detr * doc refine * Revert "doc refine" This reverts commit b69da4f. * doc refine * doc refine * updabase_detr, detr, and le layers/transformdoc * fix doc in base_detr * add origin repo link * add origin repo link * refine doc * refine doc * refine doc * refine doc * refine doc * refine doc * refine doc * refine doc * doc: add doc of the first edition of Deformable DETR * batch_size to bs * refine doc * refine doc * feat: add config comments of specific module * refactor: refactor base DETR class TransformerDetector * fix: fix wrong return typehint of forward_encoder in TransformerDetector * refactor: refactor DETR * refactor: refactor Deformable DETR * refactor: refactor forward_encoder and pre_decoder * fix: fix bugs of new edition * refactor: small modifications * fix: move get_reference_points to deformable_encoder * refactor: merge init_&inter_reference to references in Deformable DETR * modify docstring of get_valid_ratio in Deformable DETR * add some docstring * doc: add docstring of deformable_detr.py * doc: add docstring of deformable_detr_head.py * doc: modify docstring of deformable detr * doc: add docstring of deformable_detr_head.py * doc: modify docstring of deformable detr * doc: add docstring of base_detr.py * doc: refine docstring of base_detr.py * doc: refine docstring of base_detr.py * a little change of MLP * a little change of MLP * a little change of MLP * a little change of MLP * refine config * refine config * refine config * refine doc string for detr * little refine doc string for detr.py * tiny modification * doc: refine docstring of detr.py * tiny modifications to resolve the conversations * DETRHead.predict() draft * tiny modifications to resolve conversations * refactor: modify arg names and forward strategies of bbox_head * tiny modifications to resolve the conversations * support MLP * fix docsting of function pre_decoder * fix docsting of function pre_decoder * fix docstring * modifications for resolving conversations * refactor: eradicate key_padding_mask args * refactor: eradicate key_padding_mask args * fix: fix bug of deformable detr and resolve some conversations * refactor: rename base class with DetectionTransformer and other modifications * fix: fix config of detr * fix the bug of init * fix: fix init_weight of DETR and Deformable DETR * resolve conflict * fix auto-merge bug * fix pre-commit bug * refactor: move the position of encoder and decoder * delete Transformer in ci test * delete Transformer in ci test Co-authored-by: jbwang1997 <jbwang1997@gmail.com> Co-authored-by: KeiChiTse <xqz20@mails.tsinghua.edu.cn> Co-authored-by: LYMDLUT <70597027+LYMDLUT@users.noreply.github.com> Co-authored-by: lym <letusgo126@126.com> Co-authored-by: Kei-Chi Tse <109070650+KeiChiTse@users.noreply.github.com>

* [Fix] Fix UT to be compatible with pytorch 1.6 (open-mmlab#8707) * Update * Update * force reinstall pycocotools * Fix build_cuda * docker install git * Update * comment other job to speedup process * update * uncomment * Update * Update * Add comments for --force-reinstall * [Refactor] Refactor anchor head and base head with boxlist (open-mmlab#8625) * Refactor anchor head * Update * Update * Update * Add a series of boxes tools * Fix box type to support n x box_dim boxes * revert box type changes * Add docstring * refactor retina_head * Update * Update * Fix comments * modify docstring of coder and ioucalculator * Replace with_boxlist with use_box_type * fix: fix config of detr-r18 * fix: modified import of MSDeformAttn in PixelDecoder of Mask2Former * feat: add TransformerDetector as the base detector of DETR-like detectors * refactor: refactor modules and configs of DETR * refactor: refactor DETR-related modules in transformer.py * refactor: refactor DETR-related modules in transformer.py * fix: add type comments in detr.py * correct trainloop in detr_r50 config * fix: modify the parent class of DETRHead to BaseModule * refactor: refactor modules and configs of Deformable DETR * fix: modify the usage of num_query * fix: modify the usage of num_query in configs * refactor: replace input_proj of detr with ChannelMapper neck * refactor: delete multi_apply in DETRHead.forward() * Update detr_r18_8xb2-500e_coco.py using channel mapper for r18 * change the name of detection_transfomer.py to base_detr.py * refactor: modify construct binary masks section of forward_pretransformer * refactor: utilize abstractmethod * update ABCmeta to make sure reload class TransformerDetector * some annotation * some annotation * some annotation * refactor: delete _init_transformer in detectors * refactor: modify args of deformable detr * refactor: modify about super().__init__() * Update detr_head.py Remove the multi feat lvl in function 'predict_by_feat' * Update detr.py update init_weights * some annotation for head * to make sure the head args the same as detector * to make sure the head args the same as detector * some bug * fix: fix bugs of num_pred in DeformableDETRHead * add kwargs to transformer * support MLP and sineembed position * detele positional encodeing * delete useless postnorm * Revert "add kwargs to transformer" This reverts commit a265c1a. * Update detr_head.py Update type and shape of args * Update detr_head.py fix args docstring in predict_by_feat * Update base_detr.py Update docstring for forward_pretransformer * Update deformable_detr.py Fix docstring * to support conditional detr with reload forward_transformer * fix: update config files of Two-stage and Box-refine * replace all bs with batch_size in detr-related files * update deformable.py and transformer.py * update docstring in base_detr * update docstring in base_detr, detr * doc refine * Revert "doc refine" This reverts commit b69da4f. * doc refine * doc refine * updabase_detr, detr, and le layers/transformdoc * fix doc in base_detr * add origin repo link * add origin repo link * refine doc * refine doc * refine doc * refine doc * refine doc * refine doc * refine doc * refine doc * doc: add doc of the first edition of Deformable DETR * batch_size to bs * refine doc * refine doc * feat: add config comments of specific module * refactor: refactor base DETR class TransformerDetector * fix: fix wrong return typehint of forward_encoder in TransformerDetector * refactor: refactor DETR * refactor: refactor Deformable DETR * refactor: refactor forward_encoder and pre_decoder * fix: fix bugs of new edition * refactor: small modifications * fix: move get_reference_points to deformable_encoder * refactor: merge init_&inter_reference to references in Deformable DETR * modify docstring of get_valid_ratio in Deformable DETR * add some docstring * doc: add docstring of deformable_detr.py * doc: add docstring of deformable_detr_head.py * doc: modify docstring of deformable detr * doc: add docstring of deformable_detr_head.py * doc: modify docstring of deformable detr * doc: add docstring of base_detr.py * doc: refine docstring of base_detr.py * doc: refine docstring of base_detr.py * a little change of MLP * a little change of MLP * a little change of MLP * a little change of MLP * refine config * refine config * refine config * refine doc string for detr * little refine doc string for detr.py * tiny modification * doc: refine docstring of detr.py * tiny modifications to resolve the conversations * DETRHead.predict() draft * tiny modifications to resolve conversations * refactor: modify arg names and forward strategies of bbox_head * tiny modifications to resolve the conversations * support MLP * fix docsting of function pre_decoder * fix docsting of function pre_decoder * fix docstring * modifications for resolving conversations * refactor: eradicate key_padding_mask args * refactor: eradicate key_padding_mask args * fix: fix bug of deformable detr and resolve some conversations * refactor: rename base class with DetectionTransformer and other modifications * fix: fix config of detr * fix the bug of init * fix: fix init_weight of DETR and Deformable DETR * resolve conflict * fix auto-merge bug * fix pre-commit bug * refactor: move the position of encoder and decoder * delete Transformer in ci test * delete Transformer in ci test Co-authored-by: jbwang1997 <jbwang1997@gmail.com> Co-authored-by: KeiChiTse <xqz20@mails.tsinghua.edu.cn> Co-authored-by: LYMDLUT <70597027+LYMDLUT@users.noreply.github.com> Co-authored-by: lym <letusgo126@126.com> Co-authored-by: Kei-Chi Tse <109070650+KeiChiTse@users.noreply.github.com>

mm-assistant bot assigned hhaAndroid Aug 23, 2022

jbwang1997 added the v-3.x label Aug 23, 2022

jbwang1997 requested a review from BIGWangYuDong August 23, 2022 14:04

BIGWangYuDong reviewed Aug 24, 2022

View reviewed changes

jbwang1997 force-pushed the anchor_head branch from 0b775fc to ad9d351 Compare August 25, 2022 08:03

zytx121 previously approved these changes Aug 25, 2022

View reviewed changes

jbwang1997 dismissed zytx121’s stale review via 587ae2c August 26, 2022 02:05

jbwang1997 force-pushed the anchor_head branch 3 times, most recently from f829736 to d160f64 Compare August 30, 2022 15:15

jbwang1997 force-pushed the anchor_head branch from d160f64 to eecaee7 Compare September 1, 2022 08:13

ZwwWayne added this to the 3.0.0rc1 milestone Sep 5, 2022

jbwang1997 mentioned this pull request Sep 5, 2022

[Refactor] Refactor standard roi head with boxlist. #8658

Merged

jbwang1997 added 11 commits September 5, 2022 14:17

Refactor anchor head

ed6b8a6

Update

5002890

Update

7803436

Update

6e29790

Add a series of boxes tools

c83d73d

Fix box type to support n x box_dim boxes

87d9626

revert box type changes

98cc8e9

Add docstring

b443e92

refactor retina_head

cbe9077

Update

ff8e0b3

Update

f96d514

jbwang1997 force-pushed the anchor_head branch from dfc98fd to f96d514 Compare September 5, 2022 06:19

ZwwWayne requested review from BIGWangYuDong and zytx121 September 5, 2022 06:24

RangiLyu reviewed Sep 5, 2022

View reviewed changes

mmdet/models/task_modules/coders/base_bbox_coder.py Outdated Show resolved Hide resolved

RangiLyu reviewed Sep 5, 2022

View reviewed changes

mmdet/models/task_modules/coders/base_bbox_coder.py Show resolved Hide resolved

RangiLyu reviewed Sep 5, 2022

View reviewed changes

mmdet/models/task_modules/coders/base_bbox_coder.py Outdated Show resolved Hide resolved

RangiLyu reviewed Sep 5, 2022

View reviewed changes

Fix comments

87fc8f7

zytx121 mentioned this pull request Sep 6, 2022

[Refactor] refactor RetinaNet based on boxlist open-mmlab/mmrotate#472

Merged

3 tasks

modify docstring of coder and ioucalculator

8a67365

zytx121 previously approved these changes Sep 8, 2022

View reviewed changes

RangiLyu reviewed Sep 8, 2022

View reviewed changes

mmdet/models/dense_heads/anchor_head.py Show resolved Hide resolved

RangiLyu reviewed Sep 8, 2022

View reviewed changes

mmdet/models/task_modules/coders/base_bbox_coder.py Outdated Show resolved Hide resolved

Replace with_boxlist with use_box_type

532a1ec

jbwang1997 dismissed zytx121’s stale review via 532a1ec September 8, 2022 14:16

RangiLyu approved these changes Sep 9, 2022

View reviewed changes

ZwwWayne reviewed Sep 9, 2022

View reviewed changes

ZwwWayne merged commit c9b4782 into open-mmlab:dev-3.x Sep 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Refactor] Refactor anchor head and base head with boxlist #8625

[Refactor] Refactor anchor head and base head with boxlist #8625

jbwang1997 commented Aug 23, 2022

BIGWangYuDong Aug 24, 2022

jbwang1997 Aug 25, 2022

BIGWangYuDong Aug 24, 2022

BIGWangYuDong Aug 24, 2022

jbwang1997 Aug 25, 2022

zytx121 commented Aug 31, 2022

RangiLyu Sep 5, 2022

liuyanyi commented Sep 5, 2022

zytx121 left a comment

RangiLyu left a comment

ZwwWayne Sep 9, 2022

[Refactor] Refactor anchor head and base head with boxlist #8625

[Refactor] Refactor anchor head and base head with boxlist #8625

Conversation

jbwang1997 commented Aug 23, 2022

Motivation

Modification

BC-breaking (Optional)

Use cases (Optional)

Checklist

BIGWangYuDong Aug 24, 2022

Choose a reason for hiding this comment

jbwang1997 Aug 25, 2022

Choose a reason for hiding this comment

BIGWangYuDong Aug 24, 2022

Choose a reason for hiding this comment

BIGWangYuDong Aug 24, 2022

Choose a reason for hiding this comment

jbwang1997 Aug 25, 2022

Choose a reason for hiding this comment

zytx121 commented Aug 31, 2022

RangiLyu Sep 5, 2022

Choose a reason for hiding this comment

liuyanyi commented Sep 5, 2022

zytx121 left a comment

Choose a reason for hiding this comment

RangiLyu left a comment

Choose a reason for hiding this comment

ZwwWayne Sep 9, 2022

Choose a reason for hiding this comment