add MaskRCNN #781

yuyu2172 · 2019-02-12T12:30:34Z

~~Merge after #778~~

…ask-rcnn

yuyu2172 · 2019-02-14T03:19:33Z

Wierd segfault during training...
Experiemented with n_gpu=1, batchsize=1.
For debugging purpose, I changed MultithreadIterator to SerialIterator.

 *** Process received signal ***........]  1.42%                                                                        
 Signal: Segmentation fault (11)........] 17.32%                                                                        
 Signal code: Address not mapped (1)                                                                                    
 Failing at address: 0x67ee6000s, 21:36:26.539275.                                                                      
 [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7fbb433f5390]                                                   
 [ 1] /usr/local/lib/python3.6/site-packages/cv2.cpython-36m-x86_64-linux-gnu.so(+0x7971a0)[0x7fb9d82c31a0]             
 [ 2] /usr/local/lib/python3.6/site-packages/cv2.cpython-36m-x86_64-linux-gnu.so(+0x61ce58)[0x7fb9d8148e58]             
 [ 3] /usr/local/lib/python3.6/site-packages/cv2.cpython-36m-x86_64-linux-gnu.so(+0x7cf482)[0x7fb9d82fb482]             
 [ 4] /usr/local/lib/python3.6/site-packages/cv2.cpython-36m-x86_64-linux-gnu.so(+0x7d0879)[0x7fb9d82fc879]             
 [ 5] /usr/local/lib/python3.6/site-packages/cv2.cpython-36m-x86_64-linux-gnu.so(+0x2ef784)[0x7fb9d7e1b784]             
 [ 6] /usr/local/lib/libpython3.6m.so.1.0(_PyCFunction_FastCallDict+0x13c)[0x7fbb43733d6c]                              
 [ 7] /usr/local/lib/libpython3.6m.so.1.0(_PyCFunction_FastCallKeywords+0x4d)[0x7fbb43733bfd]                           
 [ 8] /usr/local/lib/libpython3.6m.so.1.0(+0x18b9fb)[0x7fbb4378c9fb]                                                    
 [ 9] /usr/local/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x1363)[0x7fbb43785a83]                              
 [10] /usr/local/lib/libpython3.6m.so.1.0(+0x18be0a)[0x7fbb4378ce0a]                                                    
 [11] /usr/local/lib/libpython3.6m.so.1.0(+0x18badb)[0x7fbb4378cadb]                                                    
 [12] /usr/local/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x34f)[0x7fbb43784a6f]                               
 [13] /usr/local/lib/libpython3.6m.so.1.0(+0x18c0f3)[0x7fbb4378d0f3]                                                    
 [14] /usr/local/lib/libpython3.6m.so.1.0(+0x18badb)[0x7fbb4378cadb]                                                    
 [15] /usr/local/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x1363)[0x7fbb43785a83]                              
 [16] /usr/local/lib/libpython3.6m.so.1.0(+0x18be0a)[0x7fbb4378ce0a]                                                    
 [17] /usr/local/lib/libpython3.6m.so.1.0(+0x18badb)[0x7fbb4378cadb]                                                    
 [18] /usr/local/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x34f)[0x7fbb43784a6f]                               
 [19] /usr/local/lib/libpython3.6m.so.1.0(_PyFunction_FastCallDict+0x41b)[0x7fbb4378eacb]                               
 [20] /usr/local/lib/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x10e)[0x7fbb436fa86e]                                 
 [21] /usr/local/lib/libpython3.6m.so.1.0(_PyObject_Call_Prepend+0x61)[0x7fbb436fb1f1]                                  
 [22] /usr/local/lib/libpython3.6m.so.1.0(PyObject_Call+0x47)[0x7fbb436fac17]                                           
 [23] /usr/local/lib/libpython3.6m.so.1.0(+0x147506)[0x7fbb43748506]                                                    
 [24] /usr/local/lib/libpython3.6m.so.1.0(PyObject_Call+0x47)[0x7fbb436fac17]                                           
 [25] /usr/local/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x1c69)[0x7fbb43786389]                              
 [26] /usr/local/lib/libpython3.6m.so.1.0(_PyFunction_FastCallDict+0x41b)[0x7fbb4378eacb]                               
 [27] /usr/local/lib/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x10e)[0x7fbb436fa86e]                                 
 [28] /usr/local/lib/libpython3.6m.so.1.0(_PyObject_Call_Prepend+0x61)[0x7fbb436fb1f1]                                  
 [29] /usr/local/lib/libpython3.6m.so.1.0(PyObject_Call+0x47)[0x7fbb436fac17]                                           
 *** End of error message ***

other configs

failed
- PIL for read_image
- PIL for resize
- Serial Iterator
failed
- PIL for read_image
- PIL for resize
- Serial Iterator
- Use the latest RoiAlign

yuyu2172 · 2019-02-15T08:03:22Z

$ mpiexec -n 8 python3 train_multi.py --batchsize 8
--------------------------------------------------------------------------
WARNING: There are more than one active ports on host 'coco1-worker-0', but the
default subnet GID prefix was detected on more than one of these
ports.  If these ports are connected to different physical IB
networks, this configuration will fail in Open MPI.  This version of
Open MPI requires that every physically separate IB subnet that is
used between connected MPI processes must have different subnet ID
values.

Please see this FAQ entry for more details:

  http://www.open-mpi.org/faq/?category=openfabrics#ofa-default-subnet-gid

NOTE: You can turn off this warning by setting the MCA parameter
      btl_openib_warn_default_gid_prefix to 0.
--------------------------------------------------------------------------
[coco1-worker-0:115997] 7 more processes have sent help message help-mpi-btl-openib.txt / default subnet prefix
[coco1-worker-0:115997] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
0 (1, 3, 1344, 576)
6 (1, 3, 800, 1216)
4 (1, 3, 1088, 800)
7 (1, 3, 800, 1216)
15 (1, 3, 800, 1088)
19 (1, 3, 1088, 800)
19 (1, 3, 704, 1344)
41 (1, 3, 800, 1088)
[coco1-worker-0:116005] Failed to cuMemcpy GPU memory, rc=-1
--------------------------------------------------------------------------
The call to cuMemcpyAsync failed. This is a unrecoverable error and will
cause the program to abort.
  cuMemcpyAsync(0x7f5f2f22ac00, 0x7f5df4dfb000, 8192) returned value 1
Check the cuda.h file for what the return value means.
--------------------------------------------------------------------------
Exception in main training loop: MPI_ERR_TRUNCATE: message truncated
Traceback (most recent call last):
  File "/home/yuyu2172/chainer/chainer/training/trainer.py", line 316, in run
    update()
  File "/home/yuyu2172/chainer/chainer/training/updaters/standard_updater.py", line 170, in update
    self.update_core()
  File "/home/yuyu2172/chainer/chainer/training/updaters/standard_updater.py", line 182, in update_core
    optimizer.update(loss_func, *in_arrays)
  File "/home/yuyu2172/chainer/chainermn/optimizers.py", line 28, in update
    self.communicator.bcast_data(target)
  File "/home/yuyu2172/chainer/chainermn/communicators/mpi_communicator_base.py", line 615, in bcast_data
    self.mpi_comm.Bcast(buf)
  File "mpi4py/MPI/Comm.pyx", line 579, in mpi4py.MPI.Comm.Bcast
Will finalize trainer extensions and updater before reraising the exception.
Traceback (most recent call last):
  File "train_multi.py", line 249, in <module>
    main()
  File "train_multi.py", line 245, in main
    trainer.run()
  File "/home/yuyu2172/chainer/chainer/training/trainer.py", line 349, in run
    six.reraise(*exc_info)
  File "/usr/local/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/home/yuyu2172/chainer/chainer/training/trainer.py", line 316, in run
    update()
  File "/home/yuyu2172/chainer/chainer/training/updaters/standard_updater.py", line 170, in update
    self.update_core()
  File "/home/yuyu2172/chainer/chainer/training/updaters/standard_updater.py", line 182, in update_core
    optimizer.update(loss_func, *in_arrays)
  File "/home/yuyu2172/chainer/chainermn/optimizers.py", line 28, in update
    self.communicator.bcast_data(target)
  File "/home/yuyu2172/chainer/chainermn/communicators/mpi_communicator_base.py", line 615, in bcast_data
    self.mpi_comm.Bcast(buf)
  File "mpi4py/MPI/Comm.pyx", line 579, in mpi4py.MPI.Comm.Bcast
mpi4py.MPI.Exception: MPI_ERR_TRUNCATE: message truncated

yuyu2172 · 2019-02-15T12:24:32Z

Seemingly working with the current master
Chainer: chainer/chainer@afe9033
CuPy: cupy/cupy@155228f

EDIT:

*** Process received signal ***........] 48.41%
Signal: Segmentation fault (11)##......] 89.34%
Signal code: Address not mapped (1)
Failing at address: 0x6af2d000, 9:44:55.033078.
[ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7fc50f859390]
[ 1] /usr/local/lib/python3.6/site-packages/cv2.cpython-36m-x86_64-linux-gnu.so(+0x7971a0)[0x7fc422b991a0]
[ 2] /usr/local/lib/python3.6/site-packages/cv2.cpython-36m-x86_64-linux-gnu.so(+0x61ce58)[0x7fc422a1ee58]
[ 3] /usr/local/lib/python3.6/site-packages/cv2.cpython-36m-x86_64-linux-gnu.so(+0x7cf482)[0x7fc422bd1482]
[ 4] /usr/local/lib/python3.6/site-packages/cv2.cpython-36m-x86_64-linux-gnu.so(+0x7d0879)[0x7fc422bd2879]
[ 5] /usr/local/lib/python3.6/site-packages/cv2.cpython-36m-x86_64-linux-gnu.so(+0x2ef784)[0x7fc4226f1784]
[ 6] /usr/local/lib/libpython3.6m.so.1.0(_PyCFunction_FastCallDict+0x13c)[0x7fc50fb97d6c]
[ 7] /usr/local/lib/libpython3.6m.so.1.0(_PyCFunction_FastCallKeywords+0x4d)[0x7fc50fb97bfd]
[ 8] /usr/local/lib/libpython3.6m.so.1.0(+0x18b9fb)[0x7fc50fbf09fb]
[ 9] /usr/local/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x1363)[0x7fc50fbe9a83]
[10] /usr/local/lib/libpython3.6m.so.1.0(+0x18be0a)[0x7fc50fbf0e0a]
[11] /usr/local/lib/libpython3.6m.so.1.0(+0x18badb)[0x7fc50fbf0adb]
[12] /usr/local/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x34f)[0x7fc50fbe8a6f]
[13] /usr/local/lib/libpython3.6m.so.1.0(+0x18c0f3)[0x7fc50fbf10f3]
[14] /usr/local/lib/libpython3.6m.so.1.0(+0x18badb)[0x7fc50fbf0adb]
[15] /usr/local/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x1363)[0x7fc50fbe9a83]
[16] /usr/local/lib/libpython3.6m.so.1.0(+0x18be0a)[0x7fc50fbf0e0a]
[17] /usr/local/lib/libpython3.6m.so.1.0(+0x18badb)[0x7fc50fbf0adb]
[18] /usr/local/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x34f)[0x7fc50fbe8a6f]
[19] /usr/local/lib/libpython3.6m.so.1.0(_PyFunction_FastCallDict+0x41b)[0x7fc50fbf2acb]
[20] /usr/local/lib/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x10e)[0x7fc50fb5e86e]
[21] /usr/local/lib/libpython3.6m.so.1.0(_PyObject_Call_Prepend+0x61)[0x7fc50fb5f1f1]
[22] /usr/local/lib/libpython3.6m.so.1.0(PyObject_Call+0x47)[0x7fc50fb5ec17]
[23] /usr/local/lib/libpython3.6m.so.1.0(+0x147506)[0x7fc50fbac506]
[24] /usr/local/lib/libpython3.6m.so.1.0(PyObject_Call+0x47)[0x7fc50fb5ec17]
[25] /usr/local/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x1c69)[0x7fc50fbea389]
[26] /usr/local/lib/libpython3.6m.so.1.0(_PyFunction_FastCallDict+0x41b)[0x7fc50fbf2acb]
[27] /usr/local/lib/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x10e)[0x7fc50fb5e86e]
[28] /usr/local/lib/libpython3.6m.so.1.0(_PyObject_Call_Prepend+0x61)[0x7fc50fb5f1f1]
[29] /usr/local/lib/libpython3.6m.so.1.0(PyObject_Call+0x47)[0x7fc50fb5ec17]
*** End of error message ***

EDIT:
Segfault problem seems to have stopped after #798

Hakuyume · 2019-04-11T07:57:18Z

chainercv/links/model/fpn/mask_head.py

+    _canonical_scale = 224
+    _roi_size = 14
+    _roi_sample_ratio = 2
+    segm_size = _roi_size * 2


Why is this member public?

This is necessary for mask_head_loss_pre.
To circumvent this problem, should we make mask_head_loss_pre as a method of MaskHead?

I see. It is ok to make this member public.

Hakuyume · 2019-04-11T08:00:47Z

chainercv/links/model/fpn/faster_rcnn.py

+
+            :obj:`imgs`, ":math:`[(3, H, W)]`", :obj:`float32`, \
+            "RGB, :math:`[0, 255]`"
+            :obj:`rois`, ":math:`[(R', 4)]`", :obj:`float32`, \


Can we add a line between inputs part and outputs part? (or just make two tables)

Hakuyume · 2019-04-16T05:33:38Z

chainercv/links/model/fpn/mask_utils.py

+    # pad with repeated border values), we manually zero-pad the masks by 1
+    # pixel prior to resizing back to the original image resolution.
+    # This prevents "top hat" artifacts. We therefore need to expand
+    # the reference boxes by an appropriate factor.


We have two similar blocks. Can we merge them?

yuyu2172 · 2019-04-19T08:53:43Z

Define min_size and max_size in train_multi.py.
use only cv2 backend in mask_utils.py
- Change the behavior of resize. Make it break when chainer.config.cv_resize_backend == 'cv2' and cv2 is not installed.
- Force cv2 backend in mask_utils.py

yuyu2172 · 2019-04-19T11:55:05Z

pfnCI, test this please

pfn-ci-bot · 2019-04-19T11:55:12Z

Successfully created a job for commit 6acc48f (2975879):

Dashboard for commit 6acc48f

Hakuyume · 2019-04-19T14:01:11Z

pfnCI, test this please

pfn-ci-bot · 2019-04-19T14:01:20Z

Successfully created a job for commit 6acc48f (2975879):

Dashboard for commit 6acc48f

yuyu2172 · 2019-04-22T04:08:37Z

pfnCI, test this please

pfn-ci-bot · 2019-04-22T04:08:45Z

Successfully created a job for commit 58ef6c7 (15c5cdc):

Dashboard for commit 58ef6c7

yuyu2172 · 2019-04-23T00:54:21Z

pfnCI, test this please

pfn-ci-bot · 2019-04-23T00:54:27Z

Successfully created a job for commit 6c4dc80 (9fad50d):

Dashboard for commit 6c4dc80

yuyu2172 force-pushed the mask-rcnn branch from 49a2915 to 516d23d Compare February 12, 2019 12:40

[wip] add mask_rcnn

b3e8ac0

yuyu2172 force-pushed the mask-rcnn branch from 516d23d to b3e8ac0 Compare February 12, 2019 13:00

yuyu2172 added 2 commits February 12, 2019 22:01

Merge remote-tracking branch 'yuyu2172/coco-instances-dataset' into m…

31ed235

…ask-rcnn

misc

bcd68fa

knorth55 added this to the 0.13 milestone Feb 12, 2019

knorth55 added the feature label Feb 12, 2019

yuyu2172 added 3 commits February 13, 2019 08:56

Merge remote-tracking branch 'origin/master' into mask-rcnn

64cc16c

doc

f1d4e46

remove unnecessary print

1e10d32

yuyu2172 force-pushed the mask-rcnn branch 4 times, most recently from 1bd2c8a to f402118 Compare February 13, 2019 07:37

misc

e468545

yuyu2172 force-pushed the mask-rcnn branch 4 times, most recently from bceb057 to 6b48d9a Compare February 13, 2019 14:07

misc

75fb3ff

yuyu2172 force-pushed the mask-rcnn branch from 6b48d9a to 75fb3ff Compare February 13, 2019 14:09

handle the case when #RoI is 0

43e8acc

yuyu2172 added 5 commits February 18, 2019 10:18

delete mask option for MaskRCNN.prepare

a4319e0

use MultiprocessIterator

496dd93

train now works

a8fcf2a

some speed up

0d9ba65

reduce copy

97497ea

Hakuyume reviewed Apr 11, 2019

View reviewed changes

Hakuyume reviewed Apr 16, 2019

View reviewed changes

reflect comments

c02350d

yuyu2172 added 5 commits April 19, 2019 19:57

force cv2 in mask_utils

dd7f9fa

change config.pbtxt

9c6f944

do not set min_size and max_size as public

f37010e

merge

8226baf

remove data-dir argument

ba5dee1

yuyu2172 mentioned this pull request Apr 19, 2019

Set backends explicitly for read_image and rotate #852

Merged

yuyu2172 added the no-compat label Apr 19, 2019

fix test

27c5236

yuyu2172 force-pushed the mask-rcnn branch from 6931696 to 27c5236 Compare April 19, 2019 11:31

yuyu2172 added 2 commits April 19, 2019 20:49

Merge remote-tracking branch 'origin/master' into mask-rcnn

3a04c63

timeout to 3600

6acc48f

yuyu2172 added 2 commits April 20, 2019 07:36

fix flake

4fbcb06

fix bug

58ef6c7

yuyu2172 force-pushed the mask-rcnn branch from 512e4f0 to 58ef6c7 Compare April 20, 2019 03:34

Merge remote-tracking branch 'origin/master' into HEAD

6c4dc80

Hakuyume approved these changes Apr 23, 2019

View reviewed changes

Hakuyume merged commit 938c86f into chainer:master Apr 23, 2019

yuyu2172 deleted the mask-rcnn branch May 14, 2019 14:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add MaskRCNN #781

add MaskRCNN #781

yuyu2172 commented Feb 12, 2019 •

edited

Loading

yuyu2172 commented Feb 14, 2019 •

edited

Loading

yuyu2172 commented Feb 15, 2019

yuyu2172 commented Feb 15, 2019 •

edited

Loading

Hakuyume Apr 11, 2019

yuyu2172 Apr 18, 2019 •

edited

Loading

Hakuyume Apr 19, 2019

Hakuyume Apr 11, 2019

yuyu2172 Apr 18, 2019

Hakuyume Apr 16, 2019

yuyu2172 Apr 18, 2019

yuyu2172 commented Apr 19, 2019

yuyu2172 commented Apr 19, 2019

pfn-ci-bot commented Apr 19, 2019

Hakuyume commented Apr 19, 2019

pfn-ci-bot commented Apr 19, 2019

yuyu2172 commented Apr 22, 2019

pfn-ci-bot commented Apr 22, 2019

yuyu2172 commented Apr 23, 2019

pfn-ci-bot commented Apr 23, 2019

add MaskRCNN #781

add MaskRCNN #781

Conversation

yuyu2172 commented Feb 12, 2019 • edited Loading

yuyu2172 commented Feb 14, 2019 • edited Loading

other configs

yuyu2172 commented Feb 15, 2019

yuyu2172 commented Feb 15, 2019 • edited Loading

Hakuyume Apr 11, 2019

Choose a reason for hiding this comment

yuyu2172 Apr 18, 2019 • edited Loading

Choose a reason for hiding this comment

Hakuyume Apr 19, 2019

Choose a reason for hiding this comment

Hakuyume Apr 11, 2019

Choose a reason for hiding this comment

yuyu2172 Apr 18, 2019

Choose a reason for hiding this comment

Hakuyume Apr 16, 2019

Choose a reason for hiding this comment

yuyu2172 Apr 18, 2019

Choose a reason for hiding this comment

yuyu2172 commented Apr 19, 2019

yuyu2172 commented Apr 19, 2019

pfn-ci-bot commented Apr 19, 2019

Hakuyume commented Apr 19, 2019

pfn-ci-bot commented Apr 19, 2019

yuyu2172 commented Apr 22, 2019

pfn-ci-bot commented Apr 22, 2019

yuyu2172 commented Apr 23, 2019

pfn-ci-bot commented Apr 23, 2019

yuyu2172 commented Feb 12, 2019 •

edited

Loading

yuyu2172 commented Feb 14, 2019 •

edited

Loading

yuyu2172 commented Feb 15, 2019 •

edited

Loading

yuyu2172 Apr 18, 2019 •

edited

Loading