FineTune on Custom instance coco dataset

[mathisTH]

Hello ! Thanks for your work

I'm facing two issue right now :(

I'm trying to finetune OneFormer (swin tiny) on my custom coco instance dataset using the template you provide but i'm facing a weird behavior when a batch is full of "background" images (without annotations) Dice and Mask loss are 0 because of the Hungarian Matching returning empty index list. So i'm wondering if it's a normal behavior and if i should continue the training as is?
I had to reduce the batch size to two so it might be a part of the answer?

On an other hand when i try to train OneFormer with SwinL i'm facing inf value in the task_mlp output tensor. Do you have an idea why? I've tried to add input regularization for the tasks tensor (cf oneformer_model forward) seems to fixed it but i've reached the same point with the Mask/Dice loss 0 value so i've stop further investigation

Do you have any suggestions?

Again thanks for your work !