2D to 3D attention training question

[andrewbutts]

Hiya, nice work! I just had a clarifying question:

When replicating the existing model’s 2D attention for 3D, do you freeze the portion of the 3D attention that relates a particular camera’s view to itself? Or do you let the training process affect it as well? If so, what’s the rationale and what effect does that have compared to leaving that part frozen?

Thanks! I hope my question makes sense!

[seasonSH]

Nothing is frozen in our model during training. We fine-tuned the whole model.
In fact, we tried to freeze some parts (e.g. conv and crossattn) in our experiments but saw no clear improvement. Therefore, we just fine-tune the whole model for simplicity.

[Friedrich-M]

I was wondering whether you directly finetune the whole unet parameters or use the Lora.

[seasonSH]

whole unet