MLP for embedding camera matrix

[rfeinman]

Hi @seasonSH, could you provide some insight about the architecture of the 2-layer MLP for the camera matrix embedding? I'm guessing that it looks something like the below code, but if you could please correct me that would be great.

import torch
import torch.nn as nn

# 2-layer MLP
camera_embedding = nn.Sequential(
    nn.Linear(4*4, 512), 
    nn.SiLU(),
    nn.Linear(512, time_embed_dim),
    nn.LayerNorm(time_embed_dim)
)

# embed the 4x4 camera matrix
camera_embed = camera_embedding(camera)

# add result to time embedding
time_embed = time_embed + camera_embed

[yashkant]

+1 for this question!

[seasonSH]

It has exactly the same architecture as timestep embedding MLP except the input channels