Release Multi-HMR on Hugging Face

[NielsRogge]

Hi @fabienbaradel,

I'm Niels and work as part of the open-source team at Hugging Face 🤗 (we met at ECCV!). Congrats on your work, I indexed the paper page here: https://huggingface.co/papers/2402.14654. The paper page lets people discuss about your paper and lets them find artifacts about it (your model for instance) you can also claim the paper as yours which will show up on your public profile at HF.

Would you like to host the model you've pre-trained on https://huggingface.co/models, rather than internal hosting? Hosting on Hugging Face will give you more visibility/improve discoverability and is free. We can add tags in the model cards so that people find the models easier, link it to the paper page, etc.

Everything could ideally be published at https://huggingface.co/naver.

Uploading models

If you're down, leaving a guide here. If it's a PyTorch model, you can use PyTorchModelHubMixin class which adds from_pretrained and push_to_hub to the model which lets you to upload the model and people to download and use models right away.
If you do not want this and directly want to upload model through UI or however you want, people can also use hf_hub_download.

After uploaded, we can also link the models to the paper page (read here) so people can discover your model.

You can also build a demo to your model on Spaces we can provide you an A100 grant.

What do you think?

Kind regards,

Niels

[fabienbaradel]

Hi @NielsRogge
Thanks for your help, I added model/dataset/space to the paper page:

• The demo space is running.
• For the dataset I have added the .zip file which contains the entire data+annots but not sure I should do that, is it okay ?
• Finally for the model it is empty at the moment, I will try to add them this week.
  Thanks for your help,

[NielsRogge]

Hi @fabienbaradel,

Was nice chatting at ECCV, awesome release, https://huggingface.co/papers/2402.14654 looks awesome now 👌

Regarding the dataset, if you unzip it and if it's one of these formats supported here, it could be made compatible with the Datasets library. Besides that, we support images, see this guide for going from a local dir of images + annotations to the hub.

Will ping AK for amplifying this on Twitter (perhaps once the model is up).

[NielsRogge]

Hi @fabienbaradel following up on this, would it be possible to upload the models and link them together with the Space in the README of this repo?

[fabienbaradel]

Hi @NielsRogge thank you for following on this. I added links in the README to the SPACE and to one checkpoint, please see my two last commits (59533c0 and e7cdcd0).
Does it work for you ?
If that's the case I will upload other checkpoints.
Thanks for your help,

[NielsRogge]

Hi,

Thank you! There's actually a better alternative here, namely the PyTorchModelHubMixin.

I see the Multi-HMR model could inherit from it, like so:

from huggingface_hub import PyTorchModelHubMixin

class Model(nn.Module, PyTorchModelHubMixin, repo_url="https://github.com/naver/multi-hmr", pipeline_tag="image-to-3d", license="mit")

This way, you can do the following:

from .model import Model

model = Model(...)

# push to hub
model.push_to_hub("naver/multi-hmr")

# reload
model = Model.from_pretrained("naver/multi-hmr")

This will automatically push a config.json along with weights in a safetensors file to the hub, and you can reload from it. It will also ensure downloads work. This was also adopted by Dust3r and Mast3r.

Let me know if I need to submit a PR!

Also, we'd encourage to avoid pushing weights to a Space: https://huggingface.co/spaces/naver/multi-hmr/tree/main/models/multiHMR, and instead push them to dedicated model repositories, which can then be linked to the Space which uses it using from_pretrained.