🤗 Hub and TensorFlowTTS integration for Inference API

[osanseviero]

Hi TensorSpeech team! I propose building on top of the existing integration with 🤗 Hub to enable the Inference API and widgets

Current Status

• With Proposal to integrate into 🤗 Hub #555, users can easily download models from the Hub.
• With Add TensorFlowTTS as library + code snippet huggingface/huggingface_hub#55, TensorFlowTTS is now a searchable library in the Hub.

• With the same PR, users now have access to a code snippet that shows how to load the model. You can use text-to-mel and mel-to-wav so the code snippet is different.

What can we do next?
Our next step is to integrate it to the Inference API. This would make the widget in the model repos work and allow anyone to make requests to the API, which is very exciting!

The main challenge is that TTS is a two-step process: converting text to mel, and mel to speech. At the moment, every single repo only has one of the two components, so there is no way to make inference work end-to-end.

What do you think of bundling the two models? That is, in a single repo we would have Fastspeech2 and Melgan for example. We would only do this for the repos for which the Inference API is wanted, so it wouldn't be for all of them.

Happy to hear your thoughts,
Omar

cc @patrickvonplaten @julien-c

[julien-c]

on my (end user) side, bundling models into an easy-to-use end to end TTS "model" (for at least a few of your repos) would be awesome!

We already have a TTS widget so we could showcase your models super quickly.

[dathudeptrai]

@julien-c @osanseviero Great!. So I need to write an End-2-End Model class that combines 2 models into 1 model ? or just need to put all of them into 1 repo? :D

[osanseviero]

I think that ideally we would have a single class in a way that we could do something like this:

from tensorflow_tts.inference import AutoProcessor, TFE2EAutoModel

processor = AutoProcessor.from_pretrained(REPO_ID)
model = TFE2EAutoModel.from_pretrained(REPO_ID)
audios = model.inference(mels)

The corresponding repo would contain both the text to mel and mel to speech models, but it would be a nice improvement in my opinion. What do you think about it? It would also simplify usage considerably for other users

[dathudeptrai]

Yeah, that would be the best choice :D. Will do that in this week :D. @osanseviero

[osanseviero]

Hi @dathudeptrai.

Thanks for all the great work around TensorFlowTTS 😄 I was wondering if you got a chance to work on this.

Cheers!

[dathudeptrai]

Hi @dathudeptrai.

Thanks for all the great work around TensorFlowTTS I was wondering if you got a chance to work on this.

Cheers!

:D. I'm struggling with some other stuff, I will let you know when I finish the implementation :D. Hope it can finish this week :D.

[dathudeptrai]

@osanseviero Hi, sorry for the late implementation, I just won 1st Kaggle NLP competition and now I have more free time to do this feature :D. I want to share my solution here in case you interested in it (https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/248253), and ofc, i used HuggingFace Transformer :D.

[osanseviero]

This is awesome @dathudeptrai! Congratulations!!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.