- This project is implemented in Pytorch (>1.8). Thus please install Pytorch first.
- ctcdecode==0.4 [parlance/ctcdecode]，for beam search decode.
- For these who failed install ctcdecode (and it always does), you can download ctcdecode here, unzip it, and try
cd ctcdecodeandpip install . - Pealse follow this link to install pytorch geometric
- You can install other required modules by conducting
pip install -r requirements.txtpip install transformers
- PHOENIX2014 dataset: Download the RWTH-PHOENIX-Weather 2014 Dataset [download link].
- PHOENIX2014-T datasetDownload the RWTH-PHOENIX-Weather 2014 Dataset [download link]
- CSL dataset： Request the CSL Dataset from this website [download link]
Download datasets and extract them, no further data preprocessing needed.
Here we provide the performance of the model and its corresponding weights.
| Dataset | Backbone | Dev WER | Test WER | Pretrained model |
|---|---|---|---|---|
| Phoenix14 | Resnet34 | 17.00 | 17.76 | [Google Drive] |
| Phoenix14T | Resnet34 | 15.96 | 18.48 | [Google Drive] |
| CSL-Daily | Resnet34 | 25.14 | 23.89 | [Google Drive] |
To evaluate the pretrained model, choose the dataset from phoenix2014/phoenix2014-T/CSL/CSL-Daily in line 3 in ./config/baseline.yaml first, and run the command below：
python main.py --load-weights path_to_weight.pt --phase test
python main.py --load-weights ./phoenix2014-T/best_model.pt --phase test
python main.py --load-weights ./phoenix2014/best_model.pt --phase test
python main.py --load-weights ./csl-daily/best_model.pt --phase test
To Training the SignDAGC model, choose the dataset from phoenix2014/phoenix2014-T/CSL/CSL-Daily in line 3 in ./config/baseline.yaml first, and run the command below：
python main.py
Multi-machine training (In fact, the results of the Multi-machine run are not good):
python -m torch.distributed.launch --nproc_per_node=2 main.py --device 0,1
Our code is based on SignGraph and GreedyViG.
The code for the sign language translation part is visible SignDAGC-SLT.