Skip to content

Latest commit

 

History

History
63 lines (38 loc) · 3.26 KB

README.md

File metadata and controls

63 lines (38 loc) · 3.26 KB

Prerequisites

  • This project is implemented in Pytorch (>1.8). Thus please install Pytorch first.
  • ctcdecode==0.4 [parlance/ctcdecode]，for beam search decode.
  • For these who failed install ctcdecode (and it always does), you can download ctcdecode here, unzip it, and try cd ctcdecode and pip install .
  • Pealse follow this link to install pytorch geometric
  • You can install other required modules by conducting pip install -r requirements.txt pip install transformers

Data Preparation

  1. PHOENIX2014 dataset: Download the RWTH-PHOENIX-Weather 2014 Dataset [download link].
  2. PHOENIX2014-T datasetDownload the RWTH-PHOENIX-Weather 2014 Dataset [download link]
  3. CSL dataset： Request the CSL Dataset from this website [download link]

Download datasets and extract them, no further data preprocessing needed.

SLR

Weights

Here we provide the performance of the model and its corresponding weights.

Dataset Backbone Dev WER Test WER Pretrained model
Phoenix14 Resnet34 17.00 17.76 [Google Drive]
Phoenix14T Resnet34 15.96 18.48 [Google Drive]
CSL-Daily Resnet34 25.14 23.89 [Google Drive]

Evaluate

To evaluate the pretrained model, choose the dataset from phoenix2014/phoenix2014-T/CSL/CSL-Daily in line 3 in ./config/baseline.yaml first, and run the command below：

python main.py --load-weights path_to_weight.pt --phase test

python main.py --load-weights ./phoenix2014-T/best_model.pt --phase test

python main.py --load-weights ./phoenix2014/best_model.pt --phase test

python main.py --load-weights ./csl-daily/best_model.pt --phase test

Training

To Training the SignDAGC model, choose the dataset from phoenix2014/phoenix2014-T/CSL/CSL-Daily in line 3 in ./config/baseline.yaml first, and run the command below：

python main.py

Multi-machine training (In fact, the results of the Multi-machine run are not good):

python -m torch.distributed.launch --nproc_per_node=2 main.py --device 0,1

Acknowledgments

Our code is based on SignGraph and GreedyViG.

SLT

The code for the sign language translation part is visible SignDAGC-SLT.