Not able to see the updated best_model.pth or best_model_x.pth after using continue_path

[RamakrishnaChaitanya]

I continued the training the fastpitch model that I started from scratch on a custom dataset using --continue_path argument. A best model was created at step 1325347 during first training from scratch and since then it is the only best model although the training is now at step 1492000 (~90 epoch's later). I'm using

"save_step": 10000,

The overall evaluation loss (avg_loss) seems to be reducing i.e., 2.84 at step 1325347 to 2.76 at step 1492000. I have observed the overriding of best_model_x.pth files while training the model from scratch, but that behavior is not observed after resuming the training using --continue_path argument.

FYI, while resuming the training, i made few changes in the configuration file i.e.,

"save_n_checkpoints": 2 (changed from 5 to 2 to save memory space), and 
"eval_batch_size": 16 (changed from 8 to 16 to improve the speed)
"binary_align_loss_alpha": 0 (changed from 0.1 to 0 to disable this loss)

Had i made any mistakes? Am i missing anything @eginhard ?

[eginhard]

In theory it should restore the previous best loss and continue saving those checkpoints, but I'd have to double check that it's working correctly. In the meantime you can use the regular checkpoint_XX.pth files that are saved every save_step steps if necessary.

[RamakrishnaChaitanya]

Checkpoints are getting generated and saved, periodically. However, best_model.pth and best_model_x.pth (x represents step) are not getting updated.

[RamakrishnaChaitanya]

I was getting like this after resuming the training.

It was mentioned that "Restoring from checkpoint_150000.pth" and "restoring best loss from best_model_1325347.pth". Is this the right way, as the fresh run is using both the checkpoint and best_model_x.pth?

In this fresh run, i was able to get both the latest best_model.pth and best_model_x.pth during the beginning of the first epoch, itself. I'm not sure how this happened!

[eginhard]

Yes, this is what should happen. With --continue_path you resume from the latest step and only the loss is read from the best model to update it later if necessary.

[RamakrishnaChaitanya]

Thankyou!