action: Clean up the logic to handle go-mod-directory

action: Clean up the logic to handle go-mod-directory #6466