Singularity tempfile/TMPDIR error: Tutorials fail

[RhettRautsaw]

Describe the issue:
I am following the tutorial for PacBio HiFi data. When I reach the step for calling singularity to run_deepvariant (this step), I receive an error which appears to be associated with the tempfile/TMPDIR path:

Command:

BIN_VERSION="1.3.0"
mkdir -p deepvariant1

singularity exec --bind /usr/lib/locale/ \
  docker://google/deepvariant:${BIN_VERSION} \
    /opt/deepvariant/bin/run_deepvariant \
    --model_type PACBIO \
    --ref reference/GRCh38_no_alt_analysis_set.fasta \
    --reads input/HG003.GRCh38.chr20.pFDA_truthv2.bam \
    --output_vcf deepvariant1/output.vcf.gz \
    --num_shards $(nproc) \
    --regions chr20

Error 1

INFO:    Using cached SIF image
I0403 10:34:56.987876 23171167450944 run_deepvariant.py:345] Re-using the directory for intermediate results in /tmp/tmp40dn43xh

***** Intermediate results will be written to /tmp/tmp40dn43xh in docker. ****


***** Running the command:*****
time seq 0 15 | parallel -q --halt 2 --line-buffer /opt/deepvariant/bin/make_examples --mode calling --ref "reference/GRCh38_no_alt_analysis_set.fasta" --reads "input/HG003.GRCh38.chr20.pFDA_truthv2.bam" --examples "/[tmp/tmp40dn43xh/make_examples.tfrecord@16.gz](mailto:tmp/tmp40dn43xh/make_examples.tfrecord@16.gz)" --add_hp_channel --alt_aligned_pileup "diff_channels" --noparse_sam_aux_fields --pileup_image_width "199" --norealign_reads --regions "chr20" --nosort_by_haplotypes --vsc_min_fraction_indels "0.12" --task {}

Error in tempfile() using template /local_scratch/pbs.4762337.pbs02/parXXXXX.par: Parent directory (/local_scratch/pbs.4762337.pbs02/) does not exist at /usr/bin/parallel line 3889.

I can set export TMPDIR = "." and this bypasses this error only to receive a different error stating that it cannot find any of the files that are downloaded in the previous steps of the tutorial.

Error 2

INFO:    Using cached SIF image
I0404 16:29:50.730109 22987118802752 run_deepvariant.py:345] Re-using the directory for intermediate results in ./tmpkj84jstw

***** Intermediate results will be written to ./tmpkj84jstw in docker. ****


***** Running the command:*****
time seq 0 15 | parallel -q --halt 2 --line-buffer /opt/deepvariant/bin/make_examples --mode calling --ref "reference/GRCh38_no_alt_analysis_set.fasta" --reads "input/HG003.GRCh38.chr20.pFDA_truthv2.bam" --examples "./tmpkj84jstw/make_examples.tfrecord@16.gz" --add_hp_channel --alt_aligned_pileup "diff_channels" --noparse_sam_aux_fields --pileup_image_width "199" --norealign_reads --regions "chr20" --nosort_by_haplotypes --vsc_min_fraction_indels "0.12" --task {}

[E::hts_open_format] Failed to open file "input/HG003.GRCh38.chr20.pFDA_truthv2.bam" : No such file or directory
Traceback (most recent call last):
  File "./Bazel.runfiles_bpldxvlm/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 180, in <module>
    app.run(main)
  File "/home/rrautsa/Bazel.runfiles_bpldxvlm/runfiles/absl_py/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/home/rrautsa/Bazel.runfiles_bpldxvlm/runfiles/absl_py/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "./Bazel.runfiles_bpldxvlm/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 166, in main
    options = default_options(add_flags=True, flags_obj=FLAGS)
  File "./Bazel.runfiles_bpldxvlm/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 128, in default_options
    samples_in_order, sample_role_to_train = one_sample_from_flags(
  File "./Bazel.runfiles_bpldxvlm/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 85, in one_sample_from_flags
    sample_name = make_examples_core.assign_sample_name(
  File "/home/rrautsa/Bazel.runfiles_bpldxvlm/runfiles/com_google_deepvariant/deepvariant/make_examples_core.py", line 131, in assign_sample_name
    with sam.SamReader(reads_filenames.split(',')[0]) as sam_reader:
  File "/home/rrautsa/Bazel.runfiles_bpldxvlm/runfiles/com_google_deepvariant/third_party/nucleus/io/genomics_reader.py", line 221, in __init__
    self._reader = self._native_reader(input_path, **kwargs)
  File "/home/rrautsa/Bazel.runfiles_bpldxvlm/runfiles/com_google_deepvariant/third_party/nucleus/io/sam.py", line 260, in _native_reader
    return NativeSamReader(input_path, **kwargs)
  File "/home/rrautsa/Bazel.runfiles_bpldxvlm/runfiles/com_google_deepvariant/third_party/nucleus/io/sam.py", line 227, in __init__
    self._reader = sam_reader.SamReader.from_file(
ValueError: Not found: Could not open input/HG003.GRCh38.chr20.pFDA_truthv2.bam

...REPEAT ABOVE ERROR {NPROC} TIMES...

parallel: This job failed:
/opt/deepvariant/bin/make_examples --mode calling --ref reference/GRCh38_no_alt_analysis_set.fasta --reads input/HG003.GRCh38.chr20.pFDA_truthv2.bam --examples ./tmpkj84jstw/make_examples.tfrecord@16.gz --add_hp_channel --alt_aligned_pileup diff_channels --noparse_sam_aux_fields --pileup_image_width 199 --norealign_reads --regions chr20 --nosort_by_haplotypes --vsc_min_fraction_indels 0.12 --task 12

real	0m4.843s
user	0m3.036s
sys	0m0.866s

Setup

• Operating system: CentOS Linux release 8.2.2004 (Core)
• DeepVariant version: 1.3.0
• Installation method (Docker, built from source, etc.): Singularity/Docker
• Type of data: Tutorial Data

Does the quick start test work on your system?
The same error occurs in the quick start test with Error in tempfile() using template... as above.

[pichuan]

Hi @RhettRautsaw ,

Can you take a look at
#524 (comment)
or
#530 (comment)
and see if either those help with your situation?

Let me know. Thanks!

[RhettRautsaw]

Hi @pichuan,

Thanks for replying so quickly. I have tried both of those things. Doing export TMPDIR="$PWD/tmp_dir" causes the same as Error 1 above.

INFO:    Using cached SIF image
I0404 17:45:45.958600 23016861075264 run_deepvariant.py:345] Re-using the directory for intermediate results in /tmp/tmp_ui_t3bd

***** Intermediate results will be written to /tmp/tmp_ui_t3bd in docker. ****


***** Running the command:*****
time seq 0 15 | parallel -q --halt 2 --line-buffer /opt/deepvariant/bin/make_examples --mode calling --ref "reference/GRCh38_no_alt_analysis_set.fasta" --reads "input/HG003.GRCh38.chr20.pFDA_truthv2.bam" --examples "/tmp/tmp_ui_t3bd/make_examples.tfrecord@16.gz" --add_hp_channel --alt_aligned_pileup "diff_channels" --noparse_sam_aux_fields --pileup_image_width "199" --norealign_reads --regions "chr20" --nosort_by_haplotypes --vsc_min_fraction_indels "0.12" --task {}

Error in tempfile() using template /scratch1/rrautsa/deepvariant_test/tmp_dir/parXXXXX.par: Parent directory (/scratch1/rrautsa/deepvariant_test/tmp_dir/) does not exist at /usr/bin/parallel line 3889.

real	0m0.257s
user	0m0.121s
sys	0m0.125s

And perhaps I don't understand singularity enough, but it seems that no matter how I change the --bind input it gives me the following error:

INFO:    Using cached SIF image
FATAL:   failed to open /bin/sh for inspection: failed to open elf binary /bin/sh: open /bin/sh: no such file or directory

[pichuan]

@RhettRautsaw Can you share the command you ran when you use --bind?

[pichuan]

Hi @RhettRautsaw , we haven't heard from you for a while, so I'll close this now.
Feel free to reopen.