Image /mnt overriding my machine's /mnt causing errors

[japhill]

I ran this using singularity. I tried to tell the system to read and write files to a folder in my machine's /mnt/ folder. I keep getting an error. After inspecting, it looks like this image has an empty /mnt/ directory that is not writable. This is a problem for us and many users because it is very common to store large amounts of data in the /mnt/ folder on servers that access shared space from a common storage device. Please tell your Dockerfile to "RUN rm -rf /mnt/" or something (I'm not a docker expert by any means). The deepvariant docker container clearly does not need /mnt/

Setup

• Centos 7
• deepvariant 1.3.0
• Singularity run pulling from here: docker://google/deepvariant:"1.3.0"
• quickstart example

Steps to reproduce:
...please note that /mnt/share is an NFS mount. My server mounts a drive on another machine running nfs.service
mkdir -p /mnt/share/jasontest
cd /mnt/share/jasontest
INPUT_DIR="${PWD}/quickstart-testdata"
DATA_HTTP_DIR="https://storage.googleapis.com/deepvariant/quickstart-testdata"
mkdir -p ${INPUT_DIR}
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/NA12878_S1.chr20.10_10p1mb.bam
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/NA12878_S1.chr20.10_10p1mb.bam.bai
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/test_nist.b37_chr20_100kbp_at_10mb.bed
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/test_nist.b37_chr20_100kbp_at_10mb.vcf.gz
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/test_nist.b37_chr20_100kbp_at_10mb.vcf.gz.tbi
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/ucsc.hg19.chr20.unittest.fasta
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/ucsc.hg19.chr20.unittest.fasta.fai
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/ucsc.hg19.chr20.unittest.fasta.gz
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/ucsc.hg19.chr20.unittest.fasta.gz.fai
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/ucsc.hg19.chr20.unittest.fasta.gz.gzi
BIN_VERSION="1.3.0"
OUTPUT_DIR="${PWD}/quickstart-output"
mkdir -p "${OUTPUT_DIR}"
singularity run -B /usr/lib/locale/:/usr/lib/locale/ docker://google/deepvariant:"${BIN_VERSION}" /opt/deepvariant/bin/run_deepvariant --model_type=WGS --ref="${INPUT_DIR}"/ucsc.hg19.chr20.unittest.fasta --reads="${INPUT_DIR}"/NA12878_S1.chr20.10_10p1mb.bam --regions "chr20:10,000,000-10,010,000" --output_vcf="${OUTPUT_DIR}"/output.vcf.gz --output_gvcf="${OUTPUT_DIR}"/output.g.vcf.gz --intermediate_results_dir "${OUTPUT_DIR}/intermediate_results_dir" --num_shards=1

stack trace:

I0317 09:40:21.184321 140398386173760 run_deepvariant.py:341] Creating a directory for intermediate results in /mnt/share/jasontest/quickstart-output/intermediate_results_dir
Traceback (most recent call last):
File "/opt/deepvariant/bin/run_deepvariant.py", line 493, in
app.run(main)
File "/usr/local/lib/python3.8/dist-packages/absl/app.py", line 312, in run
_run_main(main, args)
File "/usr/local/lib/python3.8/dist-packages/absl/app.py", line 258, in _run_main
sys.exit(main(argv))
File "/opt/deepvariant/bin/run_deepvariant.py", line 460, in main
intermediate_results_dir = check_or_create_intermediate_results_dir(
File "/opt/deepvariant/bin/run_deepvariant.py", line 343, in check_or_create_intermediate_results_dir
os.makedirs(intermediate_results_dir)
File "/usr/lib/python3.8/os.py", line 213, in makedirs
makedirs(head, exist_ok=exist_ok)
File "/usr/lib/python3.8/os.py", line 213, in makedirs
makedirs(head, exist_ok=exist_ok)
File "/usr/lib/python3.8/os.py", line 213, in makedirs
makedirs(head, exist_ok=exist_ok)
[Previous line repeated 1 more time]
File "/usr/lib/python3.8/os.py", line 223, in makedirs
mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/mnt/share'

Does the quick start test work on your system?
The quick test works on my system as long as my data is not in the /mnt/ folder.

Any additional context:

[pichuan]

Hi @japhill ,
Just to make sure I understand this - are you saying that the Docker image has /mnt in it, and as result was causing problem with Singularity?

I do see a /mnt directory:

$ sudo docker run google/deepvariant:1.3.0 ls -lh /
total 48K
lrwxrwxrwx   1 root root    7 Oct  6 16:47 bin -> usr/bin
drwxr-xr-x   2 root root 4.0K Apr 15  2020 boot
drwxr-xr-x   5 root root  340 Mar 23 23:34 dev
drwxr-xr-x   1 root root 4.0K Mar 23 23:34 etc
drwxr-xr-x   2 root root 4.0K Apr 15  2020 home
lrwxrwxrwx   1 root root    7 Oct  6 16:47 lib -> usr/lib
lrwxrwxrwx   1 root root    9 Oct  6 16:47 lib32 -> usr/lib32
lrwxrwxrwx   1 root root    9 Oct  6 16:47 lib64 -> usr/lib64
lrwxrwxrwx   1 root root   10 Oct  6 16:47 libx32 -> usr/libx32
drwxr-xr-x   2 root root 4.0K Oct  6 16:47 media
drwxr-xr-x   2 root root 4.0K Oct  6 16:47 mnt
drwxr-xr-x   1 root root 4.0K Dec  6 23:17 opt
dr-xr-xr-x 525 root root    0 Mar 23 23:34 proc
drwx------   1 root root 4.0K Dec  6 23:15 root
drwxr-xr-x   5 root root 4.0K Oct  6 16:58 run
lrwxrwxrwx   1 root root    8 Oct  6 16:47 sbin -> usr/sbin
drwxr-xr-x   2 root root 4.0K Oct  6 16:47 srv
dr-xr-xr-x  13 root root    0 Mar 23 23:34 sys
drwxrwxrwt   1 root root 4.0K Dec  6 23:19 tmp
drwxr-xr-x   1 root root 4.0K Oct  6 16:47 usr
drwxr-xr-x   1 root root 4.0K Oct  6 16:58 var

which is empty, so I think your suggestion of something like "RUN rm -rf /mnt/" makes sense. I'll also do a quick search to see if there are better approaches here.

Thanks for the feedback. I'll track internally and make sure this is updated in the next release.

[pichuan]

@japhill But for now, I also wonder if you can use the -B option in Singularity?
https://sylabs.io/guides/3.1/user-guide/bind_paths_and_mounts.html#user-defined-bind-paths
Can you try it and let me know if it works for you?

[akolesnikov]

Just to add to the previous post. Your file system is not accessible from the container. Using -B option you may map (mount) some directories from your file system to be accessible from within the container.

There is a similar issue with the similar content. #506 (comment)

[japhill]

Thank you for the help. I'm somewhat new to Singularity and how it works, so this has been helpful.

[pichuan]

Hi @japhill , to give you an update, I plan to add this section to our FAQ in the next release:

---

Issues with /mnt/

User reported that sometimes their setup uses /mnt/, which exists in our Docker image, and it has caused an issue in Singularity.

You can use -B in Singularity to avoid this issue. See:
#530 (comment) for more details.

---

Given that this solution works, and is a standard Singularity flag, I won't plan to remove /mnt from our Docker images in the future.

[japhill]

Thank you very much. That will help out the folks new to docker/singularity. After using the -B option, we are good to go now! From: Pi-Chuan Chang ***@***.***> Sent: Friday, March 25, 2022 7:19 PM To: google/deepvariant ***@***.***> Cc: Jason Phillips ***@***.***>; Mention ***@***.***> Subject: Re: [google/deepvariant] Image /mnt overriding my machine's /mnt causing errors (Issue #530) Hi @japhill<https://github.com/japhill> , to give you an update, I plan to add this section to our FAQ in the next release:

…

________________________________ Issues with /mnt/ User reported that sometimes their setup uses /mnt/, which exists in our Docker image, and it has caused an issue in Singularity. You can use -B in Singularity to avoid this issue. See: #530 (comment)<#530 (comment)> for more details.

________________________________ Given that this solution works, and is a standard Singularity flag, I won't plan to remove /mnt from our Docker images in the future. — Reply to this email directly, view it on GitHub<#530 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AHAS6GHJOPUE7DQPYAT264TVBZCWLANCNFSM5Q7A6FXQ>. You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>>