Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Image /mnt overriding my machine's /mnt causing errors #530

Closed
japhill opened this issue Mar 17, 2022 · 6 comments
Closed

Image /mnt overriding my machine's /mnt causing errors #530

japhill opened this issue Mar 17, 2022 · 6 comments

Comments

@japhill
Copy link

japhill commented Mar 17, 2022

I ran this using singularity. I tried to tell the system to read and write files to a folder in my machine's /mnt/ folder. I keep getting an error. After inspecting, it looks like this image has an empty /mnt/ directory that is not writable. This is a problem for us and many users because it is very common to store large amounts of data in the /mnt/ folder on servers that access shared space from a common storage device. Please tell your Dockerfile to "RUN rm -rf /mnt/" or something (I'm not a docker expert by any means). The deepvariant docker container clearly does not need /mnt/

Setup

  • Centos 7
  • deepvariant 1.3.0
  • Singularity run pulling from here: docker://google/deepvariant:"1.3.0"
  • quickstart example

Steps to reproduce:
...please note that /mnt/share is an NFS mount. My server mounts a drive on another machine running nfs.service
mkdir -p /mnt/share/jasontest
cd /mnt/share/jasontest
INPUT_DIR="${PWD}/quickstart-testdata"
DATA_HTTP_DIR="https://storage.googleapis.com/deepvariant/quickstart-testdata"
mkdir -p ${INPUT_DIR}
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/NA12878_S1.chr20.10_10p1mb.bam
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/NA12878_S1.chr20.10_10p1mb.bam.bai
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/test_nist.b37_chr20_100kbp_at_10mb.bed
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/test_nist.b37_chr20_100kbp_at_10mb.vcf.gz
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/test_nist.b37_chr20_100kbp_at_10mb.vcf.gz.tbi
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/ucsc.hg19.chr20.unittest.fasta
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/ucsc.hg19.chr20.unittest.fasta.fai
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/ucsc.hg19.chr20.unittest.fasta.gz
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/ucsc.hg19.chr20.unittest.fasta.gz.fai
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/ucsc.hg19.chr20.unittest.fasta.gz.gzi
BIN_VERSION="1.3.0"
OUTPUT_DIR="${PWD}/quickstart-output"
mkdir -p "${OUTPUT_DIR}"
singularity run -B /usr/lib/locale/:/usr/lib/locale/ docker://google/deepvariant:"${BIN_VERSION}" /opt/deepvariant/bin/run_deepvariant --model_type=WGS --ref="${INPUT_DIR}"/ucsc.hg19.chr20.unittest.fasta --reads="${INPUT_DIR}"/NA12878_S1.chr20.10_10p1mb.bam --regions "chr20:10,000,000-10,010,000" --output_vcf="${OUTPUT_DIR}"/output.vcf.gz --output_gvcf="${OUTPUT_DIR}"/output.g.vcf.gz --intermediate_results_dir "${OUTPUT_DIR}/intermediate_results_dir" --num_shards=1

stack trace:

I0317 09:40:21.184321 140398386173760 run_deepvariant.py:341] Creating a directory for intermediate results in /mnt/share/jasontest/quickstart-output/intermediate_results_dir
Traceback (most recent call last):
File "/opt/deepvariant/bin/run_deepvariant.py", line 493, in
app.run(main)
File "/usr/local/lib/python3.8/dist-packages/absl/app.py", line 312, in run
_run_main(main, args)
File "/usr/local/lib/python3.8/dist-packages/absl/app.py", line 258, in _run_main
sys.exit(main(argv))
File "/opt/deepvariant/bin/run_deepvariant.py", line 460, in main
intermediate_results_dir = check_or_create_intermediate_results_dir(
File "/opt/deepvariant/bin/run_deepvariant.py", line 343, in check_or_create_intermediate_results_dir
os.makedirs(intermediate_results_dir)
File "/usr/lib/python3.8/os.py", line 213, in makedirs
makedirs(head, exist_ok=exist_ok)
File "/usr/lib/python3.8/os.py", line 213, in makedirs
makedirs(head, exist_ok=exist_ok)
File "/usr/lib/python3.8/os.py", line 213, in makedirs
makedirs(head, exist_ok=exist_ok)
[Previous line repeated 1 more time]
File "/usr/lib/python3.8/os.py", line 223, in makedirs
mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/mnt/share'

Does the quick start test work on your system?
The quick test works on my system as long as my data is not in the /mnt/ folder.

Any additional context:

@pichuan
Copy link
Collaborator

pichuan commented Mar 23, 2022

Hi @japhill ,
Just to make sure I understand this - are you saying that the Docker image has /mnt in it, and as result was causing problem with Singularity?

I do see a /mnt directory:

$ sudo docker run google/deepvariant:1.3.0 ls -lh /
total 48K
lrwxrwxrwx   1 root root    7 Oct  6 16:47 bin -> usr/bin
drwxr-xr-x   2 root root 4.0K Apr 15  2020 boot
drwxr-xr-x   5 root root  340 Mar 23 23:34 dev
drwxr-xr-x   1 root root 4.0K Mar 23 23:34 etc
drwxr-xr-x   2 root root 4.0K Apr 15  2020 home
lrwxrwxrwx   1 root root    7 Oct  6 16:47 lib -> usr/lib
lrwxrwxrwx   1 root root    9 Oct  6 16:47 lib32 -> usr/lib32
lrwxrwxrwx   1 root root    9 Oct  6 16:47 lib64 -> usr/lib64
lrwxrwxrwx   1 root root   10 Oct  6 16:47 libx32 -> usr/libx32
drwxr-xr-x   2 root root 4.0K Oct  6 16:47 media
drwxr-xr-x   2 root root 4.0K Oct  6 16:47 mnt
drwxr-xr-x   1 root root 4.0K Dec  6 23:17 opt
dr-xr-xr-x 525 root root    0 Mar 23 23:34 proc
drwx------   1 root root 4.0K Dec  6 23:15 root
drwxr-xr-x   5 root root 4.0K Oct  6 16:58 run
lrwxrwxrwx   1 root root    8 Oct  6 16:47 sbin -> usr/sbin
drwxr-xr-x   2 root root 4.0K Oct  6 16:47 srv
dr-xr-xr-x  13 root root    0 Mar 23 23:34 sys
drwxrwxrwt   1 root root 4.0K Dec  6 23:19 tmp
drwxr-xr-x   1 root root 4.0K Oct  6 16:47 usr
drwxr-xr-x   1 root root 4.0K Oct  6 16:58 var

which is empty, so I think your suggestion of something like "RUN rm -rf /mnt/" makes sense. I'll also do a quick search to see if there are better approaches here.

Thanks for the feedback. I'll track internally and make sure this is updated in the next release.

@pichuan
Copy link
Collaborator

pichuan commented Mar 23, 2022

@japhill But for now, I also wonder if you can use the -B option in Singularity?
https://sylabs.io/guides/3.1/user-guide/bind_paths_and_mounts.html#user-defined-bind-paths
Can you try it and let me know if it works for you?

@akolesnikov
Copy link
Collaborator

Just to add to the previous post. Your file system is not accessible from the container. Using -B option you may map (mount) some directories from your file system to be accessible from within the container.

There is a similar issue with the similar content. #506 (comment)

@japhill
Copy link
Author

japhill commented Mar 24, 2022

Thank you for the help. I'm somewhat new to Singularity and how it works, so this has been helpful.

@japhill japhill closed this as completed Mar 24, 2022
@pichuan
Copy link
Collaborator

pichuan commented Mar 25, 2022

Hi @japhill , to give you an update, I plan to add this section to our FAQ in the next release:


Issues with /mnt/

User reported that sometimes their setup uses /mnt/, which exists in our Docker image, and it has caused an issue in Singularity.

You can use -B in Singularity to avoid this issue. See:
#530 (comment) for more details.


Given that this solution works, and is a standard Singularity flag, I won't plan to remove /mnt from our Docker images in the future.

@japhill
Copy link
Author

japhill commented Mar 26, 2022 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants