Error in during reference based clustering

[Shirley-Q]

Hi,
I have an error in during reference based clustering when I run the OPERA-MS. I am not able to solve it. Thank you! Looking forward to your reply. Thank you very much!
The content of 'sequence_similarity_clustering.err' is followed.
/home/emma/QXJ/nanopore/trim_nanopore_out//intermediate_files/hierarchical_clustering/clusters

rm -r /home/emma/QXJ/nanopore/trim_nanopore_out//intermediate_files/reference_clustering/MASH/
rm: cannot remove '/home/emma/QXJ/nanopore/trim_nanopore_out//intermediate_files/reference_clustering/MASH/': No such file or directory

mkdir -p /home/emma/QXJ/nanopore/trim_nanopore_out//intermediate_files/reference_clustering/MASH/

mkdir /home/emma/QXJ/nanopore/trim_nanopore_out//intermediate_files/reference_clustering/MASH//INTER_FILES/

mkdir /home/emma/QXJ/nanopore/trim_nanopore_out//intermediate_files/reference_clustering/MASH//PARTIAL_SKETCH/
Use of uninitialized value $contig_cluster in hash element at /home/emma/OPERA-MS/bin/sequence_similarity_clustering.pl line 888.

/home/emma/OPERA-MS//utils//mash paste /home/emma/QXJ/nanopore/trim_nanopore_out//intermediate_files/reference_clustering/MASH//cluster.msh /home/emma/QXJ/nanopore/trim_nanopore_out//intermediate_files/reference_clustering/MASH//PARTIAL_SKETCH//partial_Sketch*
ERROR: The file "/home/emma/QXJ/nanopore/trim_nanopore_out//intermediate_files/reference_clustering/MASH//PARTIAL_SKETCH//partial_Sketch*" does not look like a sketch.

/home/emma/OPERA-MS//utils//mash dist -p 42 -d 0.90 /home/emma/OPERA-MS//genomeDB_Sketch.msh /home/emma/QXJ/nanopore/trim_nanopore_out//intermediate_files/reference_clustering/MASH//cluster.msh > /home/emma/QXJ/nanopore/trim_nanopore_out//intermediate_files/reference_clustering/MASH//mash_dist.dat 2> /home/emma/QXJ/nanopore/trim_nanopore_out//intermediate_files/reference_clustering/MASH//mash_dist.err
Error in during bin/mash dist. Please see /home/emma/QXJ/nanopore/trim_nanopore_out//intermediate_files/reference_clustering/MASH//mash_dist.err for details.
The detail of 'mash_dist.err' is as followed.
terminate called after throwing an instance of 'kj::ExceptionImpl'
what(): src/kj/io.c++:42: failed: expected n >= minBytes; Premature EOF
stack: 0x45ed59 0x4623aa 0x463ca7 0x46316d 0x45a736 0x42892b 0x42e603 0x417ad1 0x438ea2 0x425076 0x40377c 0x7fd5ec5a8830 0x403db1
Aborted (core dumped)

[dbertran78]

Dear Shirley-Q,
Can you please run the following command to create a complete log files:
sh /home/emma/OPERA-MS/bin/get_log.sh /home/emma/QXJ/nanopore/trim_nanopore_out/
OPERA-MS-LOG.tar.gz will be create in your working directory.

Please send OPERA-MS-LOG.tar.gz, and the command line used for the OPERA-MS run to tong_chengxuan@gis.a-star.edu.sg and bertrandd@gis.a-star.edu.sg

Best regards,

--- Denis

[ADSakura]

Hi,
I have a similar error of 'kj::ExceptionImpl' and the 'mash_dist.err' is as followed.

terminate called after throwing an instance of 'kj::ExceptionImpl'
what(): src/capnp/serialize.c++:159: failed: expected segmentCount < 512; Message has too many segments.
stack: 0x45ed59 0x4623aa 0x45ab26 0x42892b 0x42e603 0x417ad1 0x438ea2 0x425076 0x40377c 0x7f8a469a0830 0x403db1
Aborted (core dumped)

How did you solve this problem? Could you give me any advice about fixing this? Thanks.

[dbertran78]

Dear ADSakura,
We will need additional information to help you solve this issue.
Could you please generate the complete log file using the OPERA-MS/bin/get_log.sh script.
Please send the generated OPERA-MS-LOG.tar.gz, and the command line you used for the OPERA-MS run to tong_chengxuan@gis.a-star.edu.sg and bertrandd@gis.a-star.edu.sg

Best regards,

--- Denis

[ValeriiaLadyhina]

Hi!

I also got the error of reference based clustering and need help.

Here are the logs:

1. "mkdir -p /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files

mkdir -p /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A

version v0.9.0

/export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms//megahit_v1.0.4-beta_LINUX_CPUONLY_x86_64-bin//megahit --num-cpu-threads 20 -1 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/bowtie2/clean_from_host_genome/Sample_1_5A_L001_host_removed_R1.fastq.gz -2 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/bowtie2/clean_from_host_genome/Sample_1_5A_L001_host_removed_R2.fastq.gz -o /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/megahit_assembly> /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/megahit.out 2> /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/megahit.err

rm -r /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/megahit_assembly/intermediate_contigs

mkdir -p /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/read_mapping

/export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms/perl /export/opt/sw/opera-ms/0.8.3/OPERA-MS//OPERA-LG/bin//OPERA-long-read.pl --contig-file /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/megahit_assembly/final.contigs.fa --kmer 60 --long-read-file /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/minimap2/clean_from_host_genome/ready/Sample_1_5A_NP_host_removed.fastq --output-prefix opera --output-directory /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/read_mapping --num-of-processors 20 --opera /export/opt/sw/opera-ms/0.8.3/OPERA-MS//OPERA-LG/bin/ --illumina-read1 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/bowtie2/clean_from_host_genome/Sample_1_5A_L001_host_removed_R1.fastq.gz --illumina-read2 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/bowtie2/clean_from_host_genome/Sample_1_5A_L001_host_removed_R2.fastq.gz --samtools-dir /export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms/ --perl-dir /export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms/ --minimap2 /export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms/ --short-read-tooldir /export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms/ --skip-opera 1 --skip-short-read-mapping 1 > /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/read_mapping/OPERA-long-read.out 2> /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/read_mapping/OPERA-long-read.err

mkdir /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/coverage_estimation

cp /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/opera-ms-utils.config /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/coverage_estimation/cov_estimate.config

/export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms/perl /export/opt/sw/opera-ms/0.8.3/OPERA-MS//bin/cov_estimate.pl /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/coverage_estimation/cov_estimate.config > /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/coverage_estimation/cov_estimate.out 2> /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/coverage_estimation/cov_estimate.err

/export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms/perl /export/opt/sw/opera-ms/0.8.3/OPERA-MS/bin/filter_edge_coverage_ratio.pl /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/read_mapping /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/coverage_estimation/contigs_340_80 2> /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/read_mapping/filter_edge_coverage_ratio.err

mkdir -p /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering

mkdir /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering/pairedEdges_i0

ln -s /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/read_mapping/pairedEdges_i0 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering/pairedEdges_i0/pairedEdges_i0

mkdir /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering/pairedEdges_i1

ln -s /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/read_mapping/pairedEdges_i1 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering/pairedEdges_i1/pairedEdges_i1

mkdir /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering/pairedEdges_i2

ln -s /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/read_mapping/pairedEdges_i2 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering/pairedEdges_i2/pairedEdges_i2

mkdir /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering/pairedEdges_i3

ln -s /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/read_mapping/pairedEdges_i3 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering/pairedEdges_i3/pairedEdges_i3

mkdir /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering/pairedEdges_i4

ln -s /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/read_mapping/pairedEdges_i4 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering/pairedEdges_i4/pairedEdges_i4

mkdir /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering/pairedEdges_i5

ln -s /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/read_mapping/pairedEdges_i5 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering/pairedEdges_i5/pairedEdges_i5

/export/opt/sw/opera-ms/0.8.3/OPERA-MS/bin/sigma /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering/sigma.config > /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering/sigma.out 2> /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering/sigma.err

/export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms/perl /export/opt/sw/opera-ms/0.8.3/OPERA-MS/bin/refine_r_estimate.pl /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/coverage_estimation/contigs_340_80 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering /export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms/ 20 6000000 /export/opt/sw/opera-ms/0.8.3/OPERA-MS/ 2> /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering/refine_r.err

mkdir -p /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering

/export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms/perl /export/opt/sw/opera-ms/0.8.3/OPERA-MS/bin/sequence_similarity_clustering.pl /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/read_mapping /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/megahit_assembly/final.contigs.fa 20 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS_DB /export/opt/sw/opera-ms/0.8.3/OPERA-MS/ /export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms/ /export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms//MUMmer3.23/ NULL 2> /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/sequence_similarity_clustering.err

Error in during reference based clustering. Please see /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/sequence_similarity_clustering.err for details."

2. "cat sequence_similarity_clustering.err
   /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering/clusters

rm -r /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH/
rm: cannot remove ‘/proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH/’: No such file or directory

mkdir -p /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH/

mkdir /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//INTER_FILES/

mkdir /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//PARTIAL_SKETCH/

/export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms//mash sketch -p 20 -k 21 -s 1000 -o /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//PARTIAL_SKETCH//partial_Sketch0 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//INTER_FILES//* > /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//mash_sketch.out 2> /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//mash_sketch.err

/export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms//mash dist -p 20 -d 0.90 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS_DB/genomes.msh /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//PARTIAL_SKETCH//partial_Sketch0.msh > /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//mash_dist_0.dat 2> /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//mash_dist.err
sh: line 1: 6795 Floating point exception(core dumped) /export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms//mash dist -p 20 -d 0.90 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS_DB/genomes.msh /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//PARTIAL_SKETCH//partial_Sketch0.msh > /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//mash_dist_0.dat 2> /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//mash_dist.err
Error in during bin/mash dist. Please see /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//mash_dist.err for details."

3. But file mash_dist.err doesn't have any information.

Can you please help me out?

Best regards,
Valeriia

[jsgounot]

Hi Valeriia,

please open a new issue, your error message is not the same. Here you have a weird floating point exception error with mash:

sh: line 1: 6795 Floating point exception(core dumped) /export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms//mash dist -p 20 -d 0.90 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS_DB/genomes.msh /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//PARTIAL_SKETCH//partial_Sketch0.msh > /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//mash_dist_0.dat 2> /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//mash_dist.err

I can't know exactly what's happening here, but I would guess a memory error. How much memory were you giving here?

Can you try to increase this value and rerun this command line alone?

/export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms//mash dist -p 20 -d 0.90 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS_DB/genomes.msh /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//PARTIAL_SKETCH//partial_Sketch0.msh > /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//mash_dist_0.dat

Can you also try to run mash help command to see if this is not a compilation/system issue?

/export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms//mash dist -h

Regards,
JSG