Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

job separation by chromosome #57

Merged
merged 7 commits into from
Aug 19, 2024
Merged

job separation by chromosome #57

merged 7 commits into from
Aug 19, 2024

Conversation

hyunhwan-bcm
Copy link
Contributor

No description provided.

@hyunhwan-bcm hyunhwan-bcm marked this pull request as ready for review August 15, 2024 15:45
@hyunhwan-bcm hyunhwan-bcm changed the title [WIP] vep output separation by chromosome for feature engineering part 1 vep output separation by chromosome for feature engineering part 1 Aug 15, 2024
@jylee-bcm
Copy link
Contributor

Good job! It would be good if you add the statistics in the message regardless of whether it's improved or not, so we can track.

@hyunhwan-bcm
Copy link
Contributor Author

The memory/running time stats for running on a WES sample

❯ nextflow run ../../AI_MARRVEL/main.nf \
  --ref_dir /Volumes/HWAN-T7/data-dependency \
  --input_vcf $PWD/data/679529/679529.hard-filtered.vcf.gz --input_hpo $PWD/data/679529_HPOs.txt \
  --outdir output/679529/ -with-docker chaozhongliu/aim-lite --no_filter_exonic -with-trace trace.html
Nextflow 24.04.4 is available - Please consider updating your version to it

N E X T F L O W ~ version 24.04.3

Launching `../../AI_MARRVEL/main.nf` [stoic_laplace] DSL2 - revision: 5b75fd25c1

executor >  local (14)
[49/e6f1f6] INDEX_VCF                  [100%] 1 of 1 ✔
[57/137c65] VCF_PRE_PROCESS            [100%] 1 of 1 ✔
[20/649683] ANNOT_PHRANK               [100%] 1 of 1 ✔
[b4/7842ed] ANNOT_ENSMBLE              [100%] 1 of 1 ✔
[a5/dedfc2] TO_GENE_SYM                [100%] 1 of 1 ✔
[4e/d8da12] PHRANK_SCORING             [100%] 1 of 1 ✔
[73/d78a3f] HPO_SIM                    [100%] 1 of 1 ✔
[ee/35b6ab] REMOVE_MITO_AND_UNKOWN_CHR [100%] 1 of 1 ✔
[73/d98e9b] FILTER_BED                 [100%] 1 of 1 ✔
[7b/bb38e4] FILTER_PROBAND             [100%] 1 of 1 ✔
[6d/40f119] VEP_ANNOTATE               [100%] 1 of 1 ✔
[74/aa3aab] FEATURE_ENGINEERING_PART1  [100%] 1 of 1 ✔
[d0/3ed0c2] FEATURE_ENGINEERING_PART2  [100%] 1 of 1 ✔
[c6/eb1b69] PREDICTION                 [100%] 1 of 1 ✔
Completed at: 15-Aug-2024 01:44:27
Duration    : 25m 27s
CPU hours   : 5.1
Succeeded   : 14

@hyunhwan-bcm
Copy link
Contributor Author

hyunhwan-bcm commented Aug 15, 2024 •
edited
Loading

Trace

task_id	hash	native_id	name	status	exit	submit	duration	realtime	%cpu	peak_rss	peak_vmem	rchar	wchar
2	49/e6f1f6	19352	INDEX_VCF	COMPLETED	0	2024-08-15 01:19:01.549	4s	3.5s	122.9%	16.3 MB	889.1 MB	261.4 MB	221.6 MB
1	73/d78a3f	19399	HPO_SIM	COMPLETED	0	2024-08-15 01:19:05.540	18.2s	17.7s	94.9%	874.8 MB	1.6 GB	119.3 MB	72.5 MB
3	20/649683	19570	ANNOT_PHRANK	COMPLETED	0	2024-08-15 01:19:23.777	2.5s	2s	141.7%	25.6 MB	1.4 GB	467.7 MB	450.2 MB
4	57/137c65	19602	VCF_PRE_PROCESS	COMPLETED	0	2024-08-15 01:19:26.272	16.6s	16.2s	96.7%	23.9 MB	618.9 MB	136.5 MB	106.5 MB
5	b4/7842ed	19960	ANNOT_ENSMBLE	COMPLETED	0	2024-08-15 01:19:42.907	7.1s	6.6s	115.1%	360.2 MB	2.1 GB	147.8 MB	120.8 MB
6	ee/35b6ab	20057	REMOVE_MITO_AND_UNKOWN_CHR	COMPLETED	0	2024-08-15 01:19:49.998	3.6s	3.1s	73.3%	19.4 MB	613.2 MB	70.2 MB	49.8 MB
7	a5/dedfc2	20103	TO_GENE_SYM	COMPLETED	0	2024-08-15 01:19:53.555	3.4s	2.9s	130.1%	41.2 MB	1.7 GB	197.5 MB	171.4 MB
8	73/d98e9b	20149	FILTER_BED	COMPLETED	0	2024-08-15 01:19:56.940	678ms	87ms	117.4%	6.4 MB	304.1 MB	10.3 MB	10.3 MB
9	4e/d8da12	20175	PHRANK_SCORING	COMPLETED	0	2024-08-15 01:19:57.621	1.4s	899ms	101.5%	40.2 MB	633.9 MB	2.9 MB	125.7 KB
10	7b/bb38e4	20215	FILTER_PROBAND	COMPLETED	0	2024-08-15 01:19:59.019	14.5s	14s	94.9%	41 MB	637.8 MB	171.8 MB	48.2 MB
11	6d/40f119	20322	VEP_ANNOTATE	COMPLETED	0	2024-08-15 01:20:13.511	4m 25s	4m 24s	885.5%	818.6 MB	2.9 GB	53.5 GB	278.7 MB
12	74/aa3aab	21784	FEATURE_ENGINEERING_PART1	COMPLETED	0	2024-08-15 01:24:38.216	18m 36s	18m 35s	106.0%	679.2 MB	1.8 GB	10.2 GB	348.3 MB
13	d0/3ed0c2	23574	FEATURE_ENGINEERING_PART2	COMPLETED	0	2024-08-15 01:43:13.859	55s	54.5s	160.2%	3.3 GB	4.9 GB	1.4 GB	6.9 MB
14	c6/eb1b69	23667	PREDICTION	COMPLETED	0	2024-08-15 01:44:08.858	18.8s	18.2s	175.1%	693.8 MB	3.2 GB	382.6 MB	94.7 MB

@jylee-bcm
Copy link
Contributor

If this job reduced the memory consumption of the feature engineering 1, it's remarkable! can you post the previous version as well?

@hyunhwan-bcm
Copy link
Contributor Author

hyunhwan-bcm commented Aug 15, 2024 •
edited
Loading

If this job reduced the memory consumption of the feature engineering 1, it's remarkable! can you post the previous version as well?

task_id	hash	native_id	name	status	exit	submit	duration	realtime	%cpu	peak_rss	peak_vmem	rchar	wchar
2	d8/860449	43384	HPO_SIM	COMPLETED	0	2024-08-15 11:29:14.943	20.1s	18.1s	93.8%	874.4 MB	1.6 GB	119.3 MB	72.5 MB
1	a6/b51c2f	43518	INDEX_VCF	COMPLETED	0	2024-08-15 11:29:35.009	4s	3.5s	121.6%	16.3 MB	889.1 MB	261.4 MB	221.6 MB
3	ff/91956c	43560	ANNOT_PHRANK	COMPLETED	0	2024-08-15 11:29:39.024	2.6s	2.1s	141.8%	25.6 MB	1.4 GB	467.7 MB	450.2 MB
4	7e/b99085	43592	VCF_PRE_PROCESS	COMPLETED	0	2024-08-15 11:29:41.608	16.5s	16s	96.5%	23.7 MB	618.9 MB	136.5 MB	106.5 MB
5	3a/7a5f67	43699	ANNOT_ENSMBLE	COMPLETED	0	2024-08-15 11:29:58.134	7.1s	6.6s	115.3%	360.2 MB	2.1 GB	147.8 MB	120.8 MB
6	48/41f612	43764	REMOVE_MITO_AND_UNKOWN_CHR	COMPLETED	0	2024-08-15 11:30:05.216	3.5s	3.1s	72.3%	19.2 MB	613.2 MB	70.2 MB	49.8 MB
7	a6/3b73bd	43807	TO_GENE_SYM	COMPLETED	0	2024-08-15 11:30:08.715	3.4s	2.9s	130.1%	41.2 MB	1.7 GB	197.5 MB	171.4 MB
8	7e/7aa3e2	43845	FILTER_BED	COMPLETED	0	2024-08-15 11:30:12.072	648ms	77ms	115.0%	6.4 MB	304.1 MB	10.3 MB	10.3 MB
9	6c/6a2851	43871	PHRANK_SCORING	COMPLETED	0	2024-08-15 11:30:12.724	1.3s	884ms	100.8%	39.2 MB	633.3 MB	2.9 MB	125.7 KB
10	86/e0a7bf	43911	FILTER_PROBAND	COMPLETED	0	2024-08-15 11:30:14.077	14.4s	13.9s	94.8%	41 MB	637.8 MB	171.8 MB	48.2 MB
11	bb/1990fe	44018	VEP_ANNOTATE	COMPLETED	0	2024-08-15 11:30:28.462	4m 17s	4m 15s	881.6%	1 GB	3.2 GB	53.5 GB	278.7 MB
12	b3/a1f2c1	45373	FEATURE_ENGINEERING_PART1	COMPLETED	0	2024-08-15 11:34:45.218	17m 12s	17m 11s	100.0%	1.8 GB	3 GB	637.5 MB	146.1 MB
13	30/7dd972	50565	FEATURE_ENGINEERING_PART2	COMPLETED	0	2024-08-15 11:51:56.806	54.2s	53.7s	160.5%	3.3 GB	4.9 GB	1.4 GB	6.9 MB
14	72/31bb85	50888	PREDICTION	COMPLETED	0	2024-08-15 11:52:51.065	18.7s	18.2s	175.3%	692.4 MB	3.2 GB	382.5 MB	94.6 MB

@jylee-bcm
Copy link
Contributor

Thanks. It seems memory usage is reduced to half!

jylee-bcm
jylee-bcm approved these changes Aug 15, 2024
View reviewed changes
main.nf
@@ -259,7 +283,8 @@ process VEP_ANNOTATE {
path vep_idx

output:
path "${params.run_id}-vep.txt"
path "${vcf.baseName}-vep.txt", emit: vep_output
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good work!

main.nf
VEP_ANNOTATE(
FILTER_PROBAND.out,
SPLIT_VCF_BY_CHROMOSOME.out.chr_vcfs.flatten(),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good!

@hyunhwan-bcm
Copy link
Contributor Author

Testing on a WES sample, changed --buffer_size for vep for 10 to avoid an error - it might be macOS specific, I haven't seen any error like that.

nextflow run ../../AI_MARRVEL/main.nf \
  --ref_dir /Volumes/HWAN-T7/data-dependency \
  --input_vcf $PWD/data/679529/679529.hard-filtered.vcf.gz --input_hpo $PWD/data/679529_HPOs.txt \
  --outdir output/679529_new/ -with-docker chaozhongliu/aim-lite --no_filter_exonic -process.cpus 4 -process.memory 4.GB -with-trace trace-new.txt

[d5/1d7ef4] INDEX_VCF                      | 1 of 1 ✔
[45/20ffe0] VCF_PRE_PROCESS                | 1 of 1 ✔
[a2/586e50] ANNOT_PHRANK                   | 1 of 1 ✔
[09/e14e22] ANNOT_ENSMBLE                  | 1 of 1 ✔
[9c/13bbab] TO_GENE_SYM                    | 1 of 1 ✔
[22/f8d5fd] PHRANK_SCORING                 | 1 of 1 ✔
[5a/b89892] HPO_SIM                        | 1 of 1 ✔
[6d/74b5a7] REMOVE_MITO_AND_UNKOWN_CHR     | 1 of 1 ✔
[3d/b43bc4] FILTER_BED                     | 1 of 1 ✔
[cd/d1283f] FILTER_PROBAND                 | 1 of 1 ✔
[9a/6ef5bb] SPLIT_VCF_BY_CHROMOSOME        | 1 of 1 ✔
[69/9765f8] VEP_ANNOTATE (19)              | 24 of 24 ✔
[cb/6bcd19] FEATURE_ENGINEERING_PART1 (24) | 24 of 24 ✔
[86/9c0dc5] FEATURE_ENGINEERING_PART2 (24) | 24 of 24 ✔
[e2/09ffb1] MERGE_RESULTS                  | 1 of 1 ✔
[a4/fd5050] PREDICTION                     | 1 of 1 ✔
Completed at: 15-Aug-2024 16:55:21
Duration    : 13m 36s
CPU hours   : 2.4
Succeeded   : 85

task_id	hash	native_id	name	status	exit	submit	duration	realtime	%cpu	peak_rss	peak_vmem	rchar	wchar
2	d5/1d7ef4	55188	INDEX_VCF	COMPLETED	0	2024-08-15 16:41:45.558	4.4s	3.8s	122.6%	16 MB	889.1 MB	261.4 MB	221.6 MB
3	a2/586e50	55265	ANNOT_PHRANK	COMPLETED	0	2024-08-15 16:41:49.965	3.1s	2.6s	146.6%	25.3 MB	1.4 GB	467.7 MB	450.2 MB
5	09/e14e22	55321	ANNOT_ENSMBLE	COMPLETED	0	2024-08-15 16:41:53.100	8.1s	7.5s	119.7%	360 MB	2.1 GB	147.8 MB	120.8 MB
6	9c/13bbab	55387	TO_GENE_SYM	COMPLETED	0	2024-08-15 16:42:01.225	3.7s	3.2s	128.9%	40.9 MB	1.7 GB	197.5 MB	171.4 MB
7	22/f8d5fd	55435	PHRANK_SCORING	COMPLETED	0	2024-08-15 16:42:04.968	1.5s	1s	99.2%	38.1 MB	632.1 MB	2.9 MB	125.6 KB
1	5a/b89892	55195	HPO_SIM	COMPLETED	0	2024-08-15 16:41:45.565	21.5s	20.9s	94.7%	874.3 MB	1.6 GB	119.3 MB	72.5 MB
4	45/20ffe0	55267	VCF_PRE_PROCESS	COMPLETED	0	2024-08-15 16:41:49.969	19.5s	18.9s	94.8%	23.7 MB	1.2 GB	136.5 MB	106.5 MB
8	6d/74b5a7	55496	REMOVE_MITO_AND_UNKOWN_CHR	COMPLETED	0	2024-08-15 16:42:09.440	3.9s	3.4s	73.3%	19.2 MB	613.2 MB	70.2 MB	49.8 MB
9	3d/b43bc4	55553	FILTER_BED	COMPLETED	0	2024-08-15 16:42:13.377	683ms	76ms	120.7%	6.5 MB	304.1 MB	10.3 MB	10.3 MB
10	cd/d1283f	55582	FILTER_PROBAND	COMPLETED	0	2024-08-15 16:42:14.074	14.7s	14.2s	94.9%	41 MB	637.8 MB	171.8 MB	48.2 MB
11	9a/6ef5bb	55707	SPLIT_VCF_BY_CHROMOSOME	COMPLETED	0	2024-08-15 16:42:28.830	1.4s	944ms	101.1%	6.4 MB	304 MB	7.7 MB	1.4 MB
21	34/9059ad	55793	VEP_ANNOTATE (10)	COMPLETED	0	2024-08-15 16:42:30.321	28.2s	26.1s	79.1%	560.2 MB	1.9 GB	343.3 MB	2.7 MB
16	2e/0043ad	55826	VEP_ANNOTATE (5)	COMPLETED	0	2024-08-15 16:42:30.334	34.9s	33.4s	78.0%	550.3 MB	1.9 GB	468.4 MB	3.1 MB
17	5e/92f764	55729	VEP_ANNOTATE (6)	COMPLETED	0	2024-08-15 16:42:30.291	45s	44s	92.7%	580.5 MB	2 GB	658.6 MB	6.1 MB
18	ba/7ccad1	55813	VEP_ANNOTATE (7)	COMPLETED	0	2024-08-15 16:42:30.327	48.6s	47.4s	96.6%	602.2 MB	2 GB	761 MB	6.5 MB
13	8d/0fac94	55774	VEP_ANNOTATE (2)	COMPLETED	0	2024-08-15 16:42:30.312	1m 8s	1m 7s	94.8%	603.5 MB	2.3 GB	1 GB	7.1 MB
26	e6/0e0788	56695	VEP_ANNOTATE (15)	COMPLETED	0	2024-08-15 16:43:18.937	44.6s	43.3s	92.1%	656.8 MB	2 GB	627 MB	5.6 MB
19	e4/a6d2dc	55761	VEP_ANNOTATE (8)	COMPLETED	0	2024-08-15 16:42:30.307	1m 34s	1m 33s	95.5%	755.1 MB	2.2 GB	1.5 GB	11.2 MB
20	77/389baa	55746	VEP_ANNOTATE (9)	COMPLETED	0	2024-08-15 16:42:30.301	1m 35s	1m 34s	99.0%	738.5 MB	2.3 GB	1.5 GB	12.4 MB
15	e1/96898b	56612	VEP_ANNOTATE (4)	COMPLETED	0	2024-08-15 16:43:15.280	1m 10s	1m 9s	95.8%	608.8 MB	2 GB	1.1 GB	10.2 MB
14	e2/0cb300	56499	VEP_ANNOTATE (3)	COMPLETED	0	2024-08-15 16:43:05.260	1m 21s	1m 19s	103.3%	692.3 MB	2.2 GB	1.3 GB	12.3 MB
23	05/b68e06	55736	VEP_ANNOTATE (12)	COMPLETED	0	2024-08-15 16:42:30.297	2m	1m 59s	98.1%	518.5 MB	2.1 GB	2 GB	16.5 MB
28	e1/48457e	56847	VEP_ANNOTATE (17)	COMPLETED	0	2024-08-15 16:43:38.612	1m 5s	1m 4s	90.1%	428.5 MB	1.8 GB	1.1 GB	8.6 MB
25	c4/0fe92b	57036	VEP_ANNOTATE (14)	COMPLETED	0	2024-08-15 16:44:03.509	42s	40.6s	83.2%	523.7 MB	1.9 GB	747.9 MB	4.5 MB
24	75/d68b63	57171	VEP_ANNOTATE (13)	COMPLETED	0	2024-08-15 16:44:05.437	46.1s	45.1s	93.5%	907.5 MB	2.3 GB	705.8 MB	4.3 MB
22	f9/0efd0c	56407	VEP_ANNOTATE (11)	COMPLETED	0	2024-08-15 16:42:58.502	1m 59s	1m 58s	106.7%	662.7 MB	2.3 GB	1.9 GB	15.9 MB
35	d1/45dd89	57788	VEP_ANNOTATE (24)	COMPLETED	0	2024-08-15 16:44:51.588	9.6s	8.1s	71.0%	97.5 MB	765.6 MB	137.9 MB	516.7 KB
12	be/1dfee6	55784	VEP_ANNOTATE (1)	COMPLETED	0	2024-08-15 16:42:30.317	2m 38s	2m 36s	101.1%	624.3 MB	2.3 GB	2.5 GB	19.2 MB
27	31/92b1b9	57108	VEP_ANNOTATE (16)	COMPLETED	0	2024-08-15 16:44:04.029	1m 16s	1m 15s	99.3%	628.9 MB	2.2 GB	1.2 GB	10.3 MB
29	ec/7bfa7f	57345	VEP_ANNOTATE (18)	COMPLETED	0	2024-08-15 16:44:25.777	1m 6s	1m 5s	102.0%	497.4 MB	1.9 GB	1.1 GB	10 MB
36	b9/756c4f	57949	FEATURE_ENGINEERING_PART1 (1)	COMPLETED	0	2024-08-15 16:45:01.162	32.6s	31.3s	92.8%	647 MB	1.8 GB	313.5 MB	294.9 KB
32	7a/f15565	57698	VEP_ANNOTATE (21)	COMPLETED	0	2024-08-15 16:44:45.476	49s	47.5s	109.6%	466.4 MB	2.1 GB	858.6 MB	6.3 MB
33	05/e9471a	57496	VEP_ANNOTATE (22)	COMPLETED	0	2024-08-15 16:44:30.441	1m 6s	1m 5s	103.6%	508.4 MB	2.2 GB	1.2 GB	8.2 MB
34	db/2673db	57869	VEP_ANNOTATE (23)	COMPLETED	0	2024-08-15 16:44:57.336	41s	39.8s	107.6%	457 MB	1.8 GB	908.7 MB	6.1 MB
31	d4/3013f7	57628	VEP_ANNOTATE (20)	COMPLETED	0	2024-08-15 16:44:44.038	1m 4s	1m 3s	122.0%	560.8 MB	2.2 GB	1.5 GB	10.2 MB
37	86/8cd794	58170	FEATURE_ENGINEERING_PART1 (2)	COMPLETED	0	2024-08-15 16:45:33.748	23.4s	22.7s	105.9%	647.8 MB	1.8 GB	313.7 MB	310.4 KB
30	69/9765f8	57410	VEP_ANNOTATE (19)	COMPLETED	0	2024-08-15 16:44:26.064	1m 48s	1m 47s	137.4%	604.3 MB	2.3 GB	3.5 GB	24.4 MB
38	1a/2027c7	58199	FEATURE_ENGINEERING_PART1 (3)	COMPLETED	0	2024-08-15 16:45:34.522	41.2s	40.4s	102.2%	648 MB	1.8 GB	315.1 MB	681.2 KB
39	95/57a600	58351	FEATURE_ENGINEERING_PART1 (4)	COMPLETED	0	2024-08-15 16:45:57.145	40.7s	40.2s	104.7%	648.2 MB	1.8 GB	315.3 MB	735.5 KB
41	13/a81a63	58502	FEATURE_ENGINEERING_PART1 (6)	COMPLETED	0	2024-08-15 16:46:15.730	32.3s	31.8s	106.9%	648.3 MB	1.8 GB	314.9 MB	587.7 KB
40	b2/eadfc5	58463	FEATURE_ENGINEERING_PART1 (5)	COMPLETED	0	2024-08-15 16:46:13.687	43.4s	43s	104.8%	648.4 MB	1.8 GB	315.5 MB	795.1 KB
42	85/604903	58640	FEATURE_ENGINEERING_PART1 (7)	COMPLETED	0	2024-08-15 16:46:37.834	58.1s	57.6s	103.7%	648.2 MB	1.8 GB	317.2 MB	1 MB
44	c9/3411ad	58797	FEATURE_ENGINEERING_PART1 (9)	COMPLETED	0	2024-08-15 16:46:57.139	58.8s	58.3s	103.3%	648 MB	1.8 GB	317 MB	1.1 MB
43	64/ec1937	58718	FEATURE_ENGINEERING_PART1 (8)	COMPLETED	0	2024-08-15 16:46:47.984	1m 11s	1m 11s	102.5%	649.6 MB	1.8 GB	318 MB	1.3 MB
45	11/f9a865	59017	FEATURE_ENGINEERING_PART1 (10)	COMPLETED	0	2024-08-15 16:47:35.942	1m 11s	1m 11s	103.0%	649.6 MB	1.8 GB	318 MB	1.3 MB
47	50/a2311c	59190	FEATURE_ENGINEERING_PART1 (12)	COMPLETED	0	2024-08-15 16:47:59.312	48.3s	47.7s	104.1%	648.2 MB	1.8 GB	316.2 MB	875 KB
49	c5/325951	59490	FEATURE_ENGINEERING_PART1 (14)	COMPLETED	0	2024-08-15 16:48:47.569	24.5s	23.9s	109.1%	648.4 MB	1.8 GB	314.2 MB	441.8 KB
48	d6/883e00	59451	FEATURE_ENGINEERING_PART1 (13)	COMPLETED	0	2024-08-15 16:48:47.244	25.9s	25.4s	107.9%	648.2 MB	1.8 GB	314.3 MB	440 KB
51	2d/ca0ce6	59674	FEATURE_ENGINEERING_PART1 (16)	COMPLETED	0	2024-08-15 16:49:13.140	6s	5.4s	139.9%	648.2 MB	1.8 GB	312.5 MB	32.7 KB
46	1d/f4daed	59140	FEATURE_ENGINEERING_PART1 (11)	COMPLETED	0	2024-08-15 16:47:55.964	1m 32s	1m 31s	102.3%	675.7 MB	1.8 GB	319.8 MB	1.7 MB
53	c6/939101	59802	FEATURE_ENGINEERING_PART1 (18)	COMPLETED	0	2024-08-15 16:49:27.797	59.7s	59.1s	103.5%	648.1 MB	1.8 GB	317 MB	1.1 MB
50	a1/577047	59646	FEATURE_ENGINEERING_PART1 (15)	COMPLETED	0	2024-08-15 16:49:12.066	1m 29s	1m 28s	102.3%	675.6 MB	1.8 GB	319.7 MB	1.6 MB
55	ea/cba3e7	60230	FEATURE_ENGINEERING_PART2 (1)	COMPLETED	0	2024-08-15 16:50:40.739	13.3s	12.8s	360.3%	1.7 GB	3.3 GB	1.4 GB	89.9 KB
52	31/5b6426	59733	FEATURE_ENGINEERING_PART1 (17)	COMPLETED	0	2024-08-15 16:49:19.139	1m 45s	1m 44s	102.1%	671.9 MB	1.8 GB	321.1 MB	1.9 MB
54	55/3e5f38	60130	FEATURE_ENGINEERING_PART1 (19)	COMPLETED	0	2024-08-15 16:50:27.467	59.5s	59s	103.8%	648.3 MB	1.8 GB	316.9 MB	1 MB
56	88/2ba6d3	60323	FEATURE_ENGINEERING_PART1 (20)	COMPLETED	0	2024-08-15 16:50:54.043	39.4s	38.9s	105.4%	648.2 MB	1.8 GB	315.2 MB	700.4 KB
57	d4/05856d	60408	FEATURE_ENGINEERING_PART1 (21)	COMPLETED	0	2024-08-15 16:51:03.901	47.8s	47.3s	104.7%	648.4 MB	1.8 GB	316 MB	830 KB
60	db/0dc0d5	60736	FEATURE_ENGINEERING_PART2 (2)	COMPLETED	0	2024-08-15 16:51:51.743	12.2s	11.6s	328.6%	1.7 GB	3.4 GB	1.4 GB	123.6 KB
58	51/05e05f	60548	FEATURE_ENGINEERING_PART1 (22)	COMPLETED	0	2024-08-15 16:51:27.009	39s	38.5s	105.5%	648.4 MB	1.8 GB	315.1 MB	699.5 KB
62	89/d461fd	60863	FEATURE_ENGINEERING_PART2 (3)	COMPLETED	0	2024-08-15 16:52:06.043	12.6s	12.1s	322.1%	1.7 GB	3.3 GB	1.4 GB	164.6 KB
63	67/8ba33f	60966	FEATURE_ENGINEERING_PART2 (4)	COMPLETED	0	2024-08-15 16:52:18.606	11.8s	11.2s	351.6%	1.7 GB	3.3 GB	1.4 GB	188.8 KB
59	c6/29138e	60617	FEATURE_ENGINEERING_PART1 (23)	COMPLETED	0	2024-08-15 16:51:33.414	1m	59.7s	103.5%	648.1 MB	1.8 GB	316.8 MB	1001.9 KB
64	11/5c8a86	61051	FEATURE_ENGINEERING_PART2 (5)	COMPLETED	0	2024-08-15 16:52:30.449	11.9s	11.4s	341.3%	1.7 GB	3.3 GB	1.4 GB	154.6 KB
65	d4/06ea6a	61103	FEATURE_ENGINEERING_PART2 (6)	COMPLETED	0	2024-08-15 16:52:33.658	13.4s	12.9s	303.9%	1.7 GB	3.3 GB	1.4 GB	251.9 KB
66	7b/0ca4cc	61178	FEATURE_ENGINEERING_PART2 (7)	COMPLETED	0	2024-08-15 16:52:42.348	14.8s	14.3s	303.4%	1.7 GB	3.3 GB	1.4 GB	456.5 KB
67	65/e00b2e	61230	FEATURE_ENGINEERING_PART2 (8)	COMPLETED	0	2024-08-15 16:52:47.062	14.4s	13.9s	301.5%	1.7 GB	3.3 GB	1.4 GB	263.7 KB
68	0f/257f5c	61312	FEATURE_ENGINEERING_PART2 (9)	COMPLETED	0	2024-08-15 16:52:57.116	15.6s	15s	292.3%	1.7 GB	3.3 GB	1.4 GB	362 KB
69	9d/2f8e8c	61364	FEATURE_ENGINEERING_PART2 (10)	COMPLETED	0	2024-08-15 16:53:01.505	15.7s	15s	300.0%	1.7 GB	3.3 GB	1.4 GB	318.8 KB
70	58/93d710	61458	FEATURE_ENGINEERING_PART2 (11)	COMPLETED	0	2024-08-15 16:53:12.721	14.6s	13.8s	319.6%	1.7 GB	3.3 GB	1.4 GB	280.1 KB
71	b1/82a46c	61510	FEATURE_ENGINEERING_PART2 (12)	COMPLETED	0	2024-08-15 16:53:17.249	14.4s	13.6s	305.7%	1.7 GB	3.3 GB	1.4 GB	170.8 KB
72	49/55c0a2	61593	FEATURE_ENGINEERING_PART2 (13)	COMPLETED	0	2024-08-15 16:53:27.278	13.4s	12.6s	331.1%	1.7 GB	3.3 GB	1.4 GB	209.2 KB
73	94/d88a60	61645	FEATURE_ENGINEERING_PART2 (14)	COMPLETED	0	2024-08-15 16:53:31.671	12.8s	12.3s	322.5%	1.7 GB	3.3 GB	1.4 GB	38.4 KB
74	3a/1646d3	61720	FEATURE_ENGINEERING_PART2 (15)	COMPLETED	0	2024-08-15 16:53:40.684	17.5s	16.6s	268.2%	1.7 GB	3.3 GB	1.4 GB	511.6 KB
75	34/64f898	61772	FEATURE_ENGINEERING_PART2 (16)	COMPLETED	0	2024-08-15 16:53:44.466	15s	14.4s	299.8%	1.7 GB	3.3 GB	1.4 GB	303.4 KB
76	4f/1f55c1	61875	FEATURE_ENGINEERING_PART2 (17)	COMPLETED	0	2024-08-15 16:53:58.192	15.6s	14.5s	308.4%	1.7 GB	3.3 GB	1.4 GB	426.7 KB
77	c1/b5c294	61917	FEATURE_ENGINEERING_PART2 (18)	COMPLETED	0	2024-08-15 16:53:59.455	17.3s	16.8s	271.5%	1.7 GB	3.3 GB	1.4 GB	608.6 KB
78	9d/73a423	62288	FEATURE_ENGINEERING_PART2 (19)	COMPLETED	0	2024-08-15 16:54:13.779	15.2s	13.8s	317.7%	1.7 GB	3.3 GB	1.4 GB	278.4 KB
61	cb/6bcd19	60819	FEATURE_ENGINEERING_PART1 (24)	COMPLETED	0	2024-08-15 16:52:03.981	2m 27s	2m 27s	101.4%	678.6 MB	1.8 GB	323.3 MB	2.6 MB
79	94/5f3fdf	62331	FEATURE_ENGINEERING_PART2 (20)	COMPLETED	0	2024-08-15 16:54:16.744	14.7s	14s	313.3%	1.7 GB	3.3 GB	1.4 GB	222.5 KB
80	4e/c82ccf	62423	FEATURE_ENGINEERING_PART2 (21)	COMPLETED	0	2024-08-15 16:54:28.961	15s	13.5s	331.3%	1.7 GB	3.3 GB	1.4 GB	283.2 KB
81	74/8a947e	62465	FEATURE_ENGINEERING_PART2 (22)	COMPLETED	0	2024-08-15 16:54:31.217	17.2s	16.6s	256.5%	1.7 GB	3.3 GB	1.4 GB	236.7 KB
82	5e/076d20	62500	FEATURE_ENGINEERING_PART2 (23)	COMPLETED	0	2024-08-15 16:54:31.424	17.7s	17.1s	230.9%	1.7 GB	3.3 GB	1.4 GB	346.4 KB
83	86/9c0dc5	62596	FEATURE_ENGINEERING_PART2 (24)	COMPLETED	0	2024-08-15 16:54:43.931	16.3s	15.4s	298.5%	1.7 GB	3.3 GB	1.4 GB	804.6 KB
84	e2/09ffb1	62717	MERGE_RESULTS	COMPLETED	0	2024-08-15 16:55:00.221	1s	387ms	112.9%	6.4 MB	304 MB	30.3 MB	30.3 MB
85	a4/fd5050	62867	PREDICTION	COMPLETED	0	2024-08-15 16:55:01.239	19.8s	19.3s	171.0%	733.1 MB	3.2 GB	382.5 MB	94.5 MB

@hyunhwan-bcm hyunhwan-bcm changed the title vep output separation by chromosome for feature engineering part 1 job separation by chromosome Aug 15, 2024
@hyunhwan-bcm
Copy link
Contributor Author

testing on WGS data

@hyunhwan-bcm hyunhwan-bcm merged commit a876e49 into nextflow_conversion Aug 19, 2024
2 checks passed
@hyunhwan-bcm hyunhwan-bcm mentioned this pull request Aug 19, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jylee-bcm jylee-bcm jylee-bcm approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@hyunhwan-bcm @jylee-bcm