Parameter Tuning Code Integration #193
Conversation
… what idea to use
config/egfr.yaml
Outdated
There was a problem hiding this comment.
should I add the gold standard to this?
test/evaluate/test_evaluate.py
Outdated
| edge_freq.to_csv(OUT_DIR + 'node-ensemble.csv', sep="\t", index=False) | ||
| assert filecmp.cmp(OUT_DIR + 'node-ensemble.csv', EXPECT_DIR + 'expected-node-ensemble.csv', shallow=False) | ||
|
|
||
| def test_precision_recal_curve_ensemble_nodes(self): |
There was a problem hiding this comment.
I don't know how else to test the ensemble node outputs other than looking at the image
| # adds evaluation per algorithm per dataset-goldstandard pair | ||
| # evalution per algortihm will not run unless ml include and ml aggregate_per_algorithm is set to true | ||
| aggregate_per_algorithm: true | ||
| # TODO: should we decouple parts of eval that involve ml |
There was a problem hiding this comment.
Lots of coupling happening now. I put in a solution for now in config.py, but is it worth separating the functions into their own true/ false?
There was a problem hiding this comment.
Maybe deal with some of the coupling by giving warnings and stopping the flow rather than silently shutting things off
spras/analysis/ml.py
Outdated
| @@ -142,8 +142,14 @@ def pca(dataframe: pd.DataFrame, output_png: str, output_var: str, output_coord: | |||
| if not isinstance(labels, bool): | |||
| raise ValueError(f"labels={labels} must be True or False") | |||
|
|
|||
| scaler = StandardScaler() | |||
| #TODO: MinMaxScaler changes nothing about the data | |||
There was a problem hiding this comment.
I don't know if it is better to use StandardScalar or MinMaxScalar for the binary data
spras/evaluation.py
Outdated
| for file in file_paths: | ||
| df = pd.read_table(file, sep="\t", header=0, usecols=["Node1", "Node2"]) | ||
| # TODO: do we want to include the pathways that are empty for evaluation / in the pr_df? |
There was a problem hiding this comment.
Currently the code will add a precision and recall for empty pathways. Is that something we shouldn't include?
Snakefile
Outdated
| final_input.extend(expand('{out_dir}{sep}{dataset_gold_standard_pair}-eval{sep}precision-recall-per-pathway.png',out_dir=out_dir,sep=SEP,dataset_gold_standard_pair=dataset_gold_standard_pairs)) | ||
| final_input.extend(expand('{out_dir}{sep}{dataset_gold_standard_pair}-eval{sep}precision-recall-pca-chosen-pathway.txt',out_dir=out_dir,sep=SEP,dataset_gold_standard_pair=dataset_gold_standard_pairs)) | ||
| final_input.extend(expand('{out_dir}{sep}{dataset_gold_standard_pair}-eval{sep}precision-recall-curve-ensemble-nodes.png',out_dir=out_dir,sep=SEP,dataset_gold_standard_pair=dataset_gold_standard_pairs,algorithm_params=algorithms_with_params)) | ||
| # TODO: should we provide the node ensemble frequencies |
There was a problem hiding this comment.
Since we are already calculating the node ensembles, should we give it to the user?
|
Note for self: Try seeing if it makes more sense to separate the parameter tuning and evaluation code into their classes for organization |
|
I think I need to redo the idea for using ensembling for parameter tuning. Currently the code will take in the ensemble file and build a node ensemble file by processing an ensemble of edge frequencies to identify the highest frequency associated with each node (y_scores). This is then compared to the node gold standard (y_trues). Then these are plotted (with no point labels) to a graph showing the PRC between the nodes in the output pathways and the gold standard, as well as the average PR between all the nodes. I don’t think this can be used to do ensemble parameter tuning the way I think I would do it. It could be used to help parameter tune though to get a better grid search or for evaluation in general.
|
There was a problem hiding this comment.
I think I need to redo the idea for using ensembling for parameter tuning.
Is there a way to break this huge pull request into smaller parts? Currently it combines evaluation, parameter tuning, ensembling, etc. That is making it hard to coordinate all of those decisions and also review the big pull request. I'm wondering if we can start small and merge in a subset of the file changes to lock in some progress, like the evaluation code.
If it is all interdependent, we'll have to deal with that and proceed as is.
input/gs-egfr.txt
Outdated
There was a problem hiding this comment.
Can you please add some notes about how you prepared this file and link it back to the related issue? We'll want to be able to track that later.
…ameter tuning methods
No description provided.