[ALS-4617] Performance improvements for GIC #71
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Luke-Sikina
commented
Jul 24, 2023
| @@ -180,7 +184,7 @@ public String toString() { | |||
| break; | |||
| default: | |||
| //no logic here; all enum values should be present above | |||
| System.out.println("Formatting not supported for type " + expectedResultType); | |||
| log.warn("Formatting not supported for type {}", expectedResultType); | |||
There was a problem hiding this comment.
This was showing up in the logs, so I switched it to a proper log statement.
| @@ -627,7 +627,7 @@ public PhenoCube<?> load(String key) throws Exception { | |||
| inStream.close(); | |||
| return ret; | |||
| }else { | |||
| System.out.println("ColumnMeta not found for : [" + key + "]"); | |||
| log.warn("ColumnMeta not found for : [{}]", key); | |||
There was a problem hiding this comment.
This print statement was far more problematic. This was happening 1000s of times for some GIC queries. Because sout is synchronized, this would block all the worker threads in the fork join pool, grinding HPDS to a halt. This in turn caused timeouts for simpler requests, effectively breaking everything.
processing/src/main/java/edu/harvard/hms/dbmi/avillach/hpds/processing/QueryProcessor.java
Show resolved
Hide resolved
Comment on lines
-67
to
+74
| int columnCount = paths.size() + 1; | ||
|
|
||
| List<ColumnMeta> columns = paths.stream() | ||
| List<ColumnMeta> columns = query.getFields().stream() | ||
| .map(abstractProcessor.getDictionary()::get) | ||
| .filter(Objects::nonNull) | ||
| .collect(Collectors.toList()); | ||
| List<String> paths = columns.stream() | ||
| .map(ColumnMeta::getName) | ||
| .collect(Collectors.toList()); | ||
| int columnCount = paths.size() + 1; |
There was a problem hiding this comment.
We weren't accounting for non-normalized concept paths, which GIC has in spades. This filters out concept paths not present in this institution prior to the query.
srpiatt
approved these changes
Jul 24, 2023
- Made QueryProcessor filter columns that DNE - Paralellized QueryProcessor::runQuery - Switch `sout` logging to standard slf4j logging - Faster - No blocking threads due to synchronized methods
Luke-Sikina
force-pushed
the
perf-fixes-bch
branch
from
cd2a570 to
f984570
Compare
Luke-Sikina
added
enhancement
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
soutlogging to standard slf4j logging