rewrite riskofbias exports #921
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rabstejnek
approved these changes
Oct 12, 2023
rabstejnek
added a commit
that referenced
this pull request
Jul 2, 2024
* Epi export rewrite (#911) * preliminary epi rewrite * changes * changes * changes * fix * added back something accidentally deleted * moved code * fix test * remove old stuff * cleanup: * use vectorized timestamp conversion * minor formatting --------- Co-authored-by: Andy Shapiro <shapiromatron@gmail.com> * rewrite riskofbias exports (#921) * preliminary epi rewrite * changes * changes * changes * fix * added back something accidentally deleted * moved code * fix test * remove old stuff * cleanup: * rewrite riskofbias exports * update naming for domain and metric --------- Co-authored-by: Daniel Rabstejnek <rabstejnek@gmail.com> * Epimeta export rewrite (#922) * preliminary epi rewrite * changes * changes * changes * fix * added back something accidentally deleted * moved code * fix test * remove old stuff * cleanup: * epimeta export rewrite * remove obsolete code * changes * merge fix * update admin site to browse data pivot by evidence type --------- Co-authored-by: Andy Shapiro <shapiromatron@gmail.com> * Invitro export rewrite (#958) * Convert invitro exports * Fix exporter datetime converter when datetime is None * Fix exports where there's no data * DTXSID in export should be None instead of useless dict if missing * add endpoint categories * Remove bloated dsstox dict from exports in favor of dtxsid --------- Co-authored-by: Andy Shapiro <shapiromatron@gmail.com> * Animal export rewrite (#961) * Made animal model exports, began configuring exporters * first export largely done, second export mostly done * Some cleanups, finished fourth animal exporter * Changes after more testing * Fix tests * Remove remaining flat_complete_* methods * Cleanup * Update sql_display to accept dict * Add TODO to comment to easily find it * Rename exporters to match the flat file exporter class names * Add safeguard for qs accessing and fix groupby side effects (#982) * two updates * move to correct place * exports-v2 - Updates from image review (#1058) * remove duplicate rename calls * add ci calculation * update treatment period calculation * add back space; fix caused too many false positives in comparison * properly handle a 500 response from the server in a data pivot * fix epi logic from plot review * fix invitro export when category id is null; remove category id from export * add unique rob columns * use same method for categories * refactor into a reusable method * remove pytest warning "Marks applied to fixtures have no effect" * refactor; write tests * add tests * rewrite animal export test using same pattern * add udf to openapi * remove extra whitespace (will change visuals; known change) --------- Co-authored-by: Daniel Rabstejnek <rabstejnek@gmail.com>
shapiromatron
added a commit
that referenced
this pull request
Sep 3, 2024
* Epi export rewrite (#911) * preliminary epi rewrite * changes * changes * changes * fix * added back something accidentally deleted * moved code * fix test * remove old stuff * cleanup: * use vectorized timestamp conversion * minor formatting --------- Co-authored-by: Andy Shapiro <shapiromatron@gmail.com> * rewrite riskofbias exports (#921) * preliminary epi rewrite * changes * changes * changes * fix * added back something accidentally deleted * moved code * fix test * remove old stuff * cleanup: * rewrite riskofbias exports * update naming for domain and metric --------- Co-authored-by: Daniel Rabstejnek <rabstejnek@gmail.com> * Epimeta export rewrite (#922) * preliminary epi rewrite * changes * changes * changes * fix * added back something accidentally deleted * moved code * fix test * remove old stuff * cleanup: * epimeta export rewrite * remove obsolete code * changes * merge fix * update admin site to browse data pivot by evidence type --------- Co-authored-by: Andy Shapiro <shapiromatron@gmail.com> * started simple assessment value exports * added more assessment fields * add helper method to filter querysets in custom api viewset actions * added assessment fields, and year filter * added order_by and pagination fields to filterset * changed format_time to accomodate pandas update * fixed filtering for df with empty values * add a few more options --------- Co-authored-by: Daniel Rabstejnek <rabstejnek@gmail.com> Co-authored-by: Danny Peterson <dannypeterson28@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rewrite DRF serializer based riskofbias exports with exports using our new QuerySet based approach. Tested on one project, we reduced the number of queries from 750+ to ~5. Looking at just db time on the debug toolbar, from 600ms to around 50ms; this doesn't include the python runtime decrease with converting to Django models and corresponding serializer objects.
I renamed a few columns in the export just as the prior names seemed incorrect, and since this export isn't used in any downstream views in HAWC, renaming should be harmless.