Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Incomplete Pages #31

Merged
merged 58 commits into from
Jun 1, 2021
Merged

Fix Incomplete Pages #31

merged 58 commits into from
Jun 1, 2021

Conversation

lukasturcani
Copy link
Owner

@lukasturcani lukasturcani commented May 31, 2021

Previously, when a database was organized such that there were
molecules in the molecules collection, which were not found in the
position matrix collection, stk-vis would show an incomplete page.
This commit allows stk-vis show full pages even when a database is
organized in this way.

Essentially, what was happening was that the requests to the MongoDB
database queried a fixed number of molecules from the molecules
collection and then removed in the invalid ones, for example those
missing a position matrix. This meant that molecules found in the
molecules collection which did not have a position matrix in the
position matrix collection were removed.

In order to fix this, I am using aggregate queries for all requests.
The aggregate query always requests the molecules as well as the
position matrices in one go, and if the returned entry does not have
position matrix, it is filtered out as part of the MongoDB query. This
means that when I ask MongoDB for 34 molecules, I am guaranteed that
they will all have the necessary entries in the various collections.

As part of these changes I've removed the Mongo.Entry type. I was
using the Mongo.Entry type as a generic black box for the entries
returned by a MongoDB query, before any typing / validation is done.
I've found that this type is not very useful. Depending on what kind
of query I run, and on what collection, I get entries of various types
and its useful to have functions which convert from these types into
a Request.Molecule. Because of this I've made the Mongo.Cursor type
take a type parameter, which is the unvalidated entry type. The exact
type will depend on the individual request, because when the request
is done against a molecule collection (for example in the unsorted
requests), the resulting entries will have a different schema then when
it is done against a value collection (for example in the sorted
requests). I've made which functions convert from the unvalidated
entry returned by the MongoDB queries to a Requests.MoleculeEntry
type, which is a sort of validated MongoDB molecule entry, with a fixed
schema. This can then be converted to a Requests.Molecule. I think
Requests.MoleculeEntry can be removed in a future commit and the
conversion functions should be able to return Requests.Molecule
directly. The advantage of using Requests.MoleculeEntry is that the
type can be created in JavaScript, which I assume is faster and a
little bit easier at present.

Because I'm using aggregate MongoDB queries which immediately query
for the position matrix, I don't need to do any more data queries
against the position matrix collections. So I've removed all the
Requests.PositionMatrix code, because it is no longer needed.
Similarly, because sorted requests now return all the molecular
data, I don't need the Requests.SortedCollection code.
Requests.SortedCollection was used to hold the state between
getting the value from the sorted collection and getting back the
data query from the molecule collection. Since now one query returns
all the data necessary to make a Requests.MoleculeEntry, there is
no need for this intermediate data representation.

@lukasturcani lukasturcani changed the title Lukas/fix queries Fix Incomplete Pages Jun 1, 2021
@lukasturcani lukasturcani marked this pull request as ready for review June 1, 2021 19:01
@lukasturcani lukasturcani merged commit 8d27aea into master Jun 1, 2021
@lukasturcani lukasturcani deleted the lukas/fix-queries branch June 1, 2021 19:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant