Geospatial search #8239

qqmyers · 2021-11-11T00:10:15Z

What this PR does / why we need it: Adds bounding box indexing in preparation for providing geosearch or other features

Which issue(s) this PR closes:

Closes #

Special notes for your reviewer:

Suggestions on how to test this:

Does this PR introduce a user interface change? If mockups are available, please link/include them here:

Is there a release notes update needed for this change?:

Additional documentation: design doc: https://docs.google.com/document/d/1aQCIM6wbAdgMNLTVSUhvuHQKzWDAvQ5XoYGbfint3P4/edit?usp=sharing

IQSS/8188-cache_version_differences

coveralls · 2021-11-11T00:17:46Z

Coverage increased (+0.008%) to 19.989% when pulling b5383f4 on GlobalDataverseCommunityConsortium:GDCC/geosearch into 11abccf on IQSS:develop.

qqmyers · 2022-07-11T17:40:48Z

Per discussion at the DCM that the improved geospatial indexing could be useful independent of adding further geospatial search/display capabilities, I'm moving this PR from draft to review-ready. Not sure what the Rocky shellspec issue is - probably worth re-running the build to see if there's any real issue.

pdurbin

@qqmyers this looks promising! Can you please merge from develop and add a release note? What about docs? I feel like we should add something...

conf/solr/8.11.1/schema.xml

pdurbin

I added a comment about testing.

pdurbin · 2022-09-12T15:38:58Z

src/main/java/edu/harvard/iq/dataverse/search/IndexServiceBean.java

+                            solrInputDocument.addField("solr_srpt", "ENVELOPE(" + westLon + "," + eastLon + "," + northLat + "," + southLat + ")");
+                        }
+                    }
+                }
            }

            for(String metadataBlockName : metadataBlocksWithValue) {


I rewatched Jim's recent "Experimenting with Geospatial indexing" talk ( https://osf.io/84pnw ) and it reminded me that I asked for an test to be added to SearchIT. I still want this and I'm happy to try to add it, if it make sense for me to jump in the code. 😄

Co-authored-by: Philip Durbin <philipdurbin@gmail.com>

qqmyers · 2022-09-15T23:03:08Z

I'm not sure it is possible to make an IT test yet. The solr query that works is as shown:

In the UI, I don't think we can submit filterqueries and with the API, you can submit the filter query part but I haven't seen how to send the pt and d parameters yet. If that is possible, an IT test would be straight forward.

Otherwise, to test one could use the solr console (dropping the jetty.host-localhost recommended for our solr install to make it accessible, and possibly doing an aws ec2 authorize-security-group-ingress --group-id <id> --protocol tcp --port 8983 --cidr <your ip>/32 to let a browser access the console.)

…mmunityConsortium/dataverse.git into GDCC/geosearch

pdurbin · 2022-10-26T21:43:22Z

I added some error checking and updated the docs based on how the code is now.

I talked to @atrisovic about this feature and she immediately asked "Can you sort by distance?" Good question! 😄 I haven't looked into this but maybe? See https://solr.apache.org/guide/8_11/spatial-search.html#distance-sorting-or-boosting-function-queries for sort=geodist() asc. I'd need more data to test with.

I'm also ok with this being API only for now. That's how I wrote it up in the User Guide: https://dataverse-guide--8239.org.readthedocs.build/en/8239/user/find-use-data.html#geospatial-search

I'm ready for review. We could keep hacking but there's lots of other stuff to do as well. 😄 I think this is an excellent toehold into geospatial search!

scolapasta · 2022-10-31T19:28:45Z

Moved back to "This Sprint" so that @pdurbin's recent changes can be reviewed. (@pdurbin reviewed the initial PR and was fine with the changes Jim made.)

sekmiller · 2022-11-16T15:02:50Z

src/test/java/edu/harvard/iq/dataverse/api/SearchIT.java

+        junkRadius.prettyPrint();
+        junkRadius.then().assertThat()
+                .statusCode(BAD_REQUEST.getStatusCode())
+                .body("message", CoreMatchers.equalTo("Non-number radius supplied."));



Should we add a test that's actually a valid lat, long and radius?
or a valid lat with a non-numeric long, etc.?

In 364e347 I added a test for if lat/long is too big. It turns out Solr provides a pretty useful error, which was already being bubbled up.

I also took a swing at sending a huge value like "99999999999999999999999999999999" to geo_radius but Solr didn't give an error. I dunno, we already test that this number is positive. I say we let the use send bigger and bigger numbers (these are in kilometers). Eventually, they'll figure out they just aren't going to get a geospatial hit. 😄

sekmiller · 2022-11-16T15:08:47Z

src/main/java/edu/harvard/iq/dataverse/api/Search.java

+                if (geoRadius != null && geoPoint == null) {
+                    return error(Response.Status.BAD_REQUEST, "If you supply geo_radius you must also supply geo_point.");
+                }
+
            } catch (Exception ex) {
                return error(Response.Status.BAD_REQUEST, ex.getLocalizedMessage());
            }


Do we want to catch a nonnumeric exception here and give more feedback?

Well, getGeoPoint and getGeoRadius will already throw exceptions with feedback about non-numeric values.

For example, if the user does this:

/api/search?q=*&geo_point=41.9580775,-70.6621063&geo_radius=junk

They'll get this error:

{"status":"ERROR","message":"Non-number radius supplied."}

That said, I may not be following the use cases, the concerns. I'm happy to add more tests for these.

sekmiller

Mainly looked at Phil's most recent changes. Looks good, just a couple of suggestions/questions about testing and error handling.

pdurbin

I'm replying to review feedback.

src/main/java/edu/harvard/iq/dataverse/search/IndexServiceBean.java

pdurbin · 2022-11-21T14:00:36Z

src/main/java/edu/harvard/iq/dataverse/api/Search.java

+                if (geoRadius != null && geoPoint == null) {
+                    return error(Response.Status.BAD_REQUEST, "If you supply geo_radius you must also supply geo_point.");
+                }
+
            } catch (Exception ex) {
                return error(Response.Status.BAD_REQUEST, ex.getLocalizedMessage());
            }


Well, getGeoPoint and getGeoRadius will already throw exceptions with feedback about non-numeric values.

For example, if the user does this:

/api/search?q=*&geo_point=41.9580775,-70.6621063&geo_radius=junk

They'll get this error:

{"status":"ERROR","message":"Non-number radius supplied."}

That said, I may not be following the use cases, the concerns. I'm happy to add more tests for these.

pdurbin · 2022-11-21T14:54:14Z

src/test/java/edu/harvard/iq/dataverse/api/SearchIT.java

+        junkRadius.prettyPrint();
+        junkRadius.then().assertThat()
+                .statusCode(BAD_REQUEST.getStatusCode())
+                .body("message", CoreMatchers.equalTo("Non-number radius supplied."));



In 364e347 I added a test for if lat/long is too big. It turns out Solr provides a pretty useful error, which was already being bubbled up.

I also took a swing at sending a huge value like "99999999999999999999999999999999" to geo_radius but Solr didn't give an error. I dunno, we already test that this number is positive. I say we let the use send bigger and bigger numbers (these are in kilometers). Eventually, they'll figure out they just aren't going to get a geospatial hit. 😄

sekmiller

Looks good, thanks.

mreekie · 2023-03-30T18:53:06Z

grooming

Looking at the NetCDF deliverables.
This looks like it's a related issue that is related to active work, but was done before we were trying to tag each issue
Adding it to the backlog as cleared, and tagging it.

qqmyers added 2 commits September 9, 2021 17:47

add bonding box indexing

f74e0c2

Merge remote-tracking branch 'IQSS/develop' into

f3db9ee

IQSS/8188-cache_version_differences

qqmyers added 4 commits February 2, 2022 13:59

Merge remote-tracking branch 'IQSS/develop' into GDCC/geosearch

7b15877

Merge remote-tracking branch 'IQSS/develop' into GDCC/geosearch

d9eed2f

Merge remote-tracking branch 'IQSS/develop' into GDCC/geosearch

93a3cb6

Merge remote-tracking branch 'IQSS/develop' into GDCC/geosearch

84e9614

qqmyers marked this pull request as ready for review July 11, 2022 17:38

qqmyers added 5 commits July 28, 2022 17:59

Merge remote-tracking branch 'IQSS/develop' into GDCC/geosearch

e765a8d

Merge remote-tracking branch 'IQSS/develop' into GDCC/geosearch

d5430e4

Merge remote-tracking branch 'IQSS/develop' into GDCC/geosearch

4ec3d0f

Merge remote-tracking branch 'IQSS/develop' into GDCC/geosearch

6da3b2a

Merge remote-tracking branch 'IQSS/develop' into GDCC/geosearch

3b8c859

pdurbin self-assigned this Sep 8, 2022

pdurbin reviewed Sep 8, 2022

View reviewed changes

conf/solr/8.11.1/schema.xml Outdated Show resolved Hide resolved

conf/solr/8.11.1/schema.xml Outdated Show resolved Hide resolved

conf/solr/8.11.1/schema.xml Outdated Show resolved Hide resolved

pdurbin assigned qqmyers Sep 8, 2022

pdurbin reviewed Sep 12, 2022

View reviewed changes

qqmyers and others added 2 commits September 14, 2022 13:46

Merge remote-tracking branch 'IQSS/develop' into GDCC/geosearch

8ce67a3

Update conf/solr/8.11.1/schema.xml

1007a6b

Co-authored-by: Philip Durbin <philipdurbin@gmail.com>

qqmyers and others added 5 commits September 15, 2022 19:19

release note and addition to search doc

b903e2a

Merge branch 'GDCC/geosearch' of https://github.com/GlobalDataverseCo…

d79ba87

…mmunityConsortium/dataverse.git into GDCC/geosearch

add space

c210821

Merge branch 'develop' into GDCC/geosearch IQSS#8239

8c11b7d

Merge branch 'develop' into GDCC/geosearch IQSS#8239

de079fd

pdurbin added 2 commits October 26, 2022 15:41

add error checking for geo_point and geo_radius IQSS#8239

ff32672

update docs and release note (supported via API) IQSS#8239

6e7499e

pdurbin changed the title ~~GDCC/Geosearch indexing~~ Geospatial search Oct 26, 2022

pdurbin removed their assignment Oct 26, 2022

Merge remote-tracking branch 'IQSS/develop' into GDCC/geosearch

b5383f4

sekmiller self-assigned this Nov 15, 2022

sekmiller reviewed Nov 16, 2022

View reviewed changes

pdurbin self-assigned this Nov 18, 2022

pdurbin added 2 commits November 21, 2022 09:14

test invalid lat/long (too large) IQSS#8239

364e347

Merge branch 'develop' into GDCC/geosearch IQSS#8239

9d8332d

pdurbin reviewed Nov 21, 2022

View reviewed changes

sekmiller approved these changes Nov 22, 2022

View reviewed changes

sekmiller unassigned pdurbin and sekmiller Nov 22, 2022

kcondon self-assigned this Nov 30, 2022

kcondon merged commit b62a8a5 into IQSS:develop Nov 30, 2022

pdurbin added this to the 5.13 milestone Dec 1, 2022

jggautier mentioned this pull request Mar 2, 2023

Dataset publishing prevented or slowed by errors related to possible validation of metadata entered in Geospatial Bounding Box fields #9421

Closed

mreekie added the pm.netcdf-hdf5.d All 3 aims are currently under this deliverable label Mar 30, 2023

pdurbin mentioned this pull request May 30, 2023

Add validation for entered coordinate values in Geospatial metadata (bounding boxes, etc) #9547

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Geospatial search #8239

Geospatial search #8239

qqmyers commented Nov 11, 2021 •

edited by pdurbin

Loading

coveralls commented Nov 11, 2021 •

edited

Loading

qqmyers commented Jul 11, 2022

pdurbin left a comment

pdurbin left a comment

pdurbin Sep 12, 2022

qqmyers commented Sep 15, 2022

pdurbin commented Oct 26, 2022

scolapasta commented Oct 31, 2022

sekmiller Nov 16, 2022

pdurbin Nov 21, 2022

sekmiller Nov 16, 2022

pdurbin Nov 21, 2022

sekmiller left a comment

pdurbin left a comment

pdurbin Nov 21, 2022

pdurbin Nov 21, 2022

sekmiller left a comment

mreekie commented Mar 30, 2023

Geospatial search #8239

Geospatial search #8239

Conversation

qqmyers commented Nov 11, 2021 • edited by pdurbin Loading

coveralls commented Nov 11, 2021 • edited Loading

qqmyers commented Jul 11, 2022

pdurbin left a comment

Choose a reason for hiding this comment

pdurbin left a comment

Choose a reason for hiding this comment

pdurbin Sep 12, 2022

Choose a reason for hiding this comment

qqmyers commented Sep 15, 2022

pdurbin commented Oct 26, 2022

scolapasta commented Oct 31, 2022

sekmiller Nov 16, 2022

Choose a reason for hiding this comment

pdurbin Nov 21, 2022

Choose a reason for hiding this comment

sekmiller Nov 16, 2022

Choose a reason for hiding this comment

pdurbin Nov 21, 2022

Choose a reason for hiding this comment

sekmiller left a comment

Choose a reason for hiding this comment

pdurbin left a comment

Choose a reason for hiding this comment

pdurbin Nov 21, 2022

Choose a reason for hiding this comment

pdurbin Nov 21, 2022

Choose a reason for hiding this comment

sekmiller left a comment

Choose a reason for hiding this comment

mreekie commented Mar 30, 2023

qqmyers commented Nov 11, 2021 •

edited by pdurbin

Loading

coveralls commented Nov 11, 2021 •

edited

Loading