You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
As a user of the OpenSearch sink without for OpenSearch Serverless vector and time series collections that do not support custom document id, I would like to prevent duplicate data from entering OpenSearch.
Describe the solution you'd like
Configuration options in the OpenSearch sink that will enable querying OpenSearch for a documents that may already exist in OpenSearch to prevent duplicate documents.
- opensearch:
....
query_for_existing_document:
query_when: 'getMetadata("potential_duplicate") == true'
query_term: 'id'
action_on_found: drop // only option currently
query_duration: PT3M
Additional context
Add any other context or screenshots about the feature request here.
The text was updated successfully, but these errors were encountered:
@graytaylor0 , Could we solve this by implementing OpenSearch as a processor to find it, then use the drop_events processor to drop it. We have #1984 opened as a request to query data from OpenSearch. That could help solve this more generically.
Is your feature request related to a problem? Please describe.
As a user of the OpenSearch sink without for OpenSearch Serverless vector and time series collections that do not support custom document id, I would like to prevent duplicate data from entering OpenSearch.
Describe the solution you'd like
Configuration options in the OpenSearch sink that will enable querying OpenSearch for a documents that may already exist in OpenSearch to prevent duplicate documents.
Additional context
Add any other context or screenshots about the feature request here.
The text was updated successfully, but these errors were encountered: