Remote Vector Index Build Component -- Future Work

[jed326]

This issue is to track future work items for the remote vector index build work including both "fast follow" and "deprioritized" items.

Fast Follows

• Validations for vector repository setting
• Repository Settings Tuning
  • Upload threadpool size (currently 2x CPUs)
  • Upload buffer size (currently 16mb)
• Tuning prefix throttling, see: Optimised prefix pattern per shard for remote store data and metadata files for higher throughput OpenSearch#12567
• Buffer Size Tuning
  • BufferedInputStreams used to support retry
  • Buffer size for writing to the repository
• Default threshold size tuning
• Poll interval tuning

Long Term Improvements

• Shard level k-nn stats
• Additional Repository Metrics
• Transport APIs for getting status and cancelling jobs
• Multipart/Parallel Download -- This one has a lot of complexities, listing out the high level issues below:
  • In core the multipart download API is still marked experimental
  • There is no easy way to support decrypting individual stream parts, as there may not be any encryption context present in any given part
  • Decryption could be somewhat resolved by downloading the file completely first, then decrypting it sequentially, however that is both disk and CPU intensive
  • We also need to write the .faiss graph into an IndexOutput, which itself is a sequential operation and you cannot open multiple IndexOutputs on the same file
  • This also means that we need to internally manage a buffer that we download to in parallel, as we are unable to download in parallel directly into the IndexOutput
  • You also cannot write to an IndexOutput first and then decrypt the contents, as the IndexOutput is supposed to create an immutable file
  • This means you'd end up with up to 3x extra disk space utilization as well as additional overhead in copying/decrypting streams.
• Graph Structure Download
• Distributed retries client side

[owenhalpert]

Fast Follow:

• Track Permanently turn-off Security Manager (JSM) starting 3.0 OpenSearch#17181

LTI:

• Client retry configuration enhancements
  • If testing finds that we should increase retry count, add exponential backoff for multiple retries
  • Support configuration on retryable codes, max retry count, base delay (for retries and backoffs)
• Expand / generalize build request index parameters to allow the remote build service to choose its own parameters based on certain "hints"
  • Create API.md remote-vector-index-builder#8 (comment)
• Validate string endpoint when the user sets it, for port, scheme, expected URL pattern
  • ref1, ref2