Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ONTAP S3 Store with existence cache #1630

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

prakad1x
Copy link

@prakad1x prakad1x commented Mar 16, 2025

Implements a NetApp ONTAP S3 compatible backend with an existence cache for better performance when checking if objects exist. This provides a specialized S3 implementation for NetApp ONTAP system with optimized handling for Content Addressable Storage.

Description

This PR implements support for NetApp ONTAP S3-compatible storage as a backend for NativeLink, along with an existence cache layer that optimizes performance when checking if objects exist.
The implementation includes:

A dedicated OntapS3Store that provides specialized handling for NetApp ONTAP S3 systems with proper configuration for TLS, credentials management, and vserver settings.
An OntapS3ExistenceCache layer that maintains an in-memory cache of object digests and periodically synchronizes with the backend to reduce latency for repeated existence checks.

This implementation provides significant performance improvements for build systems that repeatedly check for the same objects, reducing network calls and latency in environments using NetApp ONTAP storage.

Fixes # (issue)

Type of change

Please delete options that aren't relevant.

  • New feature (non-breaking change which adds functionality)
  • This change requires a documentation update

How Has This Been Tested?

This implementation has been thoroughly tested through multiple approaches:

Unit tests covering both OntapS3Store and OntapS3ExistenceCache components
Published a new container image with tag v0.6.0-splm-2 to container registry
Deployed the tagged image to our staging environment connected to an actual NetApp ONTAP S3 instance
Tested parallel uploads using multipart functionality with large test artifacts
Validated correct behavior of cache synchronization over extended periods

Checklist

  • Updated documentation if needed
  • Tests added/amended
  • bazel test //... passes locally
  • PR is contained in a single commit, using git amend see some docs

This change is Reviewable

Implements a NetApp ONTAP S3 compatible backend with an existence cache for better performance when checking if objects exist. This provides a specialized S3 implementation for NetApp ONTAP system with optimized handling for Content Addressable Storage.

Signed-off-by: Kadam (EXT), Prajwal v08wha <prajwal.kadam.ext@siemens.com>
@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@aaronmondal
Copy link
Member

Initially, this seems quite interesting. I do believe that we should defer review of this to after the next aws sdk update. It was a long standing issue that the aws sdk didn't support hyper 1.x, but that's now been fixed in the latest versions. So I'd say let's first bring our S3 implementation to hyper 1.x and then we have a better baseline to implement this ontap functionality. If we do it the other way around it might complicate the migration as we'd have to migrate two stores to the new hyper.

@prakad1x Regarding the implementation, did you investigate a templating/monomorphization approach for the existing S3 store? This implementation seems very similar to the existing one, so there might be a way to implement this functionality with a lot less code. Also, could you outline how this is different from wrapping the S3 store in an existencecache?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants