oci_pull fails to pull a public gcr.io image without auth credentials

[prestonvanloon]

Testing from commit ecbce02

The following function may fail with an error and stop the pull process when auth was never required.

rules_oci/oci/private/pull.bzl

Lines 149 to 160 in ecbce02

	def _fetch_auth_via_creds_helper(rctx, raw_host, helper_name):
	executable = "{}.sh".format(helper_name)
	rctx.file(
	executable,
	content = """\
	#!/usr/bin/env bash
	exec "docker-credential-{}" get <<< "$1"
	""".format(helper_name),
	)
	result = rctx.execute([rctx.path(executable), raw_host])
	if result.return_code:
	fail("credential helper failed: \nSTDOUT:\n{}\nSTDERR:\n{}".format(result.stdout, result.stderr))

Example error:

ERROR: An error occurred during the fetch of repository 'linux_debian11_multiarch_base_linux_arm64':
   Traceback (most recent call last):
        File "/home/pvanloon/.cache/bazel/_bazel_pvanloon/4d86610fb065df612cebf796c5f3f04e/external/rules_oci/oci/private/pull.bzl", line 357, column 46, in _oci_pull_impl
                mf, mf_len = downloader.download_manifest(rctx.attr.identifier, "manifest.json")
        File "/home/pvanloon/.cache/bazel/_bazel_pvanloon/4d86610fb065df612cebf796c5f3f04e/external/rules_oci/oci/private/pull.bzl", line 280, column 74, in lambda
                download_manifest = lambda identifier, output: _download_manifest(rctx, state, identifier, output),
        File "/home/pvanloon/.cache/bazel/_bazel_pvanloon/4d86610fb065df612cebf796c5f3f04e/external/rules_oci/oci/private/pull.bzl", line 237, column 23, in _download_manifest
                result = _download(rctx, state, identifier, output, "manifests", allow_fail = True)
        File "/home/pvanloon/.cache/bazel/_bazel_pvanloon/4d86610fb065df612cebf796c5f3f04e/external/rules_oci/oci/private/pull.bzl", line 203, column 22, in _download
                auth = _get_token(rctx, state, rctx.attr.registry, rctx.attr.repository)
        File "/home/pvanloon/.cache/bazel/_bazel_pvanloon/4d86610fb065df612cebf796c5f3f04e/external/rules_oci/oci/private/pull.bzl", line 115, column 24, in _get_token
                pattern = _get_auth(rctx, state, registry)
        File "/home/pvanloon/.cache/bazel/_bazel_pvanloon/4d86610fb065df612cebf796c5f3f04e/external/rules_oci/oci/private/pull.bzl", line 78, column 55, in _get_auth
                pattern = _fetch_auth_via_creds_helper(rctx, host_raw, helper_val)
        File "/home/pvanloon/.cache/bazel/_bazel_pvanloon/4d86610fb065df612cebf796c5f3f04e/external/rules_oci/oci/private/pull.bzl", line 160, column 13, in _fetch_auth_via_creds_helper
                fail("credential helper failed: \nSTDOUT:\n{}\nSTDERR:\n{}".format(result.stdout, result.stderr))
Error in fail: credential helper failed: 
STDOUT:
You do not currently have an active account selected. See https://cloud.google.com/sdk/docs/authorizing for more information.

If I disable the auth entirely with this patch, the pull works without issue. Obviously, this is not a fix but a proof of concept that auth is not required for every oci_pull.

diff --git a/oci/private/pull.bzl b/oci/private/pull.bzl
index ed339e7..57032ab 100644
--- a/oci/private/pull.bzl
+++ b/oci/private/pull.bzl
@@ -70,7 +70,7 @@ def _get_auth(rctx, state, registry):
     config = state["config"]
 
     # first look into per registry credHelpers if it exists
-    if "credHelpers" in config:
+    if "credHelpers" in config and False:
         for host_raw in config["credHelpers"]:
             host = _strip_host(host_raw)
             if host == registry:

A simple reproduction using gcp cloud shell

• gcloud cloud-shell ssh
• Checkout a project with rules_oci and add the following oci_pull

oci_pull(
    name = "linux_debian11_multiarch_base",  # Debian bullseye
    digest = "sha256:9b8e0854865dcaf49470b4ec305df45957020fbcf17b71eeb50ffd3bc5bf885d",  # 2023-05-17
    image = "gcr.io/distroless/cc-debian11",
    platforms = [
        "linux/amd64",
        "linux/arm64",
    ],
    reproducible = True,
)

• bazel build @linux_debian11_multiarch_base//:linux_debian11_multiarch_base
• Observe a fatal error

[prestonvanloon]

I believe that #284 is related as well when you have no auth credential helper

[prestonvanloon]

@thesayyn has there been any update on this issue? thank you

[prestonvanloon]

I've updated to the latest release of rules_oci (v1.3.4) and still having issues. It seems that rules_oci will fallback to using curl and that returns a different sha.

WARNING: Could not fetch the manifest. Either there was an authentication issue or trying to pull an image with OCI image media types.
--
  | Falling back to using `curl`. See https://github.com/bazelbuild/bazel/issues/17829 for the context.
  | WARNING: Could not fetch the manifest. Either there was an authentication issue or trying to pull an image with OCI image media types.
  | Falling back to using `curl`. See https://github.com/bazelbuild/bazel/issues/17829 for the context.
...
Error in download: com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException: Checksum was e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 but wanted 9b8e0854865dcaf49470b4ec305df45957020fbcf17b71eeb50ffd3bc5bf885d

See a build failure here: https://buildkite.com/prysmatic-labs/prysm/builds/76220#018aa92a-25cd-4c5c-bfb1-df605799312f/35-90

[thesayyn]

Sounds like Curl doesn't properly fail...

[prestonvanloon]

Another thing I noticed is that oci_pull does not work well with --experimental_remote_downloader due to the non-hermetic nature of a fallback to curl or some other issue.

When removing that flag, CI is working again, albeit possibly slower to fetch dependencies.

[thesayyn]

We have removed the Curl fallback now, i am not sure what needs to be done here. Does docker pull work fine with the same docker config?

[prestonvanloon]

@thesayyn we have been running for two weeks with the experimental remote downloader flag and I haven't seen any issues. We are using rules_oci v1.7.4 without issues. Thanks for following up!