Builtins exports.bzl file is called after the WORKSPACE

[keith]

Description of the bug:

After this commit a3f0bfa, using rules_swift in some cases results in this error:

ERROR: Traceback (most recent call last):
        File "/private/var/tmp/_bazel_ksmiley/1c823ac8225acd936431a04c304d3fec/external/rules_proto/proto/private/rules/proto_descriptor_set.bzl", line 53, column 32, in <toplevel>
                "deps": attr.label_list(
Error in label_list: Illegal argument: element in 'providers' is of unexpected type. Either all elements should be providers, or all elements should be lists of providers, but got an element of type NoneType.
ERROR: Error computing the main repository mapping: at /private/var/tmp/_bazel_ksmiley/1c823ac8225acd936431a04c304d3fec/external/build_bazel_rules_apple/apple/apple.bzl:41:5: at /private/var/tmp/_bazel_ksmiley/1c823ac8225acd936431a04c304d3fec/external/build_bazel_rules_apple/apple/internal/experimental_mixed_language_library.bzl:4:5: at /private/var/tmp/_bazel_ksmiley/1c823ac8225acd936431a04c304d3fec/external/build_bazel_rules_swift/swift/swift.bzl:79:5: at /private/var/tmp/_bazel_ksmiley/1c823ac8225acd936431a04c304d3fec/external/build_bazel_rules_swift/swift/internal/swift_proto_library.bzl:17:6: at /private/var/tmp/_bazel_ksmiley/1c823ac8225acd936431a04c304d3fec/external/rules_proto/proto/defs.bzl:18:6: initialization of module 'proto/private/rules/proto_descriptor_set.bzl' failed

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Here's a repro project protorepro.zip. Run USE_BAZEL_VERSION=last_green bazelisk build .... Testing with the commit before that does not have the issue. Using rules_proto @ the current HEAD (71c4fc69900946093ac5c82d81efd19fa522d060) does not fix the issue.

Which operating system are you running Bazel on?

macOS

What is the output of bazel info release?

No response

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

[keith]

cc @comius

[comius]

cc @brandjon @Wyverald @meteorcloudy
The root cause of this is, that builtins exports.bzl file is called after the WORKSPACE evaluation.

Reproducer:

WORKSPACE:
load(":proto.bzl", "X")

BUILD: 
--empty--

proto.bzl:
print(ProtoInfo)
X = "foo"

bazel build ... will print None

[comius]

Possible workaround might be to split bzl files containing repository rules in Apple repo, so that they don't load rules_proto (or anything that uses ProtoInfo).

[keith]

Yea that could work for this case since it's unlikely folks need that specific rule at this stage

[keith]

looks like rules_clojure has the same issue through a different codepath https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/2903#0186cce4-f095-461b-a896-2efe15924483

[PatriosTheGreat]

Hey everyone.

I'm also getting this error trying to build Tensorflow from github on Windows using the latest bazel binaries: https://github.com/bazelbuild/bazel/releases/tag/7.0.0-pre.20230302.1.

Steps to reproduce:

git clone https://github.com/tensorflow/tensorflow.git
cd tensorflow
bazel build //tensorflow/...

Error:

ERROR: Traceback (most recent call last):
        File "kg2r3nzx/external/io_bazel_rules_closure/closure/protobuf/closure_proto_library.bzl", line 146, column 32, in <toplevel>
                "deps": attr.label_list(
Error in label_list: Illegal argument: element in 'providers' is of unexpected type. Either all elements should be providers, or all elements should be lists of providers, but got an element of type NoneType.
ERROR: Error computing the main repository mapping: at tensorflow/workspace2.bzl:51:6: at kg2r3nzx/external/io_bazel_rules_closure/closure/defs.bzl:23:6: initialization of module 'closure/protobuf/closure_proto_library.bzl' failed

[Wyverald]

Could we assign some dummy Provider value to these symbols before exports.bzl is eval'd? I'm not too sure what is expected of provider-type values, but this might be easier than asking all downstream users to migrate.

[brandjon]

The root cause of this is, that builtins exports.bzl file is called after the WORKSPACE evaluation.

It's not that it's called "after", it's that its result is not at all used for WORKSPACE-loaded .bzl files. Builtins injection only applies to BUILD-loaded .bzl files.

See BazelStarlarkEnvironment -- createBuildBzlEnvUsingInjection starts from the uninjected environment and applies the given changes. There's no such logic for WORKSPACE-loaded .bzls.

(The None comes from the native-defined value intended to be overridden by builtins injection.)

[meteorcloudy]

I think this should be fixed from Bazel side since it's an incompatible change causing many downstream breakages, but is probably avoidable.

[meteorcloudy]

This should be fixed by b1341be

[keith]

Not fixed at 8b60d6d, cc @benjaminp

[meteorcloudy]

@keith Can you provide more information on what's still broken?

[keith]

Here's a new repro: protorepro.zip

Seems to be the exact same error as before you just have to be using bzlmod and not the WORKSPACE

[brentleyjones]

bzlmod is still broken in the latest rolling release:

ERROR: Traceback (most recent call last):
        File "/Users/brentley/Developer/rules_xcodeproj/examples/integration/bazel-output-base/rules_xcodeproj.noindex/build_output_base/external/rules_proto~5.3.0-21.7/proto/private/rules/proto_descriptor_set.bzl", line 53, column 32, in <toplevel>
                "deps": attr.label_list(
Error in label_list: Illegal argument: element in 'providers' is of unexpected type. Either all elements should be providers, or all elements should be lists of providers, but got an element of type Provider.

[meteorcloudy]

@comius @hvadehra Any idea?

[benjaminp]

b1341be disclaimed efforts to fix bzlmod. So, this state is not surprising.

[meteorcloudy]

/cc @Wyverald for awareness

[keith]

I can confirm the new commit fixes the real world case of this. thanks all!

[meteorcloudy]

Let's try backport both b1341be and c2553cc to Bazel 6.3.0

[meteorcloudy]

@bazel-io fork 6.3.0

[iancha1992]

Let's try backport both b1341be and c2553cc to Bazel 6.3.0

@meteorcloudy looks like b1341be is good to cherry-pick. However, when we try to cherry-pick c2553cc, we seem to need another PR or commit because of conflicts from 2 files ("src/main/java/com/google/devtools/build/lib/packages/BazelStarlarkEnvironment.java" and "src/main/java/com/google/devtools/build/lib/skyframe/BzlLoadFunction.java").

For example, createBzlToplevelsWithoutNative function should be in the "src/main/java/com/google/devtools/build/lib/packages/BazelStarlarkEnvironment.java" file already in the release-6.3.0.

The conflict from the "src/main/java/com/google/devtools/build/lib/skyframe/BzlLoadFunction.java" file is from the changed code for the function, "getAndDigestPredeclaredEnvironment"

[meteorcloudy]

@iancha1992 Thanks for the info!

@hvadehra Do you think we need those fixes in 6.3.0 at all? Did starlarkification of those symbols only happen after Bazel 6?

[hvadehra]

Some rules/symbols might have moved to _builtins before 6.0 was cut. But I don't know if there are any references to those in workspace files. We can cherry pick b1341be out of an abundance of caution. We only need c2553cc for bzlmod, so I'll defer to you if that's something you want to support in 6.3.0.

[meteorcloudy]

@iancha1992 OK, then let's only cherry pick b1341be for now.