bazel: Remove hardcoded dependency on //:protoc from language runtimes
#19679
+30
−14
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Without this change, language runtimes still result in a build of `//:protoc` even with a prebuilt `proto_toolchain` registered or `--proto_compiler` set to a precompiled protoc.
|
@shaod2 Would you be available to review this? |
fmeum
force-pushed
the
19558-dont-hardcode-protoc
branch
from
d92eae2 to
3bd939d
Compare
mkruskal-google
added
the
🅰️ safe for tests
github-actions
bot
removed
the
🅰️ safe for tests
|
Hey Fabian, So IIUC, this PR is trying to make the protoc binary used by proto_library customizable. While that might be a possible long-term solution, it opens the door to a lot of potential misuse. Notably, mixing alternate implementations or different versions of protoc is typically unsupported and can have unexpected results. We have poison pills to avoid some of these situations outside of Bazel, but they haven't been added to all our languages yet. A more conservative change would be to just have some kind of flag that toggles between build-from-source and the prebuilt releases we publish to github, without allowing arbitrary binaries to be injected. Additionally, this PR is touching protobuf.bzl, which just contains our legacy internal macros/rules (which we want to delete). These have been replaced by the rules in //bazel, which were recently moved from other Bazel repos. |
These macros are still used to bootstrap the Java well-known protos for the default Java
Isn't the |
mbland
added a commit
to mbland/rules_scala
that referenced
this pull request
Mar 1, 2025
Keeps the `README` guidance in sync with what we're actually using in `WORKSPACE` for consistency's sake. @crt-31 and I found that the Windows build failure for bazelbuild#1710 mentioned in the earlier commit is related to the Windows/MSVC file path length limit. `src/google/protobuf/compiler/java/java_features.pb.h`, the path specified in the error message, doesn't exist until `protobuf` v25.0. - protocolbuffers/protobuf#12947 Furthermore, the Protobuf team currently plans to just drop MSVC support: - https://protobuf.dev/news/v30/#poison-msvc--bazel - protocolbuffers/protobuf#20085 I plan to experiment again with "Protobuf Toolchainization", which I'd tried in October when beginning the Bzlmod experiment. Here are some interesting background resources before I dig in on that: - bazelbuild/rules_proto#213 - bazelbuild/rules_proto#179 - https://github.com/bazelbuild/rules_proto/releases/tag/6.0.0 - https://github.com/aspect-build/toolchains_protoc/ - protocolbuffers/protobuf#20182 - protocolbuffers/protobuf#19679 - protocolbuffers/protobuf#19558
mbland
added a commit
to mbland/rules_scala
that referenced
this pull request
Mar 1, 2025
Registers a precompiled protocol compiler toolchain when `--incompatible_enable_proto_toolchain_resolution` is `True`. Part of bazelbuild#1482 and bazelbuild#1652. Stops `protoc` recompilation, and fixes the build breakage in bazelbuild#1710 due to `protobuf` include paths exceeding the Visual Studio path length limit. The updates to `scala_proto/scala_proto_toolchain.bzl` were inspired by: - protocolbuffers/protobuf: bazel: Remove hardcoded dependency on //:protoc from language runtimes #19679 protocolbuffers/protobuf#19679 The `proto_lang_toolchain` call was inspired by the `README` from: - https://github.com/aspect-build/toolchains_protoc/ Adds `scripts/update_protoc_integrity.py` to automatically update `scala/private/protoc/protoc_integrity.bzl`. This should make builds of `rules_scala` much faster all around. Given the fact that this feature depends on recent `protobuf` versions, and the Windows `protobuf` build breaks without it, we have a catch-22. It likely can't be separated from the rest of bazelbuild#1710, though I would prefer that. It also seems likely that we'd eventually need to do this to continue supporting Windows, per: - protocolbuffers/protobuf#12947 - https://protobuf.dev/news/v30/#poison-msvc--bazel - protocolbuffers/protobuf#20085 More background on proto toolchainization: - Proto Toolchainisation Design Doc https://docs.google.com/document/d/1CE6wJHNfKbUPBr7-mmk_0Yo3a4TaqcTPE0OWNuQkhPs/edit - bazelbuild/bazel: Protobuf repo recompilation sensitivity bazelbuild/bazel#7095 - bazelbuild/rules_proto: Implement proto toolchainisation bazelbuild/rules_proto#179 - rules_proto 6.0.0 release notes mentioning Protobuf Toolchainisation https://github.com/bazelbuild/rules_proto/releases/tag/6.0.0
mbland
added a commit
to mbland/rules_scala
that referenced
this pull request
Mar 2, 2025
This is an experiment to see if applying protocolbuffers/protobuf#19679 via a patch and registering `@rules_scala_toolchains//protoc/...:all` will fix Windows MSVC builds. Local experiments suggest that this does indeed enable the toolchainized `protoc` to take precedence and avoid compiling protobuf.
mbland
added a commit
to mbland/rules_scala
that referenced
this pull request
Mar 10, 2025
Bumps dependencies to versions that are compatible with both Bazel 7.5.0 and 8.0.0, and adds protocol compiler toolchainization in `//protoc` for `protobuf` v29 and later. Closes bazelbuild#1652. Part of bazelbuild#1482. - ScalaPB jars: 0.11.17 => 1.0.0-alpha.1 - Scalafmt: 3.9.2 => 3.9.3 - `rules_python`: 0.38.0 => 1.2.0 - `rules_cc`: 0.0.9 => 0.1.1 - `rules_java`: 7.12.4 => 8.10.0 - `protobuf`: 21.7 => 30.0 - `rules_proto`: 6.0.2 => 7.1.0 Bazel 6 is officially unsupported as of this change and the upcoming `rules_scala` 7.0.0 release. Updates `.bazelci/presubmit.yml` to bump the `7.x` build to `last_rc`. Registers a precompiled protocol compiler toolchain when `--incompatible_enable_proto_toolchain_resolution` is `True`. Otherwise, `register_toolchains(//protoc:all)` toolchains is a no-op, as it will be empty. `scripts/update_protoc_integrity.py` automatically updates `scala/private/protoc/protoc_integrity.bzl`. The `protobuf` patch is the `git diff` output from protocolbuffers/protobuf#19679, which also inspired the updates to `scala_proto/scala_proto_toolchain.bzl`. The `proto_lang_toolchain` call in the `BUILD` file generated by `protoc/private/protoc_toolchain.bzl` was inspired by the `README` from: - https://github.com/aspect-build/toolchains_protoc/ Loads `java_proto_library` from `com_google_protobuf`, replacing the officially deprecated version from `rules_java`. Bumps Scalafmt to 3.9.3 out of convenience. Updates to `README.md`, and updates to `WORKSPACE` and `third_party/repositories` files precipitated by the dependency updates, comprise the remainder of this change. --- We're no longer planning to support Bazel 6 in the next major release per @simuons's decision in: - bazelbuild#1482 (comment) The plan is now to land the Bazel 7 and 8 compatibility updates first, then land the Bzlmod change. This enables us to make only one new major version release, instead of two (whereby the first release would've continued supporting Bazel 6). It turns out the two major version plan wouldn't've been possible. Bazel 8 and `rules_java` 8 require `protobuf` >= v29, but this bump caused Windows builds to break when compiling `protoc` in bazelbuild#1710. `src/google/protobuf/compiler/java/java_features.pb.h`, the path specified in the error message, doesn't exist until `protobuf` v25.0. @crt-31 and I found that this was related to the Windows/MSVC 260 character file path length limit. What's more, the `protobuf` team plans to drop MSVC support specifically because of this path length limit. The protocol compiler toolchain prevents `protoc` recompilation, which fixes the Windows breakage while making all builds faster. Since Windows builds break since at least `protobuf` v25, but `protoc` toolchainization requires v29, the version bump and the `protoc` toolchain must land together. --- I tried several things to get protocol compiler toolchainization to work with `protobuf` v28.2, described below. However, each path only led to the same suffering described in the new "Why this requires `protobuf` v29 or later" section of the README. I discovered along the way that `protobuf` v30 isn't compatible with Bazel 6.5.0 at all. I added an explanation to the "Limited Bazel 6.5.0 compatibility" section of `README.md`. --- I experimented with using `protobuf` v28.2, `rules_proto` 6.0.2, and `rules_java` 7.12.4 and 8.10.0. I updated the `protobuf` patch for v28.2 with the following statements: ```py load("//bazel/common:proto_common.bzl", "proto_common") load("@rules_proto//proto:proto_common.bzl", "toolchains") _PROTO_TOOLCHAIN = "@rules_proto//proto:toolchain_type" _PROTO_TOOLCHAIN_ATTR = "INCOMPATIBLE_ENABLE_PROTO_TOOLCHAIN_RESOLUTION" _PROTOC_TOOLCHAINS = toolchains.use_toolchain(_PROTO_TOOLCHAIN) def _protoc_files_to_run(ctx): if getattr(proto_common, _PROTO_TOOLCHAIN_ATTR, False): ``` I updated `protoc/private/protoc_toolchain.bzl` to use `proto_common` from `rules_proto`. I also created a `rules_proto` 6.0.2 patch for `proto_toolchain()` to fix a "no such package: //proto" breakage: ```diff 6.0.2 patch for `proto_toolchain()`: ```diff diff --git i/proto/private/rules/proto_toolchain.bzl w/proto/private/rules/proto_toolchain.bzl index a091b80..def2699 100644 --- i/proto/private/rules/proto_toolchain.bzl +++ w/proto/private/rules/proto_toolchain.bzl @@ -33,7 +33,7 @@ def proto_toolchain(*, name, proto_compiler, exec_compatible_with = []): native.toolchain( name = name + "_toolchain", - toolchain_type = "//proto:toolchain_type", + toolchain_type = Label("//proto:toolchain_type"), exec_compatible_with = exec_compatible_with, target_compatible_with = [], toolchain = name, ``` I tried adding combinations of the following `--incompatible_autoload_externally` flag values to .bazelrc`: ```txt common --incompatible_autoload_externally=+@protobuf,+@rules_java ``` Nothing worked. --- After the `protobuf` v29 bump, and before the ScalaPB 1.0.0-alpha.1 bump, `scala_proto` targets would fail with the following error: ```txt ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14: ProtoScalaPBRule external/com_google_protobuf/src/google/protobuf/any_proto_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/.../bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) --jvm_extra_protobuf_generator_out: java.lang.NoSuchMethodError: 'java.lang.Object com.google.protobuf.DescriptorProtos$FieldOptions.getExtension(com.google.protobuf.GeneratedMessage$GeneratedExtension)' at scalapb.compiler.DescriptorImplicits$ExtendedFieldDescriptor.fieldOptions(DescriptorImplicits.scala:329) [ ...snip... ] java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:30) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala) ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14 Building source jar external/com_google_protobuf/src/google/protobuf/any_proto_scalapb-src.jar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/darwin_arm64-opt-exec-ST-a828a81199fe/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Without this change, language runtimes still result in a build of
//:protoceven with a prebuiltproto_toolchainregistered or--proto_compilerset to a precompiled protoc. Removing this hardcoded dependency allows a (fast) build ofjava_proto_librarytargets without a C++ toolchain assuming a prebuilt protoc.Work towards #19558