Bazel 8 + rules_java 8 updates, protoc toolchain

[mbland]

Description

Bumps dependencies to versions that are compatible with both Bazel 7.5.0 and 8.0.0, and adds protocol compiler toolchainization in //protoc for protobuf v29 and later.

Closes #1652. Part of #1482.

• ScalaPB jars: 0.11.17 => 1.0.0-alpha.1
• Scalafmt: 3.9.2 => 3.9.3
• rules_python: 0.38.0 => 1.2.0
• rules_cc: 0.0.9 => 0.1.1
• rules_java: 7.12.4 => 8.10.0
• protobuf: 21.7 => 30.0
• rules_proto: 6.0.2 => 7.1.0

Bazel 6 is officially unsupported as of this change and the upcoming rules_scala 7.0.0 release. Updates .bazelci/presubmit.yml to bump the 7.x build to last_rc.

Registers a precompiled protocol compiler toolchain when --incompatible_enable_proto_toolchain_resolution is True. Otherwise, register_toolchains(//protoc:all) toolchains is a no-op, as it will be empty.

scripts/update_protoc_integrity.py automatically updates scala/private/protoc/protoc_integrity.bzl. The protobuf patch is the git diff output from protocolbuffers/protobuf#19679, which also inspired the updates to scala_proto/scala_proto_toolchain.bzl. The proto_lang_toolchain call in the BUILD file generated by protoc/private/protoc_toolchain.bzl was inspired by the README from:

• https://github.com/aspect-build/toolchains_protoc/

Loads java_proto_library from com_google_protobuf, replacing the officially deprecated version from rules_java. Bumps Scalafmt to 3.9.3 out of convenience.

Updates to README.md, and updates to WORKSPACE and third_party/repositories files precipitated by the dependency updates, comprise the remainder of this change.

Motivation

We're no longer planning to support Bazel 6 in the next major release per @simuons's decision in:

• Support Bzlmod and add rules_scala to bazel-central-registry #1482 (comment)

The plan is now to land the Bazel 7 and 8 compatibility updates first, then land the Bzlmod change. This enables us to make only one new major version release, instead of two (whereby the first release would've continued supporting Bazel 6).

It turns out the two major version plan wouldn't've been possible. Bazel 8 and rules_java 8 require protobuf >= v29, but this bump caused Windows builds to break when compiling protoc in #1710. src/google/protobuf/compiler/java/java_features.pb.h, the path specified in the error message, doesn't exist until protobuf v25.0.

@crt-31 and I found that this was related to the Windows/MSVC 260 character file path length limit. What's more, the protobuf team plans to drop MSVC support specifically because of this path length limit.

The protocol compiler toolchain prevents protoc recompilation, which fixes the Windows breakage while making all builds faster. Since Windows builds break since at least protobuf v25, but protoc toolchainization requires v29, the version bump and the protoc toolchain must land together.

---

I tried several things to get protocol compiler toolchainization to work with protobuf v28.2, described below. However, each path only led to the same suffering described in the new "Why this requires protobuf v29 or later" section of the README.

I discovered along the way that protobuf v30 isn't compatible with Bazel 6.5.0 at all. I added an explanation to the "Limited Bazel 6.5.0 compatibility" section of README.md.

---

I experimented with using protobuf v28.2, rules_proto 6.0.2, and rules_java 7.12.4 and 8.10.0. I updated the protobuf patch for v28.2 with the following statements:

load("//bazel/common:proto_common.bzl", "proto_common")
load("@rules_proto//proto:proto_common.bzl", "toolchains")

_PROTO_TOOLCHAIN = "@rules_proto//proto:toolchain_type"
_PROTO_TOOLCHAIN_ATTR = "INCOMPATIBLE_ENABLE_PROTO_TOOLCHAIN_RESOLUTION"
_PROTOC_TOOLCHAINS = toolchains.use_toolchain(_PROTO_TOOLCHAIN)

def _protoc_files_to_run(ctx):
    if getattr(proto_common, _PROTO_TOOLCHAIN_ATTR, False):

I updated protoc/private/protoc_toolchain.bzl to use proto_common from rules_proto.

I also created a rules_proto 6.0.2 patch for proto_toolchain() to fix a "no such package: //proto" breakage:

 6.0.2 patch for `proto_toolchain()`:

```diff
diff --git i/proto/private/rules/proto_toolchain.bzl w/proto/private/rules/proto_toolchain.bzl
index a091b80..def2699 100644
--- i/proto/private/rules/proto_toolchain.bzl
+++ w/proto/private/rules/proto_toolchain.bzl
@@ -33,7 +33,7 @@ def proto_toolchain(*, name, proto_compiler, exec_compatible_with = []):

     native.toolchain(
         name = name + "_toolchain",
-        toolchain_type = "//proto:toolchain_type",
+        toolchain_type = Label("//proto:toolchain_type"),
         exec_compatible_with = exec_compatible_with,
         target_compatible_with = [],
         toolchain = name,

I tried adding combinations of the following --incompatible_autoload_externally flag values to .bazelrc`:

common --incompatible_autoload_externally=+@protobuf,+@rules_java

Nothing worked.

---

After the protobuf v29 bump, and before the ScalaPB 1.0.0-alpha.1 bump, scala_proto targets would fail with the following error:

ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14:
  ProtoScalaPBRule
  external/com_google_protobuf/src/google/protobuf/any_proto_jvm_extra_protobuf_generator_scalapb.srcjar
  failed: (Exit 1): scalapb_worker failed:
  error executing ProtoScalaPBRule command
  (from target @@com_google_protobuf//src/google/protobuf:any_proto)
  bazel-out/.../bin/src/scala/scripts/scalapb_worker
    ... (remaining 2 arguments skipped)

--jvm_extra_protobuf_generator_out:
  java.lang.NoSuchMethodError:
  'java.lang.Object com.google.protobuf.DescriptorProtos$FieldOptions.getExtension(com.google.protobuf.GeneratedMessage$GeneratedExtension)'
    at scalapb.compiler.DescriptorImplicits$ExtendedFieldDescriptor.fieldOptions(DescriptorImplicits.scala:329)
  [ ...snip... ]

java.lang.RuntimeException: Exit with code 1
    at scala.sys.package$.error(package.scala:30)
    at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44)
    at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96)
    at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49)
    at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39)
    at scripts.ScalaPBWorker.main(ScalaPBWorker.scala)

ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14
  Building source jar external/com_google_protobuf/src/google/protobuf/any_proto_scalapb-src.jar
  failed: (Exit 1): scalapb_worker failed:
  error executing ProtoScalaPBRule command
  (from target @@com_google_protobuf//src/google/protobuf:any_proto)
  bazel-out/darwin_arm64-opt-exec-ST-a828a81199fe/bin/src/scala/scripts/scalapb_worker
    ... (remaining 2 arguments skipped)

[mbland]

Hmm, an odd failure on the Windows build:

ERROR: C:/tools/msys64/home/b/_bazel_b/xknd5zlq/external/com_google_protobuf/src/google/protobuf/compiler/java/BUILD.bazel:87:11: Compiling src/google/protobuf/compiler/java/java_features.pb.cc [for tool] failed: (Exit 2): cl.exe failed: error executing CppCompile command (from target @@com_google_protobuf//src/google/protobuf/compiler/java:java_features_bootstrap) C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\bin\HostX64\x64\cl.exe ... (remaining 1 argument skipped)
--
  | external/com_google_protobuf/src/google/protobuf/compiler/java/java_features.pb.cc(6): fatal error C1083: Cannot open include file: 'google/protobuf/compiler/java/java_features.pb.h': No such file or directory

src/google/protobuf/compiler/java/java_features.pb.h most definitely exists in protobuf v29.3.

All the other builds seem to pass. If they do, I'll kick this with an empty commit to see if we can get Windows to play along, in case it was some weird transient error.

[mbland]

Yep, just windows. And my empty commit bump just failed in the same way. Hmm... why can't it find the #include file right next to the source?

[crt-31]

I looked at the windows issue... the issue is that the full include path for that file is over the char limit (260 chars).
I think its a cc_library issue, but still investigating when it got broken... I think it has to do with the param file limitations, but I'm still looking into it.

Also, I don't remember having to compile those protobuf tools locally before... I feel like they came precompiled before?

[crt-31]

OK, I found the related issue here: protocolbuffers/protobuf#12947. (I'm going to look into it more cause I use MSVC all the time and haven't run into these issues.)

I wonder if we can go back to using the prebuilt protobuf libraries, then we don't need to require the c++ compiler in order to use rules_scala.

[mbland]

@crt-31 Heh, you beat me to it...I snuck a look on my phone while out this evening and found the same issue. Got online to post it, but was too late. 😛 What's more, my next Bzlmod blog post is going to talk about how I had to patch rules_rust to work around Windows path length breakages.

And I saw the paragraph that you just responded to:

Since we don't have any good way to mitigate this issue, and because bazel+msvc usage is low compared to other ways to build on Windows, we are planning to drop support for bazel+msvc in Protobuf 34 (the Q1 2026 release). Bazel 30 already disables support for this combination, with a flag to reenable support for it which gives time to gather feedback on this as well as give folks a longer horizon to migrate their builds to either make+msvc or bazel+clang-cl which don't suffer from this artificial limitation without being stuck on Protobuf 29 or lower:

https://protobuf.dev/news/v30/#poison-msvc--bazel

The linked announcement makes reference to protocolbuffers/protobuf#20085: "Breaking Change: Dropping support for Bazel+MSVC".

But to your point above:

I wonder if we can go back to using the prebuilt protobuf libraries, then we don't need to require the c++ compiler in order to use rules_scala.

I did experiment with using prebuilt Protobuf toolchains in my proto-toolchainization branch in mbland/rules_scala back in October, when I was first experimenting with Bzlmodification. I think I may dust it off and play with it again in light of this development; it may also provide a performance win, since protobuf wants to recompile constantly. (I'm also not aware of what "going back to using prebuilt protobuf libraries" would involve. We're already using the prebuilt protobuf-java artifact; I think protobuf keeps wanting to recompile protoc.)

[mbland]

UPDATE: Please skip this comment and read my new comment about the more recent fully working and polished protoc toolchainization.

The good news is, I went ahead and implemented protoc toolchainization in the latest commit, and it works really well. Please check out what I did in 7b7ab2a and let me know what you think of the README update.

The bad news is, the Windows build is still broken in the same way. I'm pretty sure it's because protobuf v29.3 still uses native.java_proto_library, which still depends on @com_google_protobuf//:protoc: Update: Nope, it's not. It's because the precompiled toolchain wasn't registered early enough.

• https://github.com/protocolbuffers/protobuf/blob/v29.3/bazel/java_proto_library.bzl#L11-L16

def java_proto_library(**kwattrs):
    # Only use Starlark rules when they are removed from Bazel
    if not hasattr(native, "java_proto_library"):
        _java_proto_library(**kwattrs)
    else:
        native.java_proto_library(**kwattrs)

I believe this is what @fmeum is trying to fix in protocolbuffers/protobuf#19679 (which I borrowed from in scala_proto/scala_proto_toolchain.bzl).

@simuons Until this gets resolved one way or another, what do you think of making Windows builds optional for now? Then when a fix comes, we can make it required again. Or maybe I can hack something to downgrade the Windows build down to protobuf v21.7 again, as it was before this pull request (i.e., currently remains on master).

@crt-31 I know you really don't want that to happen, but it's effectively blocking the next rules_scala release at this point.

@fmeum If you've any better ideas, or my understanding is incorrect in any way, I'm all ears.

At any rate, this gives me time to think about maybe adding more tests for this latest protoc toolchainization, if we need them.

[mbland]

Hold the presses... I just pushed an experiment that fixed the Windows build. I created a patch from protocolbuffers/protobuf#19679, then added a line to WORKSPACE to register the prebuilt toolchain early. Success!

Everything else is failing right now (all the tests for all the other WORKSPACE repos I haven't modified), but I'll fix that up later. The main question is whether to actually commit this patch.

[mbland]

OK, please ignore my update from yesterday. I've learned a lot, including that the protoc failure was due to the new toolchain not getting registered early enough, not that protobuf was selecting native rules. The latest commit polishes the protoc toolchainization implementation, and the Windows build is passing again (along with all the others).

The majority of the changes in the PR are still the boilerplate updates to WORKSPACE and .bazelversion files, and automated updates to third_party/repositories/scala_*.bzl files. The most substantial changes are in:

• Updating Bazel versions in .bazelci/presubmit.yml
• Adding common --incompatible_enable_proto_toolchain_resolution to .bazelrc and tools/bazel.rc.buildkite
• README.md
• The new examples/overridden_artifacts repo and new test in test/shell/test_examples.sh
• The new //protoc package with the protoc toolchainization logic
• The protobuf patch in scala/deps.bzl from bazel: Remove hardcoded dependency on //:protoc from language runtimes protocolbuffers/protobuf#19679
• The slight update to scala/private/macros/scala_repositories.bzl
• Updates to scala/toolchains.bzl and scala/toolchains_repo.bzl for toolchainization
• Minor tweaks to scala_config.bzl
• scala_proto/scala_proto_toolchain.bzl updates based on bazel: Remove hardcoded dependency on //:protoc from language runtimes protocolbuffers/protobuf#19679
• Version bumps in scripts/create_repository.py
• The new scripts/update_protoc_integrity.py
• Adding allow_empty = True to a glob() in test/semanticdb/BUILD
• Dropping Scala 2.11 tests from test_thirdparty_version.sh and test_version.sh

Still, this is a lot more than I'd originally planned to include in this pull request. Some of these updates can probably be teased out into separate PRs.

And/or, we could go back to the original 7.x and 8.x release plan. This would mean backing out the protobuf upgrade from v21.7 to v29.3, and other updates along with it, most notably protoc toolchainization.

In either case, Bzlmod could land in 7.0 or 7.1. I'll prepare my Bzlmod branch, but it would remain a relatively light update on top of whatever changes we decide to include, and when.

Anywho, as usual, options... @simuons Let me know what you'd prefer to do here.

[mbland]

P.S. I can confirm that this branch is still compatible with Bazel 6.5.0, both with the C++ compiler flags and with protocol compiler toolchainization enabled, as well as Bazel 8.1.1.

I have protoc toolchainized Bzlmod builds working on my unstable 'bzlmod-enable' branch and both of my stable bzlmod and bzlmod-bazel-8 branches. Can confirm they work with Bazels 7.5.0, 8.1.1, rolling, and last_green.

# rolling
Starting local Bazel server (9.0.0-pre.20250210.1) and connecting to it...

# last_green
2025/03/02 22:24:18 Using unreleased version at commit 15a3526621559170c2a28e9dae5e9d7c72a82f02                                                                  
2025/03/02 22:24:18 Downloading https://storage.googleapis.com/bazel-builds/artifacts/macos_arm64/15a3526621559170c2a28e9dae5e9d7c72a82f02/bazel...

The thing that's at once a bug and a feature of the Bzlmod branches is the fact that I had to manually copy the protobuf patch and add a single_version_override to each nested module to enable protoc toolchainization. This is because you can only apply patches in the main module, and they must be regular files in the same module. In a pathological case like this, with having to apply a patch to all these nested modules, it's a drag. In a typical case, when you're working with a single module, it's nice that Bzlmod forces things to be so explicit and correct, as opposed to WORKSPACE macros doing mysterious and important work.

And of course, in the best possible worlds, protocolbuffers/protobuf#19679 will get merged, and we can upgrade to at least v29.4 and forget about this patching and single_version_override business.

At any rate, for now, the patch is helping everything work, and I've documented the hell out of it in the README.

[mbland]

I've just gone ahead and added these bumps as well, and propagated them to bzlmod-enable, bzlmod, and bzlmod-bazel-8:

• Go: 1.24.0 => 1.24.1
• Scalafmt: 3.9.1 => 3.9.2
• org.scala-sbt:compiler-interface: 1.10.7 => 1.10.8
• org.scala-sbt:util-interface: 1.10.7 => 1.10.10
• rules_java: 8.9.0 => 8.10.0

[mbland]

Just bumped:

• grpc: 1.70.0 => 1.71.1
• protobuf-java: 4.29.3 => 4.30.0
• protobuf: v29.3 => v30.0

I'm also about to try breaking out separate pull requests from this one, in case that will prove more helpful. Then we'll have the option of landing the different pieces in smaller chunks, or at least making more sense of the pile of changes in this PR.

[mbland]

@simuons and @liucijus Before you review, approve, and merge this pull request, consider reviewing and merging these others first, which I just extracted from this one:

• Bazel 7 and MSVC compatible version updates #1711
• Add examples/overridden_artifacts and test #1713
• 7.0.0 README updates and minor code changes #1714

Each of those is orthogonal to one another, and is much easier to understand. Merging them will render this pull request much smaller and more focused itself.

Also, what do you think of bumping the default Scala version from the current 2.12.20 to 2.13.16?

Finally, #1712 (which is not for review at this point) shows that even protobuf v25.5 breaks MSVC builds by default. I may play with that branch/pull request some more, but I think it's proving that upgrading from protobuf v21.7 very rapidly leads to the toolchainization solution in this PR.

[mbland]

I've found out that protocol compiler toolchainization requires protobuf v29 and later. Details are in 116dd19, both in its README.md updates and its commit message.

As a result, it seems Windows users will have to make the leap to protobuf v29 and protocol compiler toolchainization. I don't think it's worth the time trying to keep bumping #1712 to see if there's a version between 21.7 and 25.5 that will work without toolchainization. So I'm going to close that PR.

I also found out protobuf v30 is right out for Bazel 6.5.0, so I've updated that section of README.md as well.

Still thinking it's worth landing #1711, #1713, and #1714 before this one. #1715 should definitely go in soon as well. Then I can rebase this pull request to something much more reasonable. And then, bzlmod-enable is waiting in the wings.

[simuons]

Thanks, @mbland. I'll go over other PRs first and then come back to this one.

Also, what do you think of bumping the default Scala version from the current 2.12.20 to 2.13.16?

I'm not against it, I'm for it I would say, but would like to know why. Are there technical reasons?

[mbland]

I'm not against it, I'm for it I would say, but would like to know why. Are there technical reasons?

Not really. I'll leave it alone if you don't see a compelling reason to change it.

Rebasing and squashing and updating the PR description as we speak...

[mbland]

@simuons OK, this PR is all ready for review now.