Inherit exec_group setting from build actions when selecting executable

[spektrof]

Description of the feature request:

Hi,

In our team we are using tools to generate intermediate artifacts, which are used by C++ build and link actions afterwards. These tools have different binaries on different platforms, they are compatible with the platform where they were built and usually with the next major release platform, and generate the same outputs. When it comes to the cross-compilation, we need to make sure that we select the proper binary which is compatible with the execution platform, otherwise it might result an error. We encountered an issue during cross-compilation on rhel8 from a rhel7 host, when a local build action, defined by using the no-remote-exec tag, selected the rhel8 compatible binary and tried to run it on the rhel7 host, but failed. Remote actions run fine.

We'd like to have a feature which allows the executable tools to inherit the execution group setting from their related build actions.

Which category does this issue belong to?

Configurability, Local Execution, Remote Execution

What underlying problem are you trying to solve with this feature?

The executable rule attribute when using cfg="exec" doesn't inherit the execution group from the related build action, e.g. ctx.actions.run().

  attrs = {"tool" : attr.label(mandatory=True, executable=True, cfg="exec")},
  exec_groups = {
    "host" :  exec_group(exec_compatible_with=["@crossbuild//constraint:host"]),
    "rhel7" : exec_group(exec_compatible_with=["@crossbuild//constraint:rhel7"]),
    "rhel8" : exec_group(exec_compatible_with=["@crossbuild//constraint:rhel8"]),
  },

  exec_group = "host" if _has_no_remote_tag(ctx) else _target_platform_to_exec_group(ctx)

  ctx.actions.run(
    inputs = [],
    outputs = [intermediate_executable],
    executable = ctx.executable.tool,
    arguments = [intermediate_executable.path],
    mnemonic = "InterAction",
    progress_message = "Run intermediate action for {}".format(ctx.attr.name),
    execution_requirements = {tag : '1' for tag in ctx.attr.tags},
    exec_group = exec_group,
  )

, when the rhel8 exec_group is set, then I would expect that the tool selects the rhel8 compatible binary.

The executable rule attribute when using cfg="target" doesn't consider the tags when selecting the tool binary.

  attrs = {"tool" : attr.label(mandatory=True, executable=True, cfg="target")},
  exec_groups = {
    "host" :  exec_group(exec_compatible_with=["@crossbuild//constraint:host"]),
    "rhel7" : exec_group(exec_compatible_with=["@crossbuild//constraint:rhel7"]),
    "rhel8" : exec_group(exec_compatible_with=["@crossbuild//constraint:rhel8"]),
  },

when the exec_group is set to host at the build action or having any no-remote execution_requirement, the tool binary will be compatible with the target platform. This might be fine due to the definition of target platform.

Specifying exec_compatible_with at the rule helps when using cfg="exec", however it doesn't support the multi-platform build well because the attribute is non-configurable.

  intermediate_action(
    name = "remote_action",
    tool = select({
      "@crossbuild//config:rhel7" : ["tool_rhel7_bin"],
      "@crossbuild//config:rhel8" : ["tool_rhel8_bin"],
    }),
    exec_compatible_with = select({
      "@crossbuild//config:rhel7" : ["@crossbuild//constraint:rhel7"],
      "@crossbuild//config:rhel8" : ["@crossbuild//constraint:rhel8"],
    }),
  )

ERROR: package contains error: : //:remote_action: attribute "exec_compatible"with" is not configurable

Which operating system are you running Bazel on?

RHEL7

What is the output of bazel info release?

release 6.0.0 (@06deebfb5b73f848de5a0ea0e00fcfaa26788d1f)

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

We build Bazel from source by using an older release, we patch the source code when we really need to.
https://bazel.build/install/compile-source#build-bazel-using-bazel

What's the output of git remote get-url origin; git rev-parse HEAD ?

No response

Have you found anything relevant by searching the web?

https://bazel.build/extending/platforms
https://bazel.build/extending/exec-groups#platform-execution-properties

I didn't find similar use-case with cross-compilation including local only actions yet.

Any other information, logs, or outputs that you want to share?

I have a personal repository which might help to reproduce the problem and check the platform, constraint and rule implementation. Please set the .rc file accordingly.
https://github.com/spektrof/bazel_crossbuild/tree/main

How did I detect the problem?

We were setting both --platforms and --host_platform to the target platform, didn't use any exec_group or exec_compatible_with attribute, this case we didn't even set cfg for the attribute. When I started the cross-compilation on rhel8 from rhel7 host, the tool path has been resolved to the rhel8 compatible binary what it tried to run on rhel7 and failed.

bazel test :test_group1 --config rhel8_host --config rhel8 --config remote

What else did I try?

I tried to set exec_groups for rule with config.exec("target") but I couldn't make it work since it should be non-configurable and don't know how to consider tags there.

I tried to check the implementation of FilesToRunProvider but as far as I saw the executable path selection is separated from execution group, defined by the build action.

I was wondering whether there is a problem in the platform, constraint or rule implementation.

Do you expect these actions to run only locally?
Do you recommend to implement a toolchain for the tool instead?

Thank you very much in advance!

[pcjanzen]

related: #11432 and #19904

[katre]

The executable rule attribute when using cfg="exec" doesn't inherit the execution group from the related build action, e.g. ctx.actions.run().

I don't fully understand what you expect to happen here: execution platforms are chosen before actions are defined, and so cannot inherit from the action. The correct flow is to first define execution groups, then use those execution groups when defining actions.

  attrs = {"tool" : attr.label(mandatory=True, executable=True, cfg="exec")},
  exec_groups = {
    "host" :  exec_group(exec_compatible_with=["@crossbuild//constraint:host"]),
    "rhel7" : exec_group(exec_compatible_with=["@crossbuild//constraint:rhel7"]),
    "rhel8" : exec_group(exec_compatible_with=["@crossbuild//constraint:rhel8"]),
  },

Note that you have four execution groups defined here, host, rhel7, rhel8, and the default execution group (used for the tool attribute, because of cfg = "exec").

I expect that you actually want the tool attribute to use one of the three defined groups. You would do this with a slightly different syntax, using config.exec:

  attrs = {"tool" : attr.label(mandatory=True, executable=True, cfg = config.exec("host"))},

(Or using rhel7, etc, instead of host, depending on what you want.

You may also be interested in automatic execution groups, which lets you avoid manually defining the groups and just lets bazel pick the right execution group for each toolchain you require.

Let me know if this helps you out, or please let me know what part of your problem I have misunderstood.

[spektrof]

Hi,

Thank you for your reply!

execution platforms are chosen before actions are defined, and so cannot inherit from the action

After understood this and how I can affect the execution platform by the rule, I could find a solution to my problem.
I have to add the host constraint to exec_compatible_with whenever I am using the no-remote tag at similar type of rules.

platform(name="rhel7_host", parents = [":rhel7"], constraint_values = ["@crossbuild//constraint:host_platform"])

my_rule(
    name = "name",
    tool = select({
      "@crossbuild//config:rhel7" : ["tool_rhel7_bin"],
      "@crossbuild//config:rhel8" : ["tool_rhel8_bin"],
    }),
    tags = ["no-remote"],
    exec_compatible_with = ["@crossbuild//constraint:host_platform"],
  )

where attrs = {"tool" : attr.label(mandatory=True, executable=True, cfg = "exec")},

so tool_rhel7_bin will be used when running

$ bazel build ... --host_platform @crossbuild//platform:rhel7_host --platforms @crossbuild//platform:rhel8 --extra_execution_platforms @crossbuild//platform:rhel8 --config remote

and tool_rhel8_bin will be used without the tags and exec_compatible_with.

The AEGs looks great, I will check that when we start using Bazel 7+.
In our case we have other C++ actions next to this intermediate action, generated by the same rule, and already see that the C++ toolchain can use the target platform (doesn't have "@crossbuild//constraint:host_platform"), which is good.

I think my question has been answered and got a solution, so the issue can be closed.
Thanks!

[katre]

Glad this was useful! It's hard for me to figure out where to update documentation, because I am too familiar with the systems involved. Can you think of anywhere our docs on execution groups (or actions, or etc) can be improved that would have helped you sooner?

Closing since the issue is resolved but if you do have any doc suggestions I'll fork a new issue for them (or you can create a new issue and tag me on it).