Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dynamic execution ignores exec constraints #25221

Open
keith opened this issue Feb 6, 2025 · 3 comments
Open

Dynamic execution ignores exec constraints #25221

keith opened this issue Feb 6, 2025 · 3 comments
Labels
P3 We're not considering working on this, but happy to review a PR. (No assignee) team-Local-Exec Issues and PRs for the Execution (Local) team type: feature request

Comments

@keith
Copy link
Member

keith commented Feb 6, 2025

Description of the bug:

We have a test target like this:

foo_test(
   name = "foo",
   exec_group_compatible_with = {
       "test": ["//:some_gpu_requirement"],
   }
   ...
)

When developers compile on CPU machines, where the host platform does not have the //:some_gpu_requirement, but a register remote execution platform does have the constraint, I believe dynamic execution ignores this difference, and still attempts to race the test locally, even though it doesn't match the requirements.

I can probably workaround this by disabling dynamic execution for the TestRunner mnemonic, but I'm still surprised by this behavior.

If I aquery the test I can see the execution platform is set correctly, but then you can see it still runs locally with --debug_spawn_scheduler

INFO: Analyzed target //Support/test:system-info/system-info.mlir.test (0 packages loaded, 12 targets and 2 aspects configured).
INFO: local branch of Support/test/system-info/system-info.mlir.test/test.log finished and was not cancelled
INFO: CancellationException of remote branch of Support/test/system-info/system-info.mlir.test/test.log, returning null
INFO: local branch of Support/test/system-info/system-info.mlir.test/test.log finished and was not cancelled
INFO: CancellationException of remote branch of Support/test/system-info/system-info.mlir.test/test.log, returning null

Which category does this issue belong to?

No response

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

No response

Which operating system are you running Bazel on?

Linux

What is the output of bazel info release?

b4c611b

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?


If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

@sgowroji sgowroji added the team-Local-Exec Issues and PRs for the Execution (Local) team label Feb 7, 2025
@tjgq
Copy link
Contributor

tjgq commented Feb 11, 2025

To clarify - is the FR here to automatically disable dynamic execution when the local/remote platform constraints don't match (instead of expecting the user to override strategy flags)?

@keith
Copy link
Member Author

keith commented Feb 11, 2025

I think so yes, but more-so I guess this is just related to #19904

From the user perspective I feel like if I correctly wire up platforms, cases like this should work without me having to think about strategies (unless I want to)

@bigelephant29 bigelephant29 added type: feature request P3 We're not considering working on this, but happy to review a PR. (No assignee) and removed type: bug untriaged labels Feb 18, 2025
@keith
Copy link
Member Author

keith commented Mar 5, 2025

I think #24265 might provide a practical workaround, even if we want to fix this as well

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P3 We're not considering working on this, but happy to review a PR. (No assignee) team-Local-Exec Issues and PRs for the Execution (Local) team type: feature request
Projects
None yet
Development

No branches or pull requests

6 participants