Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix flakiness in CI #527

Open
4 tasks
allada opened this issue Dec 19, 2023 · 0 comments
Open
4 tasks

Fix flakiness in CI #527

allada opened this issue Dec 19, 2023 · 0 comments
Labels
no-effect-in-production

Comments

@allada
Copy link
Collaborator

allada commented Dec 19, 2023 •
edited
Loading

This is a tracking bug to track a flakey test in CI.

Flaky tests:

  • experimental_precondition_script_fails
  • entrypoint_does_invoke_if_set
  • entrypoint_injects_properties
  • entrypoint_sends_timeout_via_side_channel

Right now we are just sleeping a little before the test matters.

Example failure:
https://github.com/TraceMachina/nativelink/actions/runs/7268320030/job/19803924206
@allada allada changed the title experimental_precondition_script_fails flaky in CI Fix flakes in CI Dec 19, 2023
allada added a commit that referenced this issue Dec 19, 2023
@allada
Add small sleep in some tests to reduce flakes in CI
f15e16b 
This is to get around a docker-in-ci bug with how the host machine
is not able to sync files to disk properly.

see: #527
@allada allada mentioned this issue Dec 19, 2023
Merged
allada added a commit that referenced this issue Dec 19, 2023
@allada
Add small sleep in some tests to reduce flakes in CI (#526)
fd4e6a3 
This is to get around a docker-in-ci bug with how the host machine
is not able to sync files to disk properly.

see: #527
@MarcusSorealheis MarcusSorealheis changed the title Fix flakes in CI Fix flakiness in CI Mar 16, 2024
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
a9302dc 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298, TraceMachina#527
@allada allada mentioned this issue Feb 9, 2025
Merged
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
3c8d578 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298, TraceMachina#527
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
8579c00 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298, TraceMachina#527
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
4bcf7c1 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298, TraceMachina#527
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
5c657c5 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298, TraceMachina#527
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
8e3ad06 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298, TraceMachina#527
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
2e86f91 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298, TraceMachina#527
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
fc58824 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298, TraceMachina#527
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
bc8b804 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298, TraceMachina#527
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
72fc743 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298, TraceMachina#527
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
511f002 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298, TraceMachina#527
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
af64864 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298, TraceMachina#527
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
a73bcab 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298, TraceMachina#527
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
a47c20e 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298, TraceMachina#527
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
c2298d4 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298, TraceMachina#527
allada added a commit to allada/nativelink-fork that referenced this issue Feb 9, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
3e2df54 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298, TraceMachina#527
allada added a commit to allada/nativelink-fork that referenced this issue Feb 11, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits
56f427d 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: TraceMachina#1288, TraceMachina#513, TraceMachina#1298, TraceMachina#527
allada added a commit that referenced this issue Feb 12, 2025
@allada
[Breaking] Remove ResumableFileSlot and rely on high ulimits (#1582)
8b89c31 
ResumableFileSlot became to difficult to manage, instead of
managing resources this way, we use much higher
DEFAULT_OPEN_FILE_PERMITS, set ulimit (when unix) and warn
user if the limits are likely too low.

Breaking: Removed idle_file_descriptor_timeout_millis from config.

closes: #1288, #513, #1298, #527
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
no-effect-in-production
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@allada