Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cgroups v2 support #465

Merged
merged 2 commits into from
Jan 10, 2025
Merged

Cgroups v2 support #465

merged 2 commits into from
Jan 10, 2025

Conversation

mariash
Copy link
Member

@mariash mariash commented Dec 18, 2024

Summary

Add support for cgroups v2

  • Bumped runc to 1.2.3
  • In new runc default list of devices was changed (/dev/net/tun is removed) - Remove tun/tap from the default device rules opencontainers/runc#3468
  • Switched to containerd config v2. v1 is deprecated.
  • There are no subsystems in cgroup v2. If Tag is provided cgroup2 is mounted to /tmp/cgroup-N/unified (for N parallel tests). If Tag is not provided garden cgroup is in format /sys/fs/cgroup/garden.
  • CPU shares are now replaced with CPU weight.
  • In cgroups v2 kernel throws an error when large number is provided for CPU weight. In cgroup v1 kernel accepts the number for CPU shares and saves as MAX_SHARES. This behavior is replicated in the SharesBalancer.
  • CPUCgrouper is manually enabling cgroup controllers since bad cgroup folder is manually created.
  • CPU usage is read from cpu.stat file for cgroup v2.
  • In cgroup v2 only leaf cgroups can have processes. Cgroup for containerd garden-init is moved from /sys/fs/cgroup/garden/handle to /sys/fs/cgroup/garden/handle/init since /sys/fs/cgroup/garden/handle will contain pea cgroups and can not be leaf. Cgroup resources are manually set on /sys/fs/cgroup/garden/handle and this folder is manually cleaned up.
  • Switched to updated cloudfoundry docker images from unsupported cfgarden docker images

Backward Compatibility

Breaking Change? No

@mariash mariash requested a review from a team as a code owner December 18, 2024 17:57
@@ -17,7 +19,7 @@ func GetCGroupPath(cgroupsRootPath, subsystem, tag string, privileged, throttlin
}

if throttlingCPU {
parentCgroup = filepath.Join(parentCgroup, cgroups.GoodCgroupName)
parentCgroup = filepath.Join(parentCgroup, gardencgroups.GoodCgroupName)
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there no Throttling in cgroups v2?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cgroup v2 has CPU throttling as well. For both v1 and v2 we are doing this ourselves, by moving processes between bad and good cgroups (see throttler.go)

@mariash mariash force-pushed the cgroups-v2 branch 2 times, most recently from bc78bd0 to dfdebc8 Compare December 19, 2024 15:16
@mariash
Copy link
Member Author

mariash commented Dec 19, 2024

Let's wait for new runc release with the revert of /dev/net/tun removal - opencontainers/runc#4555 before merging this one. This way it will not be a breaking change.

@mariash mariash marked this pull request as draft December 19, 2024 19:06
Copy link
Contributor

@MarcPaquette MarcPaquette left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a few comments. Looks good overall!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing some tests for this file.

@mariash mariash force-pushed the cgroups-v2 branch 2 times, most recently from b4cb164 to 6e54aca Compare January 9, 2025 00:40
- Bumped runc to 1.2.3
- In new runc default list of devices was changed (/dev/net/tun is
  removed) - opencontainers/runc#3468
- Switched to containerd config v2. v1 is deprecated.
- There are no subsystems in cgroup v2. If Tag is provided cgroup2 is
  mounted to /tmp/cgroup-N/unified (for N parallel tests). If Tag is not
  provided garden cgroup is in format /sys/fs/cgroup/garden.
- CPU shares are now replaced with CPU weight.
- In cgroups v2 kernel throws an error when large number is provided for
  CPU weight. In cgroup v1 kernel accepts the number for CPU shares and
  saves as MAX_SHARES. This behavior is replicated in the
  SharesBalancer.
- CPUCgrouper is manually enabling cgroup controllers since bad cgroup
  folder is manually created.
- CPU usage is read from cpu.stat file for cgroup v2.
- In cgroup v2 only leaf cgroups can have processes. Cgroup for
  containerd garden-init is moved from /sys/fs/cgroup/garden/handle to
  /sys/fs/cgroup/garden/handle/init since /sys/fs/cgroup/garden/handle
  will contain pea cgroups and can not be leaf. Cgroup resources are
  manually set on /sys/fs/cgroup/garden/handle and this folder is
  manually cleaned up.
- Switched to updated cloudfoundry docker images from unsupported
  cfgarden docker images.
Bring back /dev/net/tun default device
@mariash
Copy link
Member Author

mariash commented Jan 9, 2025

Removed breaking change notice, since new runc brought back /dev/tun in default list of devices and added those tests back.

@mariash mariash marked this pull request as ready for review January 9, 2025 20:04
@MarcPaquette MarcPaquette merged commit 52f6548 into main Jan 10, 2025
1 check passed
@mariash mariash deleted the cgroups-v2 branch January 17, 2025 19:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

Successfully merging this pull request may close these issues.

2 participants