Node memory swap support

[ehashman]

Enhancement Description

• One-line enhancement description (can be used as a release note): Kubernetes nodes support swap memory.
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2400-node-swap
• Discussion Link: https://docs.google.com/document/d/1CZtRtC8W8FwW_VWQLKP9DW_2H-hcLBcH9KBbukie67M/edit# Kubelet/Kubernetes should work with Swap Enabled kubernetes#53533
• Primary contact (assignee): @iholder101 @kannon92 @pacoxu
• Responsible SIGs: Node
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.22
  • Beta1 release target (x.y): 1.28
  • Beta2 release target (x.y): 1.29-1.31
  • Beta3 release target (x.y): 1.33
  • Stable release target (x.y): 1.34
• Alpha
  • KEP (k/enhancements) update PR(s): Node system swap support #2602
  • Code (k/k) update PR(s):
    • CI/test infra changes: Setup new node image with swap enabled test-infra#22602 Add swap periodics for node test-infra#22809
    • CRI and kubelet changes Alpha node swap support kubernetes#102823
  • Docs (k/website) update PR(s): KEP-2400: Alpha node support for swap memory website#28838
• Beta1
  • KEP (k/enhancements) update PR(s):
    • KEP-2400: Update swap KEP for 1.23 beta #2858
    • KEP-2400: Graduate swap to Beta 1 #3957
  • Issues and Code (k/k) update PR(s):
    • Support cgroups v2 with auto-calculated LimitedSwap for Burstable QoS pods
      • [KEP-2400] Enable Swap Support for Cgroup v2 Only kubernetes#119427
      • [KEP-2400] Enable auto-calculated LimitedSwap for Burstable QoS pods kubernetes#119428
      • [KEP-2400] Add full cgroup v2 swap support with automatically calculated swap limit for LimitedSwap and Burstable QoS Pods kubernetes#118764
    • Add Swap stats to /stats/summary and /metrics/resource
      • [KEP-2400] Add swap memory to the Kubelet stats API kubernetes#119425
      • [KEP-2400] Add swap to stats to Summary API and Prometheus endpoints (/stats/summary and /metrics/resource) kubernetes#118865
    • Determine a set of metrics for measuring node performance with swap enabled
      • [KEP-2400] Determine a set of metrics for measuring node performance with swap enabled kubernetes#119424
      • [KEP-2400] Add swap to stats to Summary API and Prometheus endpoints (/stats/summary and /metrics/resource) kubernetes#118865
      • Add a metric for machine swap capacity google/cadvisor#3339
    • Improve test coverage and make sure tests are green
      • [KEP-2400] Swap: improve test coverage and make sure tests are green kubernetes#119430
      • [KEP-2400] Add full cgroup v2 swap support with automatically calculated swap limit for LimitedSwap and Burstable QoS Pods kubernetes#118764
  • Docs (k/website) update(s):
    • Swap graduates to Beta1: add a blog-post website#42009
    • Swap graduates to Beta1: add documentation and update feature gates website#42167
  • Misc
    • Bugfix: MachineInfo Clone() to also clone "SwapCapacity" google/cadvisor#3293
    • Bump cadvisor version to v0.47.3 kubernetes#119225
  • Archive / Outdated
• Beta2
  • KEP (k/enhancements) update PR(s):
    • 1.29 Graduate swap to beta 2 in 1.29 #4275
    • 1.30 [KEP-2400] - Update KEP with findings from beta2 #4401
    • 1.30 [KEP-2400] update latest milestone for swap KEP #4484
  • Issues and Code (k/k) update PR(s):
    • Add swap conformance tests kubernetes#122175
    • Fail Kubelet at startup if swap is configured with cgroup v1 kubernetes#122241
    • 1.30
      • [KEP-2400] add no swap as the default option for swap kubernetes#122745
      • [KEP-2400]: Fail limited swap cgroup v1 kubernetes#123738
    • 1.31
      • [Swap][Tests][KEP-2400] Add swap serial stress tests, improve NodeConformance tests and adapt NoSwap behavior kubernetes#123557
      • [KEP-2400] Avoid logging that swap cgroup controller is missing for every container kubernetes#123749
      • 1.31 [KEP-2400] Mount tmpfs memory-backed volumes with a noswap option if supported kubernetes#124060
      • 1.31 [KEP-2400]: Restrict access to swap for containers in high priority Pods kubernetes#125277
      • 1.31 [KEP-2400]: Swap e2e tests: skip swap stress tests if swap is not provisioned kubernetes#125402
  • Docs (k/website) update(s):
    • 1.30 [KEP-2400] Update swap with beta2 deprecations website#45128
• Beta3
  • KEP (k/enhancements) update PR(s): [KEP-2400] swap updates, beta3 graduation and GA criterias #4701
  • Issues and Code (k/k) update PR(s):
    • [KEP-2400] [failing-test] resource metrics e2e tests: expect swap node and container level stats kubernetes#130053
    • [KEP-2400] [Bugfix]: Ensure container-level swap metrics are collected kubernetes#129486
    • [KEP-2400] Report swap capacity as part of node.status.nodeSystemInfo kubernetes#129954
    • [KEP-2400]: Swap-aware memory eviction (with no additional APIs) kubernetes#129578
    • [KEP-2400] kubectl top: add a --show-swap option kubernetes#129458
  • Docs (k/website) update(s):
    • 2400: draft update for swap KEP website#49945
• GA (related PRs, can be assigned to a specific track later)
  • KEP (k/enhancements) update PR(s):
  • Issues and Code (k/k) update PR(s):
    • [KEP-2400] [Swap] [Discussion]: Add swap as a ResourceName for pod spec resources API kubernetes#120801
    • [KEP-2400] [Swap]: Verify memory pressure behavior with swap enabled kubernetes#120800
    • [KEP-2400] [Swap]: SwapConformance e2e testing kubernetes#120798.
    • [KEP-2400] Address swap impact on memory-backed volumes kubernetes#105978
    • Add a test for OOMKilled for LimitedSwap kubernetes#121820
    • Add the ability to set a system-reserved quantity of swap on a node
      • [KEP-2400] [Swap] Add the ability to set a system-reserved quantity of swap on a node kubernetes#120802
      • [KEP-2400] Add the ability to set a system-reserved quantity of swap on a node kubernetes#105019 dup by 120802
      • add --system-reserved support for swap(cgroup v2 only): only for burstable pods kubernetes#105271
    • Fail Kubelet at startup if swap is configured with cgroup v1 kubernetes#119327
    • [Swap Feature] Investigate what happens if a cluster has some swap enabled nodes and others that are not enabled. kubernetes#122279
  • Docs (k/website) update(s):

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

/sig node

[karan]

@cookieisaac

[ehashman]

/stage alpha
/milestone v1.22

[jrsapi]

Greetings @ehashman!
Enhancement shadow checking in with a few reminders. 1.22 Enhancements Freeze starts at 23:59:59 pst on Thursday, May 13. A few items needing review for this KEP:

• Enhancement README

• PRR approved

• Node memory swap support #2400 updated with current target and alpha information.

Thanks!

[ehashman]

All of this is covered in #2602

[jrsapi]

Greetings @ehashman!
Thanks for the follow-up. After reviewing the KEP and PRR everything looks on target. The enhancement is marked at risk but once merged we can move its status to tracked. A reminder that tomorrow 5/13 the Enhancement freeze starts at 23:59:59 PST.

Thanks!

[ehashman]

Work breakdown for 1.22

• Documentation for enabling swap (TBD - @ehashman?)
• CI environment/test updates (@ike-ma)
  • Build images with swap for 2 Linux distros
  • Add jobs to test-infra that use the images and enable the swap feature flag/kubelet option for node e2e suite
  • See also Test Plan
• CRI and kubelet changes (@ehashman)
  • Complete API changes per KEP
  • Add e2e tests as appropriate

External to k8s but still need to happen:

• Containerd update to use new CRI
• CRI-O update to use new CRI

Once the above CRI updates happen, ensure that CI environment is using latest container runtimes with updated CRI.

[jrsapi]

Greetings @ehashman ,
Enhancement shadow checking with a reminder that we are 2 weeks away from code freeze (July 8, 2021). Can you like the k/k PR/s that are needed to implement this enhancement for the 1.22 milestone?

[jrsapi]

Greetings @ehashman,
A friendly reminder that code freeze is this Thursday, July 8th and we're tracking the following k/k PR:

• Alpha node swap support kubernetes#102823
  The PR has to be merged by code freeze and currently, this KEP is marked "At Risk".
  Also, Docs PR placeholder deadline is Friday, July 9th. Here are instructions on opening a PR in dev-1.22 branch.

Thanks!

[ehashman]

PR just merged. Docs placeholder is kubernetes/website#28838

[jrsapi]

Awesome! Thanks for the update. Moving this to "Tracked".

[ehashman]

We are good to go for 1.22! Docs complete.

[ehashman]

/milestone v1.23

[ehashman]

/stage beta

[dipesh-rawat]

Hello @iholder101 @kannon92 @pacoxu 👋, v1.33 Enhancements team here.

Just checking in as we approach enhancements freeze on 02:00 UTC Friday 14th February 2025 / 19:00 PDT Thursday 13th February 2025.

This enhancement is targeting stage beta for v1.33 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: v1.33.
• KEP readme has up-to-date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here). If your production readiness review is not completed yet, please make sure to fill the production readiness questionnaire in your KEP by the PRR Freeze deadline on Thursday 6th February 2025 so that the PRR team has enough time to review your KEP.

For this KEP, we would need to update the following:

• Please ensure the KEP readme using the latest template and merge it in the k/enhancements repo. Currently, I’ve noticed the following discrepancies compared to the latest template:

  • Under the "Release Signoff Checklist" section, few checklist items are missing currently in the KEP README.md.

• Update the KEP README to include graduation criteria (here) for the Beta 3 stage based on Node memory swap support #2400 (comment).

• Update the kep.yaml to reflect the following:

  • Add latest-milestone: v1.33.
  • Specify beta: "v1.33"

• Also, please make sure to update the issue description with the correct stages and PRs. This will assist the release team in tracking progress more effectively.

The status of this enhancement is marked as At risk for enhancements freeze. Please keep the issue description up-to-date with appropriate stages as well.

If you anticipate missing enhancements freeze, you can file an exception request in advance. Thank you!

[dipesh-rawat]

Hi @iholder101 @kannon92 @pacoxu 👋, 1.33 Enhancements team here,

Just a quick friendly reminder as we approach the enhancements freeze later this week, at 02:00 UTC Friday 14th February 2025 / 19:00 PDT Thursday 13th February 2025.

The current status of this enhancement is marked as At risk for enhancement freeze. There are a few requirements mentioned in the comment #2400 (comment) that still need to be completed.

If you anticipate missing enhancements freeze, you can file an exception request in advance. Thank you!

[dipesh-rawat]

Hello @iholder101 @kannon92 @pacoxu 👋, v1.33 Enhancements team here.

Unfortunately, this enhancement did not meet requirements for enhancements freeze.

If you still wish to progress this enhancement in v1.33, please file an exception request as soon as possible, within three days. If you have any questions, you can reach out in the #release-enhancements channel on Slack and we'll be happy to help. Thanks!

Missing section from the KEP README

• Under the "Release Signoff Checklist" section, few checklist items are missing currently in the KEP README.md.

/milestone clear

[haircommander]

It looks like the update PR merged @dipesh-rawat are we missing anything here?

[dipesh-rawat]

@haircommander The KEP README has few checklist items missing under the 'Release Signoff Checklist' section based on the latest template (here).

  - [ ] e2e Tests for all Beta API Operations (endpoints)
  - [ ] (R) Ensure GA e2e tests meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) 
  - [ ] (R) Minimum Two Week Window for GA e2e tests to prove flake free

[haircommander]

thanks @dipesh-rawat we missed that. I opened #5174 and filed an extension. if that merges, will this KEP be tracked?

[dipesh-rawat]

@haircommander Thanks a lot for raising the PR #5174 promptly to fix the outstanding issue. Yes, since the exception request is approved, once the PR #5174 is merged, we can add this to the milestone and track it for the 1.33 release.

[haircommander]

@dipesh-rawat PR merged, all set here?

[dipesh-rawat]

@haircommander (and @iholder101 @kannon92 @pacoxu) Now that PRs #5174 has been merged, all the KEP requirements are in place and merged into k/enhancements, this enhancement is all good for the upcoming enhancements freeze. 🚀

The status of this enhancement is marked as tracked for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

/milestone v1.33
/label tracked/yes

[ArvindParekh]

Hello, @iholder101 @kannon92 @haircommander @pacoxu 👋, v1.33 Docs Shadow here.

Does this enhancement work planned for v1.33 require any new docs or modifications to existing docs?

If so, please follow the steps here to open a PR against the dev-1.33 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday 27th February 2025 18:00 PDT.

Also, take a look at Documenting for a release to get yourself familiarized with the docs requirement for the release.

Thank you!

[rayandas]

Hi @iholder101 @kannon92 @haircommander @pacoxu 👋 v1.33 Docs Lead here.

Gentle reminder! The deadline to raise a placeholder Docs PR is Thursday 27th February 2025 18:00 PDT.
If this enhancement work requires any new docs or modification to existing docs, please create a placeholder PR (it can be a draft PR for now) before the deadline.

Thanks!

[haircommander]

sorry for the delay, I've opened kubernetes/website#49945

[rayandas]

Thanks @haircommander

[aibarbetta]

Hi @iholder101 @kannon92 @pacoxu @haircommander 👋 -- this is Agustina (@aibarbetta) from the 1.33 Communications Team!

For the 1.33 release, we are currently in the process of collecting and curating a list of potential feature blogs, and we'd love for you to consider writing one for your enhancement!

As you may be aware, feature blogs are a great way to communicate to users about features which fall into (but not limited to) the following categories:

• This introduces some breaking change(s)
• This has significant impacts and/or implications to users
• ...Or this is a long-awaited feature, which would go a long way to cover the journey more in detail 🎉

To opt in to write a feature blog, could you please let us know and open a "Feature Blog placeholder PR" (which can be only a skeleton at first) against the website repository by Wednesday, 5th March, 2025? For more information about writing a blog, please find the blog contribution guidelines 📚

Tip

Some timeline to keep in mind:

• 02:00 UTC Wednesday, 5th March, 2025: Feature blog PR freeze
• Monday, 7th April, 2025: Feature blogs ready for review
• You can find more in the release document

Note

In your placeholder PR, use XX characters for the blog date in the front matter and file name. We will work with you on updating the PR with the publication date once we have a final number of feature blogs for this release.

[dipesh-rawat]

Hey again @iholder101 @kannon92 @pacoxu 👋, 1.33 Enhancements team here,

Just checking in as we approach code freeze at 02:00 UTC Friday 21st March 2025 / 19:00 PDT Thursday 20th March 2025. .

Here's where this enhancement currently stands:

• All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
• All PRs are ready to be merged (they have approved and lgtm labels applied) by the code freeze deadline. This includes tests.

For this enhancement, it looks like the following PRs need to be merged before code freeze (and we need to update the Issue description to include all the related PRs of this KEP):

• [KEP-2400] [failing-test] resource metrics e2e tests: expect swap node and container level stats kubernetes#130053
• [KEP-2400] [Bugfix]: Ensure container-level swap metrics are collected kubernetes#129486
• [KEP-2400] Report swap capacity as part of node.status.nodeSystemInfo kubernetes#129954
• [KEP-2400]: Swap-aware memory eviction (with no additional APIs) kubernetes#129578
• [KEP-2400] kubectl top: add a --show-swap option kubernetes#129458

Additionally, please let me know if there are any other PRs in k/k not listed above that we should track for this KEP, so that we can maintain accurate status.

If you anticipate missing code freeze, you can file an exception request in advance.

The status of this enhancement is marked as at risk for code freeze.

[aibarbetta]

Hi @iholder101 @kannon92 @pacoxu 👋, 1.33 Communications Team here again!

This is a gentle reminder for the feature blog deadline mentioned above, which is 02:00 UTC Wednesday, 5th March, 2025. I see a blog was written when this feature was in beta, has anything changed? would you like to write about it? To opt in, please let us know and open a Feature Blog placeholder PR against k/website by the deadline. If you have any questions, please feel free to reach out to us!

Tip

Some timeline to keep in mind:

• 02:00 UTC Wednesday, 5th March, 2025: Feature blog PR freeze
• Monday, 7th April, 2025: Feature blogs ready for review
• You can find more in the release document

Note

In your placeholder PR, use XX characters for the blog date in the front matter and file name. We will work with you on updating the PR with the publication date once we have a final number of feature blogs for this release.

[sathieu]

Hello,

The docs reads:

When the memory swap feature is turned on, Kubernetes data such as the content of Secret objects that were written to tmpfs now could be swapped to disk.

Will this limitation be removed in the future? (using mlockall or other)

[jabdoa2]

Hello,

The docs reads:

When the memory swap feature is turned on, Kubernetes data such as the content of Secret objects that were written to tmpfs now could be swapped to disk.

Will this limitation be removed in the future? (using mlockall or other)

This is how tmpfs works inside the Linux kernel. If you really don't want tmpfs to swap you could use ramfs instead but I personally would not recommend that. If you are worried about somebody steeling your swap disk you might want to enable encrypted swap instead. I guess those are underlying things in Linux. Swap behaviour of tmpfs is not a Kubernetes thing - this is the default Linux behaviour which we just did not have due to disabled swap.

[iholder101]

Hello,

The docs reads:

When the memory swap feature is turned on, Kubernetes data such as the content of Secret objects that were written to tmpfs now could be swapped to disk.

Will this limitation be removed in the future? (using mlockall or other)

Hey @sathieu!
In order to solve this problem, kubelet will mount the tmpfs volumes with the noswap option.
This option is fairly new. The upstream Linux kernel supports it from version >= 6.3, but many distros backported the option to much older versions. Most probably, if you're node's kernel is updated, you won't be affected by this problem.

For more info, see implementation here: kubernetes/kubernetes#124060.
And blog-post (that will be released very soon) here: https://github.com/kubernetes/website/pull/48099/files.