Skip to content

Conversation

@tallclair
Copy link
Member

What type of PR is this?

/kind feature
/kind api-change

What this PR does / why we need it:

  1. Don't allow memory limits to be decreased without restart (per https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/1287-in-place-update-pod-resources/README.md#memory-limit-decreases)
  2. ResizePolicy is no longer mutable

Does this PR introduce a user-facing change?

InPlacePodVerticalScaling: Memory limits cannot be decreased unless the memory resize restart policy is set to `RestartContainer`. Container resizePolicy is no longer mutable.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

/sig node
/milestone v1.33
/priority important-soon

@k8s-ci-robot k8s-ci-robot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Feb 14, 2025
@k8s-ci-robot k8s-ci-robot added this to the v1.33 milestone Feb 14, 2025
@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. kind/feature Categorizes issue or PR as related to a new feature. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API sig/node Categorizes an issue or PR as relevant to SIG Node. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Feb 14, 2025
@tallclair
Copy link
Member Author

/assign @thockin

@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added the sig/apps Categorizes an issue or PR as relevant to SIG Apps. label Feb 14, 2025
@k8s-triage-robot
Copy link

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

@tallclair tallclair changed the title Forbid memory limit decrease [FG:InPlacePodVerticalScaling] Forbid memory limit decrease Feb 14, 2025
Copy link
Member

@thockin thockin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

/lgtm
/approve

}
}

// TweakContainers applies the container tweaks to all containers in the pod with a masked type.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does "a masked type" mean here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, this was originally using podutil.VisitContainers, which take a container type mask, but I had to remove that due to a circular dependency, and forgot to update the comment.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 20, 2025
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: 85fc1ab139746c143447771de3d0f950297d3126

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tallclair, thockin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 20, 2025
@k8s-ci-robot k8s-ci-robot merged commit f90682b into kubernetes:master Feb 20, 2025
13 checks passed
@pacoxu
Copy link
Member

pacoxu commented Feb 21, 2025

https://testgrid.k8s.io/sig-release-master-blocking#gce-cos-master-alpha-features failed after this?
052d7a5...9bf60d0

E2E test may need some updates.

Comment on lines +5745 to +5770
// Special case: memory limits may not be decreased if resize policy is NotRequired.
var memRestartPolicy core.ResourceResizeRestartPolicy
for _, policy := range resizePolicies {
if policy.ResourceName == core.ResourceMemory {
memRestartPolicy = policy.RestartPolicy
break
}
}
if memRestartPolicy == core.NotRequired || memRestartPolicy == "" {
newLimit, hasNewLimit := newRequirements.Limits[core.ResourceMemory]
oldLimit, hasOldLimit := oldRequirements.Limits[core.ResourceMemory]
if hasNewLimit && hasOldLimit {
if newLimit.Cmp(oldLimit) < 0 {
allErrs = append(allErrs, field.Forbidden(
fldPath.Child("limits").Key(core.ResourceMemory.String()),
fmt.Sprintf("memory limits cannot be decreased unless resizePolicy is %s", core.RestartContainer)))
}
} else if hasNewLimit && !hasOldLimit {
// Adding a memory limit is implicitly decreasing the memory limit (from 'max')
allErrs = append(allErrs, field.Forbidden(
fldPath.Child("limits").Key(core.ResourceMemory.String()),
fmt.Sprintf("memory limits cannot be added unless resizePolicy is %s", core.RestartContainer)))
}
}

// TODO(tallclair): Move resizable resource checks here.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is breaking existing e2e tests

https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-kubernetes-e2e-gci-gce-alpha-features/1892735773371797504

 failed [FAILED] failed to patch pod for resize: Pod "resize-test-262pc" is invalid: spec.initContainers[0].resources.limits[memory]: Forbidden: memory limits cannot be decreased unless resizePolicy is RestartContainer
In [It] at: k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1334 @ 02/21/25 00:49:20.398

@tallclair
Copy link
Member Author

Thanks for flagging the test failures. I'm looking into it.

@tallclair
Copy link
Member Author

Tests fixed in #130357

@aojea
Copy link
Member

aojea commented Feb 24, 2025

Tests fixed in #130357

Thanks for the fast response

@NoicFank
Copy link
Contributor

@tallclair
Hello, for the same container, may I ask how to increase container memory limit without restartContainer, and decrease container limit with restartContainer.

It seems I have to mutate restartPolicy for increase/decrease memory limit scenario.

@tallclair
Copy link
Member Author

@tallclair Hello, for the same container, may I ask how to increase container memory limit without restartContainer, and decrease container limit with restartContainer.

It seems I have to mutate restartPolicy for increase/decrease memory limit scenario.

Unfortunately that's not possible today. The good news is that we're planning to lift this restriction in v1.34, and allow memory limit decreases.

@NoicFank
Copy link
Contributor

@tallclair Hello, for the same container, may I ask how to increase container memory limit without restartContainer, and decrease container limit with restartContainer.
It seems I have to mutate restartPolicy for increase/decrease memory limit scenario.

Unfortunately that's not possible today. The good news is that we're planning to lift this restriction in v1.34, and allow memory limit decreases.

OK, thanks for reply ~

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API kind/feature Categorizes issue or PR as related to a new feature. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/node Categorizes an issue or PR as relevant to SIG Node. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

Archived in project
Archived in project

Development

Successfully merging this pull request may close these issues.

7 participants