test: add e2e test case for nodepool upgrades#4505
Merged
openshift-merge-bot[bot] merged 8 commits intoMar 26, 2026
Merged
Conversation
|
Skipping CI for Draft Pull Request. |
machi1990
commented
Mar 18, 2026
Collaborator
Author
|
/test e2e-parallel |
Collaborator
Author
d5a8cce to
9e4e5bd
Compare
Collaborator
Author
|
/test e2e-parallel |
JameelB
reviewed
Mar 19, 2026
machi1990
commented
Mar 19, 2026
|
|
||
| var _ = Describe("Customer", func() { | ||
| DescribeTable("should upgrade a nodepool from", | ||
| func(ctx context.Context, nodePoolMinor string, nextMinor string) { |
Collaborator
Author
There was a problem hiding this comment.
A followup is ensuring that the sample app remains available during the upgrade
Collaborator
Author
|
/test e2e-parallel |
d63d1c5 to
99239be
Compare
machi1990
added a commit
to machi1990/ARO-HCP
that referenced
this pull request
Mar 21, 2026
…nodepool creation While working on Azure#4505 and further reviewing of Azure#4554 I realised that the frontend was missing skew validation while creating the node pool. The PR adds the skew validation by performing the following rules: - if the version is set - validate that node pool desired minor version is within the maximum of 2 minors lower skew and not higher than the control plane minor - validate that the desired node pool version isn't higher that the lowest active control plane version if this is available: only available post install - validate that when control plane major != nodepool major version then only valid skews are allowed - for now the definition of valid skew is: if cp on 5.0 then the node pool can be on 4.21 or 4.22 if cp is on 5.1 then the node pool can be on 4.21, 4.22 or 4.23 if cp is on 5.2 then the node pool can be only on 4.23 I left a TODO to put this behind an AFEC flag once Azure#4479 is merged and it'll done as a followup - if the lowest control plane version is available and node pool version set, perform the same minor skew as done for the cluster desired version but using the actual cluster version minor The change also introduces some defaulting logic in case version isn't set: in this case we default to the lowest active control plane version.
machi1990
added a commit
to machi1990/ARO-HCP
that referenced
this pull request
Mar 21, 2026
…nodepool creation While working on Azure#4505 and further reviewing of Azure#4554 I realised that the frontend was missing skew validation while creating the node pool. The PR adds the skew validation by performing the following rules: - if the version is set - validate that node pool desired minor version is within the maximum of 2 minors lower skew and not higher than the control plane minor - validate that the desired node pool version isn't higher that the lowest active control plane version if this is available: only available post install - validate that when control plane major != nodepool major version then only valid skews are allowed - for now the definition of valid skew is: if cp on 5.0 then the node pool can be on 4.21 or 4.22 if cp is on 5.1 then the node pool can be on 4.21, 4.22 or 4.23 if cp is on 5.2 then the node pool can be only on 4.23 I left a TODO to put this behind an AFEC flag once Azure#4479 is merged and it'll done as a followup - if the lowest control plane version is available and node pool version set, perform the same minor skew as done for the cluster desired version but using the actual cluster version minor The change also introduces some defaulting logic in case version isn't set: in this case we default to the lowest active control plane version.
machi1990
added a commit
to machi1990/ARO-HCP
that referenced
this pull request
Mar 21, 2026
…nodepool creation While working on Azure#4505 and further reviewing of Azure#4554 I realised that the frontend was missing skew validation while creating the node pool. The PR adds the skew validation by performing the following rules: - if the version is set - validate that node pool desired minor version is within the maximum of 2 minors lower skew and not higher than the control plane minor - validate that the desired node pool version isn't higher that the lowest active control plane version if this is available: only available post install - validate that when control plane major != nodepool major version then only valid skews are allowed - for now the definition of valid skew is: if cp on 5.0 then the node pool can be on 4.21 or 4.22 if cp is on 5.1 then the node pool can be on 4.21, 4.22 or 4.23 if cp is on 5.2 then the node pool can be only on 4.23 I left a TODO to put this behind an AFEC flag once Azure#4479 is merged and it'll done as a followup - if the lowest control plane version is available and node pool version set, perform the same minor skew as done for the cluster desired version but using the actual cluster version minor The change also introduces some defaulting logic in case version isn't set: in this case we default to the lowest active control plane version.
99239be to
0e88194
Compare
cgiradkar
pushed a commit
to machi1990/ARO-HCP
that referenced
this pull request
Mar 25, 2026
…nodepool creation While working on Azure#4505 and further reviewing of Azure#4554 I realised that the frontend was missing skew validation while creating the node pool. The PR adds the skew validation by performing the following rules: - if the version is set - validate that node pool desired minor version is within the maximum of 2 minors lower skew and not higher than the control plane minor - validate that the desired node pool version isn't higher that the lowest active control plane version if this is available: only available post install - validate that when control plane major != nodepool major version then only valid skews are allowed - for now the definition of valid skew is: if cp on 5.0 then the node pool can be on 4.21 or 4.22 if cp is on 5.1 then the node pool can be on 4.21, 4.22 or 4.23 if cp is on 5.2 then the node pool can be only on 4.23 I left a TODO to put this behind an AFEC flag once Azure#4479 is merged and it'll done as a followup - if the lowest control plane version is available and node pool version set, perform the same minor skew as done for the cluster desired version but using the actual cluster version minor The change also introduces some defaulting logic in case version isn't set: in this case we default to the lowest active control plane version.
cgiradkar
pushed a commit
to machi1990/ARO-HCP
that referenced
this pull request
Mar 25, 2026
…nodepool creation While working on Azure#4505 and further reviewing of Azure#4554 I realised that the frontend was missing skew validation while creating the node pool. The PR adds the skew validation by performing the following rules: - if the version is set - validate that node pool desired minor version is within the maximum of 2 minors lower skew and not higher than the control plane minor - validate that the desired node pool version isn't higher that the lowest active control plane version if this is available: only available post install - validate that when control plane major != nodepool major version then only valid skews are allowed - for now the definition of valid skew is: if cp on 5.0 then the node pool can be on 4.21 or 4.22 if cp is on 5.1 then the node pool can be on 4.21, 4.22 or 4.23 if cp is on 5.2 then the node pool can be only on 4.23 I left a TODO to put this behind an AFEC flag once Azure#4479 is merged and it'll done as a followup - if the lowest control plane version is available and node pool version set, perform the same minor skew as done for the cluster desired version but using the actual cluster version minor The change also introduces some defaulting logic in case version isn't set: in this case we default to the lowest active control plane version.
The strategy employed here is: - create a control plane at a minor say X.Y - find the latest z in the previous minor call it X.Y-1.z and install the nodepool on it - find the latest release images for the nodes after nodepool installed and call them previous images - find the lowest installed control plane version on the version history and use it as the target to find the next nodepool version to upgrade to - trigger the nodepool upgrade - verify that the upgrade was successfully done by: - waiting for the nodes to be ready - waiting for the release images of the new nodes to change - for the nodes to be installed on the X.Y control plane minor Special notes: The verification was done that was because the Node CR didn't include the version X.Y.Z information that'd have let us simply do "targetVersion==NodeObservedVersion". Only the kubelet version is available which isn't what we want to verify here. The current ocpVersion of nodepool is only available in the NodePool CR but that is a service only CR and not accessible from within the HCP guest cluster but only the management cluster.
Collaborator
Author
|
There has been some new tests merged. I've to rebase |
strategy: - it install the cluster to latest - find and install the nodepool to an older patch release - find the latest patch release to which we can upgrade to and perform the upgrade - verify the upgrade was successfully
…performs the verification
Full node objects can be 10KB+ due to annotations and cause Gomega to truncate output when assertions fail. Replace full node JSON dump with a compact nodeSummary struct containing only relevant fields: name, ready status, containerRuntimeVersion, and release images.
Skip because (Azure marketplace image missing in CS) pending work in https://redhat.atlassian.net/browse/ARO-24955
1b7f8ae to
ec8c17a
Compare
Collaborator
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: ahitacat, cgiradkar, JameelB, machi1990, miquelsi, mvacula02 The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
cgiradkar
pushed a commit
to machi1990/ARO-HCP
that referenced
this pull request
Mar 26, 2026
…nodepool creation While working on Azure#4505 and further reviewing of Azure#4554 I realised that the frontend was missing skew validation while creating the node pool. The PR adds the skew validation by performing the following rules: - if the version is set - validate that node pool desired minor version is within the maximum of 2 minors lower skew and not higher than the control plane minor - validate that the desired node pool version isn't higher that the lowest active control plane version if this is available: only available post install - validate that when control plane major != nodepool major version then only valid skews are allowed - for now the definition of valid skew is: if cp on 5.0 then the node pool can be on 4.21 or 4.22 if cp is on 5.1 then the node pool can be on 4.21, 4.22 or 4.23 if cp is on 5.2 then the node pool can be only on 4.23 I left a TODO to put this behind an AFEC flag once Azure#4479 is merged and it'll done as a followup - if the lowest control plane version is available and node pool version set, perform the same minor skew as done for the cluster desired version but using the actual cluster version minor The change also introduces some defaulting logic in case version isn't set: in this case we default to the lowest active control plane version.
cgiradkar
pushed a commit
to machi1990/ARO-HCP
that referenced
this pull request
Mar 27, 2026
…nodepool creation While working on Azure#4505 and further reviewing of Azure#4554 I realised that the frontend was missing skew validation while creating the node pool. The PR adds the skew validation by performing the following rules: - if the version is set - validate that node pool desired minor version is within the maximum of 2 minors lower skew and not higher than the control plane minor - validate that the desired node pool version isn't higher that the lowest active control plane version if this is available: only available post install - validate that when control plane major != nodepool major version then only valid skews are allowed - for now the definition of valid skew is: if cp on 5.0 then the node pool can be on 4.21 or 4.22 if cp is on 5.1 then the node pool can be on 4.21, 4.22 or 4.23 if cp is on 5.2 then the node pool can be only on 4.23 I left a TODO to put this behind an AFEC flag once Azure#4479 is merged and it'll done as a followup - if the lowest control plane version is available and node pool version set, perform the same minor skew as done for the cluster desired version but using the actual cluster version minor The change also introduces some defaulting logic in case version isn't set: in this case we default to the lowest active control plane version.
cgiradkar
pushed a commit
to machi1990/ARO-HCP
that referenced
this pull request
Mar 30, 2026
…nodepool creation While working on Azure#4505 and further reviewing of Azure#4554 I realised that the frontend was missing skew validation while creating the node pool. The PR adds the skew validation by performing the following rules: - if the version is set - validate that node pool desired minor version is within the maximum of 2 minors lower skew and not higher than the control plane minor - validate that the desired node pool version isn't higher that the lowest active control plane version if this is available: only available post install - validate that when control plane major != nodepool major version then only valid skews are allowed - for now the definition of valid skew is: if cp on 5.0 then the node pool can be on 4.21 or 4.22 if cp is on 5.1 then the node pool can be on 4.21, 4.22 or 4.23 if cp is on 5.2 then the node pool can be only on 4.23 I left a TODO to put this behind an AFEC flag once Azure#4479 is merged and it'll done as a followup - if the lowest control plane version is available and node pool version set, perform the same minor skew as done for the cluster desired version but using the actual cluster version minor The change also introduces some defaulting logic in case version isn't set: in this case we default to the lowest active control plane version.
cgiradkar
pushed a commit
to machi1990/ARO-HCP
that referenced
this pull request
Mar 31, 2026
…nodepool creation While working on Azure#4505 and further reviewing of Azure#4554 I realised that the frontend was missing skew validation while creating the node pool. The PR adds the skew validation by performing the following rules: - if the version is set - validate that node pool desired minor version is within the maximum of 2 minors lower skew and not higher than the control plane minor - validate that the desired node pool version isn't higher that the lowest active control plane version if this is available: only available post install - validate that when control plane major != nodepool major version then only valid skews are allowed - for now the definition of valid skew is: if cp on 5.0 then the node pool can be on 4.21 or 4.22 if cp is on 5.1 then the node pool can be on 4.21, 4.22 or 4.23 if cp is on 5.2 then the node pool can be only on 4.23 I left a TODO to put this behind an AFEC flag once Azure#4479 is merged and it'll done as a followup - if the lowest control plane version is available and node pool version set, perform the same minor skew as done for the cluster desired version but using the actual cluster version minor The change also introduces some defaulting logic in case version isn't set: in this case we default to the lowest active control plane version.
cgiradkar
pushed a commit
to machi1990/ARO-HCP
that referenced
this pull request
Apr 2, 2026
…nodepool creation While working on Azure#4505 and further reviewing of Azure#4554 I realised that the frontend was missing skew validation while creating the node pool. The PR adds the skew validation by performing the following rules: - if the version is set - validate that node pool desired minor version is within the maximum of 2 minors lower skew and not higher than the control plane minor - validate that the desired node pool version isn't higher that the lowest active control plane version if this is available: only available post install - validate that when control plane major != nodepool major version then only valid skews are allowed - for now the definition of valid skew is: if cp on 5.0 then the node pool can be on 4.21 or 4.22 if cp is on 5.1 then the node pool can be on 4.21, 4.22 or 4.23 if cp is on 5.2 then the node pool can be only on 4.23 I left a TODO to put this behind an AFEC flag once Azure#4479 is merged and it'll done as a followup - if the lowest control plane version is available and node pool version set, perform the same minor skew as done for the cluster desired version but using the actual cluster version minor The change also introduces some defaulting logic in case version isn't set: in this case we default to the lowest active control plane version.
cgiradkar
pushed a commit
to machi1990/ARO-HCP
that referenced
this pull request
Apr 2, 2026
…nodepool creation While working on Azure#4505 and further reviewing of Azure#4554 I realised that the frontend was missing skew validation while creating the node pool. The PR adds the skew validation by performing the following rules: - if the version is set - validate that node pool desired minor version is within the maximum of 2 minors lower skew and not higher than the control plane minor - validate that the desired node pool version isn't higher that the lowest active control plane version if this is available: only available post install - validate that when control plane major != nodepool major version then only valid skews are allowed - for now the definition of valid skew is: if cp on 5.0 then the node pool can be on 4.21 or 4.22 if cp is on 5.1 then the node pool can be on 4.21, 4.22 or 4.23 if cp is on 5.2 then the node pool can be only on 4.23 I left a TODO to put this behind an AFEC flag once Azure#4479 is merged and it'll done as a followup - if the lowest control plane version is available and node pool version set, perform the same minor skew as done for the cluster desired version but using the actual cluster version minor The change also introduces some defaulting logic in case version isn't set: in this case we default to the lowest active control plane version.
cgiradkar
pushed a commit
to machi1990/ARO-HCP
that referenced
this pull request
Apr 7, 2026
…nodepool creation While working on Azure#4505 and further reviewing of Azure#4554 I realised that the frontend was missing skew validation while creating the node pool. The PR adds the skew validation by performing the following rules: - if the version is set - validate that node pool desired minor version is within the maximum of 2 minors lower skew and not higher than the control plane minor - validate that the desired node pool version isn't higher that the lowest active control plane version if this is available: only available post install - validate that when control plane major != nodepool major version then only valid skews are allowed - for now the definition of valid skew is: if cp on 5.0 then the node pool can be on 4.21 or 4.22 if cp is on 5.1 then the node pool can be on 4.21, 4.22 or 4.23 if cp is on 5.2 then the node pool can be only on 4.23 I left a TODO to put this behind an AFEC flag once Azure#4479 is merged and it'll done as a followup - if the lowest control plane version is available and node pool version set, perform the same minor skew as done for the cluster desired version but using the actual cluster version minor The change also introduces some defaulting logic in case version isn't set: in this case we default to the lowest active control plane version.
cgiradkar
pushed a commit
to machi1990/ARO-HCP
that referenced
this pull request
Apr 13, 2026
…nodepool creation While working on Azure#4505 and further reviewing of Azure#4554 I realised that the frontend was missing skew validation while creating the node pool. The PR adds the skew validation by performing the following rules: - if the version is set - validate that node pool desired minor version is within the maximum of 2 minors lower skew and not higher than the control plane minor - validate that the desired node pool version isn't higher that the lowest active control plane version if this is available: only available post install - validate that when control plane major != nodepool major version then only valid skews are allowed - for now the definition of valid skew is: if cp on 5.0 then the node pool can be on 4.21 or 4.22 if cp is on 5.1 then the node pool can be on 4.21, 4.22 or 4.23 if cp is on 5.2 then the node pool can be only on 4.23 I left a TODO to put this behind an AFEC flag once Azure#4479 is merged and it'll done as a followup - if the lowest control plane version is available and node pool version set, perform the same minor skew as done for the cluster desired version but using the actual cluster version minor The change also introduces some defaulting logic in case version isn't set: in this case we default to the lowest active control plane version.
cgiradkar
pushed a commit
to machi1990/ARO-HCP
that referenced
this pull request
Apr 15, 2026
…nodepool creation While working on Azure#4505 and further reviewing of Azure#4554 I realised that the frontend was missing skew validation while creating the node pool. The PR adds the skew validation by performing the following rules: - if the version is set - validate that node pool desired minor version is within the maximum of 2 minors lower skew and not higher than the control plane minor - validate that the desired node pool version isn't higher that the lowest active control plane version if this is available: only available post install - validate that when control plane major != nodepool major version then only valid skews are allowed - for now the definition of valid skew is: if cp on 5.0 then the node pool can be on 4.21 or 4.22 if cp is on 5.1 then the node pool can be on 4.21, 4.22 or 4.23 if cp is on 5.2 then the node pool can be only on 4.23 I left a TODO to put this behind an AFEC flag once Azure#4479 is merged and it'll done as a followup - if the lowest control plane version is available and node pool version set, perform the same minor skew as done for the cluster desired version but using the actual cluster version minor The change also introduces some defaulting logic in case version isn't set: in this case we default to the lowest active control plane version.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Special notes:
What
test: add e2e test case for nodepool upgrades
Why
To verify the feature working end to end.
The strategy employed here for minor upgrade is is:
strategy for z-stream upgrade is :
- it install the cluster to latest
- find and install the nodepool to an older patch release
- find the latest patch release to which we can upgrade to and perform the upgrade
- verify the upgrade was successfully
To verify that the upgrade was successfully we do:
Special notes for your reviewer