Skip to content

Fix flaky VerifyProfileTest#testCustomValidationLastName - #55173

Open
ahus1 wants to merge 3 commits into
keycloak:mainfrom
ahus1:is-55172-flaky-verify-profile-test
Open

ahus1 wants to merge 3 commits into
keycloak:mainfrom
ahus1:is-55172-flaky-verify-profile-test

Conversation

@ahus1

@ahus1 ahus1 commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Closes #55172

The failure had two independent causes, so both are fixed here: the shared OAuth callback wait matched session_code= as if it were code=, and the profile page object returned from a submit before the browser had replaced the page, so the next assertion or submit could act on the old page.

Decisions and assumptions

  • The wait in WaitUtils now matches whole parameter names (?, & or # followed by code= or error=). This changes behavior for all tests calling parseLoginResponse(), which I assume is intended, as the old match could only succeed early by accident.
  • The synchronization lives in LoginUpdateProfilePage.Update.submit() and not in the individual tests, so all callers benefit. It waits for the submitted button to go stale, treating Chrome's "does not belong to the document" WebDriverException like Firefox's StaleElementReferenceException, and then for document.readyState to be complete. A stale element alone is not enough, as the new document can be committed but not yet parsed.
  • This also fixes OrganizationSharedIdpBrokerTest.testAuthNotePreservedOverridesEmailDomain on Firefox, which read the page source of a not yet loaded page and failed on every local run. It needed no change to the test.
  • Callers keep waitForOAuthCallback() and assertCurrent() as they assert the outcome, not only the timing.
  • Verified locally with headless Firefox and Chrome: VerifyProfileTest, RequiredActionUpdateProfileTest and OrganizationSharedIdpBrokerTest on Firefox pass. On Chrome, OrganizationSharedIdpBrokerTest and OrganizationInvitationTokenTest still fail with and without this change, due to similar races right after other page submits (for example reading getCurrentUrl() immediately after the username submit). I left those out of scope.
  • release/26.8 is the only supported branch with the migrated test and the faulty wait, so only that backport is needed.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The validation assertion can pass on the old page’s existing last-name error, so the test must wait for the submitted form to become stale to resolve the invalid-then-valid submission race.

1 open issue
What changed in this PR

This PR addresses the Firefox flakiness reported in #55172 by tightening synchronization in the profile-validation test and the shared OAuth callback wait.

Changes:

  • Adds a last-name validation-error assertion before resubmitting the profile form.
  • Matches complete code and error parameter names so session_code does not trigger the callback wait.
File Description
tests/​base/​src/​test/​java/​org/​keycloak/​tests/​forms/​VerifyProfileTest.java Checks for a validation error before the valid submission.
test-framework/​ui/​src/​main/​java/​org/​keycloak/​testframework/​ui/​webdriver/​WaitUtils.java Tightens OAuth callback parameter matching.

Review effort: Balanced

@keycloak-github-bot keycloak-github-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unreported flaky test detected, please review

@keycloak-github-bot

Copy link
Copy Markdown

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.actions.TermsAndConditionsTest#termsAccepted

Keycloak CI - Forms IT (firefox)

java.lang.RuntimeException: Could not create statement
	at org.jboss.arquillian.junit.Arquillian.methodBlock(Arquillian.java:307)
	at org.jboss.arquillian.junit.Arquillian$2.evaluate(Arquillian.java:160)
	at org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:344)
	at org.jboss.arquillian.junit.Arquillian.access$200(Arquillian.java:48)
...

Report flaky test

Closes keycloak#55172

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>
@ahus1
ahus1 force-pushed the is-55172-flaky-verify-profile-test branch from 839b180 to e9a6c2b Compare November 13, 2026 17:45

@keycloak-github-bot keycloak-github-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unreported flaky test detected, please review

@keycloak-github-bot

Copy link
Copy Markdown

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.adapter.servlet.SAMLServletAdapterTest#employeeSigPostNoIdpKeyTestCertSubjectAsKeyNameInKeyInfo

Keycloak CI - Adapter IT

org.opentest4j.AssertionFailedError: expected: <true> but was: <false>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
	at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
...

Report flaky test

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the form-submission timing and OAuth callback-detection changes.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@ahus1
ahus1 marked this pull request as ready for review November 16, 2026 10:25
@ahus1
ahus1 requested a review from a team as a code owner November 16, 2026 10:25
@ahus1
ahus1 enabled auto-merge (squash) November 16, 2026 10:25
Comment on lines 300 to 308
login("login-test");

//submit with error
verifyProfilePage.assertCurrent();
assertFalse(verifyProfilePage.isDepartmentPresent());
verifyProfilePage.update("First", " ");

//submit OK
verifyProfilePage.assertCurrent();

@gaoyikeshuer gaoyikeshuer Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this part need the same fix as well? Not a blocker, this PR LGTM

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing that out; I just pushed a more comprehensive approve. Waiting for CI to be green.

Closes keycloak#55172

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 17:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the callback-detection and page-transition synchronization changes.

No open issues 🎉

Review effort: Balanced

@ahus1

ahus1 commented Nov 16, 2026

Copy link
Copy Markdown
Member Author

@gaoyikeshuer - ready for another review. Thanks!

@gaoyikeshuer gaoyikeshuer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ahus1 ! LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky test: VerifyProfileTest#testCustomValidationLastName fails in Forms IT (firefox)

3 participants