Skip to content

Fix flaky OrganizationInvitationTokenTest by waiting for the invitation to be consumed - #55242

Open
ahus1 wants to merge 1 commit into
keycloak:mainfrom
ahus1:is-55241-flaky-organization-tests
Open

ahus1 wants to merge 1 commit into
keycloak:mainfrom
ahus1:is-55241-flaky-organization-tests

Conversation

@ahus1

@ahus1 ahus1 commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Closes #55241

The invitation tests read the member count and the current page right after a click, before the browser and the server had finished. They now poll for the expected state, with the Chrome-only WebDriverException during navigation tolerated when checking for the update profile page.

Decisions and assumptions

Closes keycloak#55241

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>
@keycloak-github-bot

Copy link
Copy Markdown

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.actions.TermsAndConditionsTest#termsAccepted

Keycloak CI - Forms IT (firefox)

java.lang.RuntimeException: Could not create statement
	at org.jboss.arquillian.junit.Arquillian.methodBlock(Arquillian.java:307)
	at org.jboss.arquillian.junit.Arquillian$2.evaluate(Arquillian.java:160)
	at org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:344)
	at org.jboss.arquillian.junit.Arquillian.access$200(Arquillian.java:48)
...

Report flaky test

org.keycloak.testsuite.actions.TermsAndConditionsTest#termsDeclinedAccount

Keycloak CI - Forms IT (firefox)

java.lang.RuntimeException: Could not create statement
	at org.jboss.arquillian.junit.Arquillian.methodBlock(Arquillian.java:307)
	at org.jboss.arquillian.junit.Arquillian$2.evaluate(Arquillian.java:160)
	at org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:344)
	at org.jboss.arquillian.junit.Arquillian.access$200(Arquillian.java:48)
...

Report flaky test

@keycloak-github-bot keycloak-github-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unreported flaky test detected, please review

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the invitation-test synchronization changes.

No open issues 🎉

What changed in this PR

This PR addresses #55241 by making organization invitation tests wait for browser and server state instead of checking immediately after submission. The separate OrganizationSharedIdpBrokerTest failures remain out of scope.

Changes:

  • Poll for the expected member count after registration and profile updates.
  • Wait for membership creation or the update-profile page after broker login.
  • Tolerate transient browser navigation errors when checking the page ID.
File Description
tests/​base/​src/​test/​java/​org/​keycloak/​tests/​organization/​admin/​OrganizationInvitationTokenTest.java Adds polling helpers and navigation-error handling to reduce invitation-test timing failures.

Review effort: Balanced

@ahus1
ahus1 marked this pull request as ready for review November 16, 2026 18:58
@ahus1
ahus1 requested a review from a team as a code owner November 16, 2026 18:58
@ahus1
ahus1 enabled auto-merge (squash) November 16, 2026 18:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky test: OrganizationInvitationTokenTest reads browser and server state before the request completes

3 participants