fix: moving waiting into the assertMessage method by shawkins · Pull Request #51096 · keycloak/keycloak

shawkins · 2026-07-22T16:51:15Z

Awaitability was being used inconsistently around assertMessage calls, so it seems better to just embed the waiting in the CLIResult logic.

also making output reading more fool-proof

Based upon the copilot comments, it will also address flakyness such as https://github.com/keycloak/keycloak/actions/runs/29988894966/job/89147575751?pr=50324#step:5:3799

close: #51070

Copilot

Pull request overview

Centralizes waiting for asynchronous CLI output and improves process stream handling.

Changes:

Reads stdout and stderr concurrently using UTF-8.
Adds Awaitility-based positive output assertions.
Simplifies logging tests to use CLIResult.assertMessage.

Reviewed changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated 1 comment.

File	Description
`RawKeycloakDistribution.java`	Concurrently captures process output.
`CLIResult.java`	Adds output-aware assertion waiting.
`LoggingDistTest.java`	Removes redundant explicit waits.

keycloak-github-bot · 2026-07-22T18:34:15Z

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.adapter.servlet.SAMLServletAdapterTest#employeeSigPostNoIdpKeyTestNoKeyNameInKeyInfo

Keycloak CI - Adapter IT Strict Cookies

org.opentest4j.AssertionFailedError: expected: <true> but was: <false>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
	at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
...

Report flaky test

keycloak-github-bot

Unreported flaky test detected, please review

keycloak-github-bot

Unreported flaky test detected, please review

keycloak-github-bot · 2026-07-22T19:52:00Z

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.forms.MultipleTabsLoginTest#multipleTabsParallelLoginTestWithAuthSessionExpiredAndRegisterClick

Keycloak CI - Forms IT (chrome)

org.opentest4j.AssertionFailedError: expected: <true> but was: <false>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
	at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
...

org.opentest4j.AssertionFailedError: expected: <true> but was: <false>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
	at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
...

Report flaky test

Copilot

Pull request overview

Copilot reviewed 2 out of 3 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 2 out of 3 changed files in this pull request and generated 2 comments.

keycloak-github-bot · 2026-07-23T13:45:41Z

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.forms.MultipleTabsLoginTest#multipleTabsParallelLoginTestWithAuthSessionExpiredAndRegisterClick

Keycloak CI - Forms IT (firefox)

org.opentest4j.AssertionFailedError: expected: <true> but was: <false>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
	at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
...

Report flaky test

keycloak-github-bot

Unreported flaky test detected, please review

Copilot

Pull request overview

Copilot reviewed 2 out of 3 changed files in this pull request and generated no new comments.

also making output reading more fool-proof close: keycloak#51070 Signed-off-by: Steve Hawkins <shawkins@redhat.com>

Copilot

Pull request overview

Copilot reviewed 2 out of 3 changed files in this pull request and generated no new comments.

shawkins requested a review from Copilot July 22, 2026 18:26

Copilot started reviewing on behalf of shawkins July 22, 2026 18:26 View session

Copilot AI reviewed Jul 22, 2026

View reviewed changes

Comment thread quarkus/tests/junit5/src/main/java/org/keycloak/it/junit5/extension/CLIResult.java Outdated

keycloak-github-bot Bot added the flaky-test label Jul 22, 2026

keycloak-github-bot Bot reviewed Jul 22, 2026

View reviewed changes

shawkins force-pushed the iss51070 branch from c46210a to e4a05cf Compare July 22, 2026 18:49

keycloak-github-bot Bot mentioned this pull request Jul 22, 2026

Flaky test: org.keycloak.testsuite.forms.MultipleTabsLoginTest#multipleTabsParallelLoginTestWithAuthSessionExpiredInTheMiddle #51074

Open

keycloak-github-bot Bot reviewed Jul 22, 2026

View reviewed changes

shawkins marked this pull request as ready for review July 22, 2026 22:08

shawkins requested a review from a team as a code owner July 22, 2026 22:08

Copilot AI review requested due to automatic review settings July 22, 2026 22:08

shawkins requested a review from a team as a code owner July 22, 2026 22:08

Copilot started reviewing on behalf of shawkins July 22, 2026 22:09 View session

Copilot AI reviewed Jul 22, 2026

View reviewed changes

Comment thread quarkus/tests/junit5/src/main/java/org/keycloak/it/junit5/extension/CLIResult.java Outdated

Copilot AI review requested due to automatic review settings July 23, 2026 12:42

shawkins force-pushed the iss51070 branch from e4a05cf to abf4b1f Compare July 23, 2026 12:42

Copilot started reviewing on behalf of shawkins July 23, 2026 12:42 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

Comment thread quarkus/tests/junit5/src/main/java/org/keycloak/it/utils/RawKeycloakDistribution.java Outdated

Comment thread quarkus/tests/junit5/src/main/java/org/keycloak/it/junit5/extension/CLIResult.java Outdated

This was referenced Jul 23, 2026

Flaky test: org.keycloak.testsuite.forms.MultipleTabsLoginTest#testEmptyBaseUrl #51101

Open

Flaky test: org.keycloak.testsuite.forms.MultipleTabsLoginTest#multipleTabsParallelLoginTest #51076

Open

keycloak-github-bot Bot reviewed Jul 23, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 23, 2026 14:07

shawkins force-pushed the iss51070 branch from abf4b1f to 6f849ba Compare July 23, 2026 14:07

Copilot started reviewing on behalf of shawkins July 23, 2026 14:11 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

fix: moving waiting into the assertMessage method

1081384

also making output reading more fool-proof close: keycloak#51070 Signed-off-by: Steve Hawkins <shawkins@redhat.com>

shawkins force-pushed the iss51070 branch from 6f849ba to 1081384 Compare July 23, 2026 14:39

Copilot AI review requested due to automatic review settings July 23, 2026 14:39

Copilot started reviewing on behalf of shawkins July 23, 2026 14:39 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

vmuzikar approved these changes Jul 23, 2026

View reviewed changes

vmuzikar enabled auto-merge (squash) July 23, 2026 14:52

vmuzikar merged commit 2be1709 into keycloak:main Jul 23, 2026
94 checks passed

Uh oh!

Conversation

shawkins commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

keycloak-github-bot Bot commented Jul 22, 2026

Unreported flaky test detected

org.keycloak.testsuite.adapter.servlet.SAMLServletAdapterTest#employeeSigPostNoIdpKeyTestNoKeyNameInKeyInfo

Uh oh!

keycloak-github-bot Bot left a comment

Choose a reason for hiding this comment

Uh oh!

keycloak-github-bot Bot left a comment

Choose a reason for hiding this comment

Uh oh!

keycloak-github-bot Bot commented Jul 22, 2026

Unreported flaky test detected

org.keycloak.testsuite.forms.MultipleTabsLoginTest#multipleTabsParallelLoginTestWithAuthSessionExpiredAndRegisterClick

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

keycloak-github-bot Bot commented Jul 23, 2026

Unreported flaky test detected

org.keycloak.testsuite.forms.MultipleTabsLoginTest#multipleTabsParallelLoginTestWithAuthSessionExpiredAndRegisterClick

Uh oh!

keycloak-github-bot Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

shawkins commented Jul 22, 2026 •

edited

Loading