Skip to content

try and make tests more stable#50995

Open
edewit wants to merge 1 commit into
keycloak:mainfrom
edewit:test-improvement
Open

try and make tests more stable#50995
edewit wants to merge 1 commit into
keycloak:mainfrom
edewit:test-improvement

Conversation

@edewit

@edewit edewit commented Jul 17, 2026

Copy link
Copy Markdown
Contributor
  • State leakage/coupling: reduced in groups/SAML flows by scoped test
    data and improved targeting.
  • Selector brittleness: reduced via table + alert helper hardening.
  • Timing races: reduced via deterministic readiness assertions and
    switch handling.
  • Environment assumptions: addressed via env-configurable server/auth
    and OID4VCI feature gating.

fixes: #50994
Signed-off-by: Erik Jan de Wit erikjan.dewit@gmail.com

Copilot AI review requested due to automatic review settings July 17, 2026 14:30
@edewit
edewit requested review from a team as code owners July 17, 2026 14:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Improves Admin UI Playwright test isolation, selectors, readiness checks, environment configuration, and feature handling.

Changes:

  • Isolates group and SAML test data.
  • Hardens table, alert, switch, and drag-and-drop interactions.
  • Adds configurable server credentials and OID4VCI gating.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 10 comments.

Show a summary per file
File Description
test/utils/table.ts Hardens table selectors and extraction.
test/utils/masthead.ts Extends alert readiness checks.
test/utils/form.ts Adds switch interaction fallbacks.
test/utils/constants.ts Adds environment-based server credentials.
test/utils/AdminClient.ts Applies shared configuration and realm-scoped cleanup.
test/groups/list.spec.ts Isolates group tests by realm and UUID.
test/clients/saml.ts Handles SAML post-binding interaction failures.
test/clients/saml.spec.ts Uses uniquely targeted SAML clients.
test/client-scope/oid4vci-client-scope.spec.ts Adds deterministic readiness and feature gating.
test/autentication/flows.spec.ts Adds drag-and-drop fallback strategies.

Comment thread js/apps/admin-ui/test/utils/form.ts Outdated
Comment thread js/apps/admin-ui/test/utils/form.ts Outdated
Comment thread js/apps/admin-ui/test/utils/form.ts Outdated
Comment thread js/apps/admin-ui/test/utils/table.ts Outdated
Comment thread js/apps/admin-ui/test/clients/saml.ts Outdated
Comment thread js/apps/admin-ui/test/autentication/flows.spec.ts Outdated
Comment thread js/apps/admin-ui/test/client-scope/oid4vci-client-scope.spec.ts Outdated
Comment thread js/apps/admin-ui/test/client-scope/oid4vci-client-scope.spec.ts Outdated
Comment thread js/apps/admin-ui/test/client-scope/oid4vci-client-scope.spec.ts Outdated
Comment thread js/apps/admin-ui/test/utils/constants.ts
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 17, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
@github-actions github-actions Bot mentioned this pull request Jul 17, 2026
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 17, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 17, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 17, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 17, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 17, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 17, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 17, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 17, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 17, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 17, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 17, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 17, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 17, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 17, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 18, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 18, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 18, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 18, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 18, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
Copilot AI review requested due to automatic review settings July 20, 2026 10:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (4)

js/apps/admin-ui/test/utils/table.ts:112

  • locator.count() does not auto-wait, unlike the previous click(). If the toolbar is still rendering, every count can be zero and this helper immediately throws; wait for an actionable candidate (for example with an auto-waiting combined locator) before deciding to use the overflow fallback.
  const exactButton = toolbar.getByRole("button", {
    name: itemName,
    exact: true,
  });
  if ((await exactButton.count()) > 0) {

js/apps/admin-ui/test/utils/constants.ts:6

  • The environment override is not consistently applied: test/identity-providers/main.ts:4-8,99-100 still hard-codes http://localhost:8080. With KEYCLOAK_SERVER_URL set, login/admin calls target one server while those provider flows target another, so all server URL consumers need to use this constant.
export const SERVER_URL = normalizeServerUrl(
  process.env.KEYCLOAK_SERVER_URL ?? DEFAULT_SERVER_URL,
);

js/apps/admin-ui/test/clients/saml.ts:54

  • Catching every error also hides missing controls, timeouts, and browser/page failures; the test can then save an unchanged form and pass on the success alert. Only skip the interaction when the control is explicitly disabled/read-only and let unexpected failures propagate.
  try {
    await switchOff(page, "#attributes\\.saml🍺force🍺post🍺binding");
  } catch {
    // Some generated SAML clients expose this setting as read-only.
  }

js/apps/admin-ui/test/autentication/flows.spec.ts:207

  • The test now becomes skipped whenever all drag strategies fail, including when the product's reorder behavior has regressed. Keep the retrying helper, but fail when it returns false and retain the success-notification assertion so this behavior remains covered.
    test.skip(
      !moved,
      "Drag-and-drop reorder could not be triggered deterministically in this runtime.",
    );

Comment thread js/apps/admin-ui/test/utils/table.ts Outdated
Comment thread js/apps/admin-ui/test/client-scope/oid4vci-client-scope.spec.ts Outdated
Comment thread js/apps/admin-ui/test/client-scope/oid4vci-client-scope.spec.ts Outdated
Comment thread js/apps/admin-ui/test/client-scope/oid4vci-client-scope.spec.ts Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

js/apps/admin-ui/test/utils/table.ts:30

  • These count() checks are immediate snapshots; waiting for tbody does not wait for its asynchronously filtered rows. Callers commonly press Enter and invoke this helper immediately (for example, clients/saml.spec.ts:102-103), so the helper can run through every zero count and throw before the requested row appears. Wait/poll for the candidate links or for table loading to finish before concluding that no match exists.
  if ((await exactTableLink.count()) > 0) {

return "read-only";
}

await switchOff(page, postBindingSwitch);
Comment on lines +55 to +63
export async function clickCreateUser(page: Page) {
const emptyAction = page.getByTestId("no-users-found-empty-action");
if ((await emptyAction.count()) > 0 && (await emptyAction.isVisible())) {
await emptyAction.click();
return;
}

await page.getByTestId("add-user").click();
}
@edewit

edewit commented Jul 27, 2026

Copy link
Copy Markdown
Contributor Author

@vmuzikar JavaScript CI runs are roughly 35–96 minutes each (typical around ~50–60 min), that makes 50 sequential runs ~40 - 80 hours of runtime, and with queue/wait time it’s usually 2 - 4 days. I could make a couple of runs but 50 seems a bit much

Copilot AI review requested due to automatic review settings July 27, 2026 09:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

js/apps/admin-ui/test/utils/table.ts:63

  • These count() probes do not auto-wait for search results. Callers such as clients/saml.spec.ts invoke this immediately after pressing Enter, so a slow response can reach this branch with zero links and throw before the requested row renders; let the final locator click auto-wait instead.
  if ((await looseTableLink.count()) > 0) {
    await looseTableLink.click();
    return;
  }

  throw new Error(`Table row item "${itemName}" not found`);

js/apps/admin-ui/test/clients/saml.spec.ts:118

  • This converts an unexpectedly disabled control into a skipped test, masking permission or rendering regressions. SamlConfig.tsx exposes this switch under FormAccess(role="manage-clients") and has no generated-client read-only mode, so this admin test should require the control to be enabled and fail if it is not.
    const postBindingResult = await clickPostBinding(page);
    // eslint-disable-next-line playwright/no-skipped-test -- Generated SAML clients can expose this setting as read-only.
    test.skip(
      postBindingResult === "read-only",
      "Post binding switch is read-only for this generated SAML client.",
    );

Comment on lines 44 to +49
test.afterAll(async () => {
await adminClient.deleteGroups();
for (let i = 0; i < 5; i++) {
await adminClient.deleteUser(username + i);
await adminClient.deleteGroups(testBed.realm);
for (const { username } of users) {
await adminClient.deleteUser(username, testBed.realm, true);
}
await testBed[Symbol.asyncDispose]();
@vmuzikar

Copy link
Copy Markdown
Contributor

@vmuzikar JavaScript CI runs are roughly 35–96 minutes each (typical around ~50–60 min), that makes 50 sequential runs ~40 - 80 hours of runtime, and with queue/wait time it’s usually 2 - 4 days. I could make a couple of runs but 50 seems a bit much

Sure, it can be less than 50. What I usually do is create a tmp branch, update the CI there to run only the selected tests, and make them run in a matrix of e.g. 20 in parallel. This gives me a good idea about the stability.

Copilot AI review requested due to automatic review settings July 27, 2026 09:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

js/apps/admin-ui/test/utils/table.ts:58

  • These count() checks are instantaneous. Callers commonly invoke this immediately after searchItem(), while the existing <tbody> is already visible but the filtered rows are still loading, so the helper can throw before Playwright's locator auto-waiting has a chance to find the link. Wait for the final matching locator instead of snapshotting its count.
  if ((await looseTableLink.count()) > 0) {


test.afterEach(async () => {
await adminClient.deleteUser("testuser7", realmName, true);
await adminClient.deleteUser("testuser8@gmail.com", realmName, true);
@edewit

edewit commented Jul 27, 2026

Copy link
Copy Markdown
Contributor Author

Copilot AI review requested due to automatic review settings July 27, 2026 11:15

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated 1 comment.

jobs:
build-keycloak:
name: Build Keycloak (flaky matrix)
if: github.event_name != 'pull_request' || github.event.pull_request.number == 50995
@vmuzikar

Copy link
Copy Markdown
Contributor

@edewit Seems somehow the temp changes to the CI got also into this PR?

@edewit

edewit commented Jul 27, 2026

Copy link
Copy Markdown
Contributor Author

right, I'm going to drop those when the tests are completed

@vmuzikar

Copy link
Copy Markdown
Contributor

right, I'm going to drop those when the tests are completed

That shouldn't be necessary. The tests run you linked seems to use tmp-flaky-matrix-50995 while this PR uses test-improvement branch.

@edewit

edewit commented Jul 27, 2026

Copy link
Copy Markdown
Contributor Author

it happened when I needed to push two more changes to the tests

@vmuzikar

Copy link
Copy Markdown
Contributor

it happened when I needed to push two more changes to the tests

Ok, no worries. Please ping me when this is ready for a final review (incl. the batch test results to double check it's stable).

Copilot AI review requested due to automatic review settings July 27, 2026 12:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

js/apps/admin-ui/test/utils/table.ts:30

  • count() does not wait for asynchronously loaded rows. Since the table body is often rendered before its data (including immediately after searchItem), this helper can now throw before the requested link appears; wait for the matching link to become visible before deciding which fallback to use.
  if ((await exactTableLink.count()) > 0) {

@edewit

edewit commented Jul 27, 2026

Copy link
Copy Markdown
Contributor Author

right removed those commit and updated the test branch, still don't have a test matrix that works

@edewit

edewit commented Jul 27, 2026

Copy link
Copy Markdown
Contributor Author

* State leakage/coupling: reduced in groups/SAML flows by scoped test
data and improved targeting.
* Selector brittleness: reduced via table + alert helper hardening.
* Timing races: reduced via deterministic readiness assertions and
switch handling.
* Environment assumptions: addressed via env-configurable server/auth
and OID4VCI feature gating.

fixes: keycloak#50994
Signed-off-by: Erik Jan de Wit <erikjan.dewit@gmail.com>
Copilot AI review requested due to automatic review settings July 27, 2026 13:59
@edewit
edewit force-pushed the test-improvement branch from 7e1424d to 5407b92 Compare July 27, 2026 13:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (3)

js/apps/admin-ui/test/utils/table.ts:30

  • These count() checks are immediate, so after searchItem() submits an asynchronous table refresh this helper can throw before the requested row is rendered. Keep Playwright's auto-waiting behavior (or explicitly wait for the target/loading state) before deciding which fallback locator to use.
  if ((await exactTableLink.count()) > 0) {

js/apps/admin-ui/test/utils/constants.ts:6

  • A configured URL with a Keycloak context path is not handled consistently: for KEYCLOAK_SERVER_URL=https://host/auth, the admin client uses /auth, but navigateTo() resolves the absolute ROOT_PATH to https://host/admin/... and discards /auth. Preserve the configured pathname when constructing browser URLs so this environment option works with http-relative-path deployments.
export const SERVER_URL = normalizeServerUrl(
  process.env.KEYCLOAK_SERVER_URL ?? DEFAULT_SERVER_URL,
);

js/apps/admin-ui/test/clients/saml.spec.ts:118

  • The SAML form does not make this switch read-only for generated clients (src/clients/add/SamlConfig.tsx:48-51); it is disabled only when the user lacks manage-clients access. Skipping here therefore hides a loading/authorization regression instead of testing the update—wait for the switch to be enabled and let the test fail if it never is.
    // eslint-disable-next-line playwright/no-skipped-test -- Generated SAML clients can expose this setting as read-only.
    test.skip(
      postBindingResult === "read-only",
      "Post binding switch is read-only for this generated SAML client.",
    );

Comment on lines +357 to +367
const features = (await this.#client.serverInfo.find({ realm })).features;
const normalizeFeatureName = (name?: string) => name?.replace(/_V\d+$/, "");

return (
features?.some(
(feature) =>
feature.enabled &&
normalizeFeatureName(feature.name) ===
normalizeFeatureName(featureName),
) ?? false
);
Comment on lines +64 to +77
async function clickCreateClientScopeAction(page: Page) {
const createScopeButton = page.getByRole("button", {
name: /Create client scope/i,
});
if ((await createScopeButton.count()) > 0) {
await createScopeButton.first().click();
return;
}

await page
.getByRole("link", { name: /Create client scope/i })
.first()
.click();
}
@vmuzikar

Copy link
Copy Markdown
Contributor

new matrix https://github.com/edewit/keycloak/actions/runs/30270472211

Seems some tests failed? Are those failures relevant?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky admin UI tests

3 participants