Skip to content

try and make tests more stable#50995

Open
edewit wants to merge 4 commits into
keycloak:mainfrom
edewit:test-improvement
Open

try and make tests more stable#50995
edewit wants to merge 4 commits into
keycloak:mainfrom
edewit:test-improvement

Conversation

@edewit

@edewit edewit commented Jul 17, 2026

Copy link
Copy Markdown
Contributor
  • State leakage/coupling: reduced in groups/SAML flows by scoped test
    data and improved targeting.
  • Selector brittleness: reduced via table + alert helper hardening.
  • Timing races: reduced via deterministic readiness assertions and
    switch handling.
  • Environment assumptions: addressed via env-configurable server/auth
    and OID4VCI feature gating.

fixes: #50994
Signed-off-by: Erik Jan de Wit erikjan.dewit@gmail.com

Copilot AI review requested due to automatic review settings July 17, 2026 14:30
@edewit
edewit requested review from a team as code owners July 17, 2026 14:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Improves Admin UI Playwright test isolation, selectors, readiness checks, environment configuration, and feature handling.

Changes:

  • Isolates group and SAML test data.
  • Hardens table, alert, switch, and drag-and-drop interactions.
  • Adds configurable server credentials and OID4VCI gating.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 10 comments.

Show a summary per file
File Description
test/utils/table.ts Hardens table selectors and extraction.
test/utils/masthead.ts Extends alert readiness checks.
test/utils/form.ts Adds switch interaction fallbacks.
test/utils/constants.ts Adds environment-based server credentials.
test/utils/AdminClient.ts Applies shared configuration and realm-scoped cleanup.
test/groups/list.spec.ts Isolates group tests by realm and UUID.
test/clients/saml.ts Handles SAML post-binding interaction failures.
test/clients/saml.spec.ts Uses uniquely targeted SAML clients.
test/client-scope/oid4vci-client-scope.spec.ts Adds deterministic readiness and feature gating.
test/autentication/flows.spec.ts Adds drag-and-drop fallback strategies.

Comment thread js/apps/admin-ui/test/utils/form.ts Outdated
Comment thread js/apps/admin-ui/test/utils/form.ts Outdated
Comment thread js/apps/admin-ui/test/utils/form.ts Outdated
Comment thread js/apps/admin-ui/test/utils/table.ts Outdated
Comment thread js/apps/admin-ui/test/clients/saml.ts Outdated
Comment thread js/apps/admin-ui/test/autentication/flows.spec.ts Outdated
Comment thread js/apps/admin-ui/test/client-scope/oid4vci-client-scope.spec.ts Outdated
Comment thread js/apps/admin-ui/test/client-scope/oid4vci-client-scope.spec.ts Outdated
Comment thread js/apps/admin-ui/test/client-scope/oid4vci-client-scope.spec.ts Outdated
Comment thread js/apps/admin-ui/test/utils/constants.ts
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 17, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
@github-actions github-actions Bot mentioned this pull request Jul 17, 2026
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 17, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 17, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 17, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 17, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 17, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 17, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 17, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 17, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 17, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 17, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 17, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 17, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 17, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 17, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 18, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 18, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 18, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 18, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 18, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
Copilot AI review requested due to automatic review settings July 20, 2026 10:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (4)

js/apps/admin-ui/test/utils/table.ts:112

  • locator.count() does not auto-wait, unlike the previous click(). If the toolbar is still rendering, every count can be zero and this helper immediately throws; wait for an actionable candidate (for example with an auto-waiting combined locator) before deciding to use the overflow fallback.
  const exactButton = toolbar.getByRole("button", {
    name: itemName,
    exact: true,
  });
  if ((await exactButton.count()) > 0) {

js/apps/admin-ui/test/utils/constants.ts:6

  • The environment override is not consistently applied: test/identity-providers/main.ts:4-8,99-100 still hard-codes http://localhost:8080. With KEYCLOAK_SERVER_URL set, login/admin calls target one server while those provider flows target another, so all server URL consumers need to use this constant.
export const SERVER_URL = normalizeServerUrl(
  process.env.KEYCLOAK_SERVER_URL ?? DEFAULT_SERVER_URL,
);

js/apps/admin-ui/test/clients/saml.ts:54

  • Catching every error also hides missing controls, timeouts, and browser/page failures; the test can then save an unchanged form and pass on the success alert. Only skip the interaction when the control is explicitly disabled/read-only and let unexpected failures propagate.
  try {
    await switchOff(page, "#attributes\\.saml🍺force🍺post🍺binding");
  } catch {
    // Some generated SAML clients expose this setting as read-only.
  }

js/apps/admin-ui/test/autentication/flows.spec.ts:207

  • The test now becomes skipped whenever all drag strategies fail, including when the product's reorder behavior has regressed. Keep the retrying helper, but fail when it returns false and retain the success-notification assertion so this behavior remains covered.
    test.skip(
      !moved,
      "Drag-and-drop reorder could not be triggered deterministically in this runtime.",
    );

Comment thread js/apps/admin-ui/test/utils/table.ts Outdated
Comment thread js/apps/admin-ui/test/client-scope/oid4vci-client-scope.spec.ts Outdated
Comment thread js/apps/admin-ui/test/client-scope/oid4vci-client-scope.spec.ts Outdated
Comment thread js/apps/admin-ui/test/client-scope/oid4vci-client-scope.spec.ts Outdated
shawkins added a commit to shawkins/keycloak that referenced this pull request Jul 20, 2026
closes keycloak#50992
resolves #34
closes keycloak#50995

Signed-off-by: Steve Hawkins <shawkins@redhat.com>
Copilot AI review requested due to automatic review settings July 20, 2026 12:25

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 2 comments.

Comment thread js/apps/admin-ui/test/realm-settings/login.ts Outdated
Comment thread js/apps/admin-ui/test/clients/assign-oid4vci-client-scope.ts Outdated
Copilot AI review requested due to automatic review settings July 21, 2026 08:24
edewit added 3 commits July 21, 2026 10:25
* State leakage/coupling: reduced in groups/SAML flows by scoped test
data and improved targeting.
* Selector brittleness: reduced via table + alert helper hardening.
* Timing races: reduced via deterministic readiness assertions and
switch handling.
* Environment assumptions: addressed via env-configurable server/auth
and OID4VCI feature gating.

fixes: keycloak#50994
Signed-off-by: Erik Jan de Wit <erikjan.dewit@gmail.com>
Signed-off-by: Erik Jan de Wit <erikjan.dewit@gmail.com>
Signed-off-by: Erik Jan de Wit <erikjan.dewit@gmail.com>
@edewit
edewit force-pushed the test-improvement branch from 1d9d2fe to 09b69e2 Compare July 21, 2026 08:25

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

js/apps/admin-ui/test/clients/assign-oid4vci-client-scope.ts:10

  • This OID4VCI path is still unconditional, so the assignment test fails while looking for this option whenever the server feature is disabled. Apply the same OID4VC_VCI gate added to oid4vci-client-scope.spec.ts before selecting the protocol so unsupported environments skip consistently.
  await selectItem(page, "#kc-protocol", "OpenID for Verifiable Credentials");

Comment thread js/apps/admin-ui/test/clients/saml.ts Outdated
Copilot AI review requested due to automatic review settings July 21, 2026 08:31

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (2)

js/apps/admin-ui/test/utils/table.ts:71

  • The final fallback leaves the table and can click an unrelated global link (for example, a matching sidebar item) while rows are still loading, because the preceding count() calls do not wait. Keep this fallback row-scoped and wait for the row instead.
  const exactLink = page.getByRole("link", { name: itemName, exact: true });
  if ((await exactLink.count()) > 0) {
    await exactLink.first().click();
    return;
  }

  await page.getByRole("link", { name: itemName }).first().click();

js/apps/admin-ui/test/utils/form.ts:129

  • check()/uncheck() can throw after dispatching the change when the switch rerenders. Unconditionally clicking in the catch path can then toggle an already-correct switch back, so re-read its state before applying the fallback.
      await clickSwitch(switchElement);

Comment on lines +24 to +29
for (const action of directAction) {
if ((await action.count()) > 0) {
await action.first().click();
return;
}
}
await page.getByTestId("name").fill(clientScopeName);
await clickSaveButton(page);
await expect(page.getByText("Client scope created")).toBeVisible();
await createOid4vciClientScope(page, clientScopeName);
Signed-off-by: Erik Jan de Wit <erikjan.dewit@gmail.com>
Copilot AI review requested due to automatic review settings July 23, 2026 07:52

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 1 comment.

Comment on lines +4 to +6
export const SERVER_URL = normalizeServerUrl(
process.env.KEYCLOAK_SERVER_URL ?? DEFAULT_SERVER_URL,
);
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky admin UI tests

2 participants