try and make tests more stable#50995
Conversation
There was a problem hiding this comment.
Pull request overview
Improves Admin UI Playwright test isolation, selectors, readiness checks, environment configuration, and feature handling.
Changes:
- Isolates group and SAML test data.
- Hardens table, alert, switch, and drag-and-drop interactions.
- Adds configurable server credentials and OID4VCI gating.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
test/utils/table.ts |
Hardens table selectors and extraction. |
test/utils/masthead.ts |
Extends alert readiness checks. |
test/utils/form.ts |
Adds switch interaction fallbacks. |
test/utils/constants.ts |
Adds environment-based server credentials. |
test/utils/AdminClient.ts |
Applies shared configuration and realm-scoped cleanup. |
test/groups/list.spec.ts |
Isolates group tests by realm and UUID. |
test/clients/saml.ts |
Handles SAML post-binding interaction failures. |
test/clients/saml.spec.ts |
Uses uniquely targeted SAML clients. |
test/client-scope/oid4vci-client-scope.spec.ts |
Adds deterministic readiness and feature gating. |
test/autentication/flows.spec.ts |
Adds drag-and-drop fallback strategies. |
closes keycloak#50992 resolves #34 closes keycloak#50995 Signed-off-by: Steve Hawkins <shawkins@redhat.com>
closes keycloak#50992 resolves #34 closes keycloak#50995 Signed-off-by: Steve Hawkins <shawkins@redhat.com>
closes keycloak#50992 resolves #34 closes keycloak#50995 Signed-off-by: Steve Hawkins <shawkins@redhat.com>
closes keycloak#50992 resolves #34 closes keycloak#50995 Signed-off-by: Steve Hawkins <shawkins@redhat.com>
closes keycloak#50992 resolves #34 closes keycloak#50995 Signed-off-by: Steve Hawkins <shawkins@redhat.com>
closes keycloak#50992 resolves #34 closes keycloak#50995 Signed-off-by: Steve Hawkins <shawkins@redhat.com>
closes keycloak#50992 resolves #34 closes keycloak#50995 Signed-off-by: Steve Hawkins <shawkins@redhat.com>
closes keycloak#50992 resolves #34 closes keycloak#50995 Signed-off-by: Steve Hawkins <shawkins@redhat.com>
closes keycloak#50992 resolves #34 closes keycloak#50995 Signed-off-by: Steve Hawkins <shawkins@redhat.com>
closes keycloak#50992 resolves #34 closes keycloak#50995 Signed-off-by: Steve Hawkins <shawkins@redhat.com>
closes keycloak#50992 resolves #34 closes keycloak#50995 Signed-off-by: Steve Hawkins <shawkins@redhat.com>
closes keycloak#50992 resolves #34 closes keycloak#50995 Signed-off-by: Steve Hawkins <shawkins@redhat.com>
closes keycloak#50992 resolves #34 closes keycloak#50995 Signed-off-by: Steve Hawkins <shawkins@redhat.com>
closes keycloak#50992 resolves #34 closes keycloak#50995 Signed-off-by: Steve Hawkins <shawkins@redhat.com>
closes keycloak#50992 resolves #34 closes keycloak#50995 Signed-off-by: Steve Hawkins <shawkins@redhat.com>
closes keycloak#50992 resolves #34 closes keycloak#50995 Signed-off-by: Steve Hawkins <shawkins@redhat.com>
closes keycloak#50992 resolves #34 closes keycloak#50995 Signed-off-by: Steve Hawkins <shawkins@redhat.com>
closes keycloak#50992 resolves #34 closes keycloak#50995 Signed-off-by: Steve Hawkins <shawkins@redhat.com>
closes keycloak#50992 resolves #34 closes keycloak#50995 Signed-off-by: Steve Hawkins <shawkins@redhat.com>
closes keycloak#50992 resolves #34 closes keycloak#50995 Signed-off-by: Steve Hawkins <shawkins@redhat.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 19 out of 19 changed files in this pull request and generated 4 comments.
Comments suppressed due to low confidence (4)
js/apps/admin-ui/test/utils/table.ts:112
locator.count()does not auto-wait, unlike the previousclick(). If the toolbar is still rendering, every count can be zero and this helper immediately throws; wait for an actionable candidate (for example with an auto-waiting combined locator) before deciding to use the overflow fallback.
const exactButton = toolbar.getByRole("button", {
name: itemName,
exact: true,
});
if ((await exactButton.count()) > 0) {
js/apps/admin-ui/test/utils/constants.ts:6
- The environment override is not consistently applied:
test/identity-providers/main.ts:4-8,99-100still hard-codeshttp://localhost:8080. WithKEYCLOAK_SERVER_URLset, login/admin calls target one server while those provider flows target another, so all server URL consumers need to use this constant.
export const SERVER_URL = normalizeServerUrl(
process.env.KEYCLOAK_SERVER_URL ?? DEFAULT_SERVER_URL,
);
js/apps/admin-ui/test/clients/saml.ts:54
- Catching every error also hides missing controls, timeouts, and browser/page failures; the test can then save an unchanged form and pass on the success alert. Only skip the interaction when the control is explicitly disabled/read-only and let unexpected failures propagate.
try {
await switchOff(page, "#attributes\\.saml🍺force🍺post🍺binding");
} catch {
// Some generated SAML clients expose this setting as read-only.
}
js/apps/admin-ui/test/autentication/flows.spec.ts:207
- The test now becomes skipped whenever all drag strategies fail, including when the product's reorder behavior has regressed. Keep the retrying helper, but fail when it returns
falseand retain the success-notification assertion so this behavior remains covered.
test.skip(
!moved,
"Drag-and-drop reorder could not be triggered deterministically in this runtime.",
);
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 25 out of 25 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (1)
js/apps/admin-ui/test/utils/table.ts:30
- These
count()checks are immediate snapshots; waiting fortbodydoes not wait for its asynchronously filtered rows. Callers commonly press Enter and invoke this helper immediately (for example,clients/saml.spec.ts:102-103), so the helper can run through every zero count and throw before the requested row appears. Wait/poll for the candidate links or for table loading to finish before concluding that no match exists.
if ((await exactTableLink.count()) > 0) {
| return "read-only"; | ||
| } | ||
|
|
||
| await switchOff(page, postBindingSwitch); |
| export async function clickCreateUser(page: Page) { | ||
| const emptyAction = page.getByTestId("no-users-found-empty-action"); | ||
| if ((await emptyAction.count()) > 0 && (await emptyAction.isVisible())) { | ||
| await emptyAction.click(); | ||
| return; | ||
| } | ||
|
|
||
| await page.getByTestId("add-user").click(); | ||
| } |
|
@vmuzikar JavaScript CI runs are roughly 35–96 minutes each (typical around ~50–60 min), that makes 50 sequential runs ~40 - 80 hours of runtime, and with queue/wait time it’s usually 2 - 4 days. I could make a couple of runs but 50 seems a bit much |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 25 out of 25 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (2)
js/apps/admin-ui/test/utils/table.ts:63
- These
count()probes do not auto-wait for search results. Callers such asclients/saml.spec.tsinvoke this immediately after pressing Enter, so a slow response can reach this branch with zero links and throw before the requested row renders; let the final locator click auto-wait instead.
if ((await looseTableLink.count()) > 0) {
await looseTableLink.click();
return;
}
throw new Error(`Table row item "${itemName}" not found`);
js/apps/admin-ui/test/clients/saml.spec.ts:118
- This converts an unexpectedly disabled control into a skipped test, masking permission or rendering regressions.
SamlConfig.tsxexposes this switch underFormAccess(role="manage-clients")and has no generated-client read-only mode, so this admin test should require the control to be enabled and fail if it is not.
const postBindingResult = await clickPostBinding(page);
// eslint-disable-next-line playwright/no-skipped-test -- Generated SAML clients can expose this setting as read-only.
test.skip(
postBindingResult === "read-only",
"Post binding switch is read-only for this generated SAML client.",
);
| test.afterAll(async () => { | ||
| await adminClient.deleteGroups(); | ||
| for (let i = 0; i < 5; i++) { | ||
| await adminClient.deleteUser(username + i); | ||
| await adminClient.deleteGroups(testBed.realm); | ||
| for (const { username } of users) { | ||
| await adminClient.deleteUser(username, testBed.realm, true); | ||
| } | ||
| await testBed[Symbol.asyncDispose](); |
Sure, it can be less than 50. What I usually do is create a tmp branch, update the CI there to run only the selected tests, and make them run in a matrix of e.g. 20 in parallel. This gives me a good idea about the stability. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 25 out of 25 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (1)
js/apps/admin-ui/test/utils/table.ts:58
- These
count()checks are instantaneous. Callers commonly invoke this immediately aftersearchItem(), while the existing<tbody>is already visible but the filtered rows are still loading, so the helper can throw before Playwright's locator auto-waiting has a chance to find the link. Wait for the final matching locator instead of snapshotting its count.
if ((await looseTableLink.count()) > 0) {
|
|
||
| test.afterEach(async () => { | ||
| await adminClient.deleteUser("testuser7", realmName, true); | ||
| await adminClient.deleteUser("testuser8@gmail.com", realmName, true); |
| jobs: | ||
| build-keycloak: | ||
| name: Build Keycloak (flaky matrix) | ||
| if: github.event_name != 'pull_request' || github.event.pull_request.number == 50995 |
|
@edewit Seems somehow the temp changes to the CI got also into this PR? |
|
right, I'm going to drop those when the tests are completed |
That shouldn't be necessary. The tests run you linked seems to use |
|
it happened when I needed to push two more changes to the tests |
Ok, no worries. Please ping me when this is ready for a final review (incl. the batch test results to double check it's stable). |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 25 out of 25 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
js/apps/admin-ui/test/utils/table.ts:30
count()does not wait for asynchronously loaded rows. Since the table body is often rendered before its data (including immediately aftersearchItem), this helper can now throw before the requested link appears; wait for the matching link to become visible before deciding which fallback to use.
if ((await exactTableLink.count()) > 0) {
|
right removed those commit and updated the test branch, still don't have a test matrix that works |
* State leakage/coupling: reduced in groups/SAML flows by scoped test data and improved targeting. * Selector brittleness: reduced via table + alert helper hardening. * Timing races: reduced via deterministic readiness assertions and switch handling. * Environment assumptions: addressed via env-configurable server/auth and OID4VCI feature gating. fixes: keycloak#50994 Signed-off-by: Erik Jan de Wit <erikjan.dewit@gmail.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 25 out of 25 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (3)
js/apps/admin-ui/test/utils/table.ts:30
- These
count()checks are immediate, so aftersearchItem()submits an asynchronous table refresh this helper can throw before the requested row is rendered. Keep Playwright's auto-waiting behavior (or explicitly wait for the target/loading state) before deciding which fallback locator to use.
if ((await exactTableLink.count()) > 0) {
js/apps/admin-ui/test/utils/constants.ts:6
- A configured URL with a Keycloak context path is not handled consistently: for
KEYCLOAK_SERVER_URL=https://host/auth, the admin client uses/auth, butnavigateTo()resolves the absoluteROOT_PATHtohttps://host/admin/...and discards/auth. Preserve the configured pathname when constructing browser URLs so this environment option works withhttp-relative-pathdeployments.
export const SERVER_URL = normalizeServerUrl(
process.env.KEYCLOAK_SERVER_URL ?? DEFAULT_SERVER_URL,
);
js/apps/admin-ui/test/clients/saml.spec.ts:118
- The SAML form does not make this switch read-only for generated clients (
src/clients/add/SamlConfig.tsx:48-51); it is disabled only when the user lacksmanage-clientsaccess. Skipping here therefore hides a loading/authorization regression instead of testing the update—wait for the switch to be enabled and let the test fail if it never is.
// eslint-disable-next-line playwright/no-skipped-test -- Generated SAML clients can expose this setting as read-only.
test.skip(
postBindingResult === "read-only",
"Post binding switch is read-only for this generated SAML client.",
);
| const features = (await this.#client.serverInfo.find({ realm })).features; | ||
| const normalizeFeatureName = (name?: string) => name?.replace(/_V\d+$/, ""); | ||
|
|
||
| return ( | ||
| features?.some( | ||
| (feature) => | ||
| feature.enabled && | ||
| normalizeFeatureName(feature.name) === | ||
| normalizeFeatureName(featureName), | ||
| ) ?? false | ||
| ); |
| async function clickCreateClientScopeAction(page: Page) { | ||
| const createScopeButton = page.getByRole("button", { | ||
| name: /Create client scope/i, | ||
| }); | ||
| if ((await createScopeButton.count()) > 0) { | ||
| await createScopeButton.first().click(); | ||
| return; | ||
| } | ||
|
|
||
| await page | ||
| .getByRole("link", { name: /Create client scope/i }) | ||
| .first() | ||
| .click(); | ||
| } |
Seems some tests failed? Are those failures relevant? |
data and improved targeting.
switch handling.
and OID4VCI feature gating.
fixes: #50994
Signed-off-by: Erik Jan de Wit erikjan.dewit@gmail.com