Skip to content

[OpenID4VCI] Implement all HAIP conformance tests (wallet_initiated, sd-jwt) - #50577

Merged
mposolda merged 1 commit into
keycloak:mainfrom
dominikschlosser:vci-conformance-gaps
Aug 12, 2026
Merged

[OpenID4VCI] Implement all HAIP conformance tests (wallet_initiated, sd-jwt)#50577
mposolda merged 1 commit into
keycloak:mainfrom
dominikschlosser:vci-conformance-gaps

Conversation

@dominikschlosser

@dominikschlosser dominikschlosser commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

@dominikschlosser dominikschlosser changed the title Vci conformance gaps [OpenID4VCI] Implement all HAIP conformance tests (wallet_initiated, sd-jwt) Jul 2, 2026
@dominikschlosser
dominikschlosser marked this pull request as ready for review July 2, 2026 15:24
@dominikschlosser
dominikschlosser requested a review from a team as a code owner July 2, 2026 15:24
Copilot AI balanced review requested due to automatic review settings July 2, 2026 15:24

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR expands the Keycloak conformance test suite for OpenID4VCI to cover the HAIP (High Assurance Interoperability Profile) plan with SD-JWT VC, including both wallet-initiated and issuer-initiated variants, and adds supporting test-runner capabilities to drive suite “WAITING” states.

Changes:

  • Extend the conformance realm/server configuration for HAIP requirements (DPoP binding, PAR/profile+policy enforcement, stricter TLS 1.2 cipher suites, randomized time claims).
  • Add many new HAIP/FAPI2 conformance test classes (issuer wallet-initiated, issuer-initiated happy flow, and negative cases).
  • Enhance the conformance runner to support multi-module variant discovery, per-module interaction callbacks, and richer browser-flow scripting.

Reviewed changes

Copilot reviewed 48 out of 48 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/VciTestSigningKey.java Places generated keystore under a realm-scoped keystores base dir for java-keystore provider validation.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/VciConformanceRealmConfig.java Adds HAIP realm/server settings (DPoP, profiles/policies, TLS ciphers, time-claim strategy) and an app client for AIA.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/VciAttesterKey.java Makes the attester key helper public for cross-package test reuse.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/VciAiaCredentialOffer.java Automates AIA login to mint and extract an issuer-initiated credential offer URI.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerUnsupportedEncryptionAlgorithmTest.java Adds HAIP issuer negative test for unsupported encryption algorithm handling.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerUnsignedMetadataTest.java Adds issuer metadata (unsigned) conformance coverage.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerUnknownCredentialIdentifierTest.java Adds issuer negative test for unknown credential identifier handling.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerUnknownCredentialConfigurationTest.java Adds issuer negative test for unknown credential configuration handling.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerSignedMetadataTest.java Moves/aligns signed metadata test into issuer package and common HAIP plan constants.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerMissingProofTest.java Adds issuer negative test for missing proof handling.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerMismatchedClientAttestationPopKeyTest.java Adds issuer negative test for mismatched client-attestation PoP key.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerKeyAttestationTest.java Moves/aligns key attestation test into issuer package and common HAIP plan constants.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerInvalidNonceTest.java Adds issuer negative test for invalid nonce handling.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerInvalidKeyAttestationSignatureTest.java Adds issuer negative test for invalid key attestation signature.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerInvalidJwtProofSignatureTest.java Adds issuer negative test for invalid JWT proof signature.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerInvalidClientAttestationSignatureTest.java Adds issuer negative test for invalid client attestation signature.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerInvalidClientAttestationPopSignatureTest.java Adds issuer negative test for invalid client attestation PoP signature.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerInitiatedRestOfferTest.java Adds (disabled) issuer-initiated REST offer test scaffold pending upstream behavior change.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerInitiatedHappyFlowTest.java Adds issuer-initiated flow support, including filtering to the plain encryption variant for now.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerHappyFlowTest.java Moves/aligns wallet-initiated happy flow test into issuer package and common HAIP plan constants.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerHappyFlowSkipNotificationTest.java Adds coverage for “skip notification” happy flow module.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerHappyFlowMultipleClientsTest.java Adds coverage for “multiple clients” happy flow module.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerHappyFlowAdditionalRequestsTest.java Adds coverage for “additional requests” happy flow module.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerClientAttestationPopWrongAudTest.java Adds issuer negative test for wrong audience in client-attestation PoP.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerClientAttestationNoSubTest.java Adds issuer negative test for client attestation without sub.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerClientAttestationExpInPastTest.java Adds issuer negative test for client attestation exp in the past.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerBatchIssuanceTest.java Adds issuer batch-issuance module coverage with realm override for batch size.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerAccessTokenInQueryTest.java Adds issuer negative test for access token in query handling.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/fapi2/IssuerUserRejectsAuthenticationTest.java Adds FAPI2 “user rejects authentication/consent” module coverage.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/fapi2/IssuerRefreshTokenTest.java Adds FAPI2 refresh token module coverage.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/fapi2/IssuerHolderOfKeyRequiredTest.java Adds FAPI2 HoK-required module coverage.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/fapi2/IssuerFapi2UnsignedRequestWithoutParTest.java Adds FAPI2 negative module for unsigned request without PAR.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/fapi2/IssuerFapi2StateOutsideRequestObjectTest.java Adds FAPI2 negative module for state only outside request object (no PAR).
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/fapi2/IssuerFapi2ReuseRequestUriTest.java Adds FAPI2 negative module for reusing consumed request_uri.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/fapi2/IssuerFapi2RequestUriReusePriorTest.java Adds module asserting request_uri reuse prior to auth completion succeeds.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/fapi2/IssuerFapi2RequestUriForDifferentClientTest.java Adds module asserting request_uri is client-bound and rejected for different client.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/fapi2/IssuerFapi2PkceRejectedTest.java Adds grouped modules for PKCE enforcement/rejection behavior.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/fapi2/IssuerFapi2NoBrowserTest.java Adds grouped “no browser required” FAPI2 modules.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/fapi2/IssuerFapi2LoginTest.java Adds grouped FAPI2 “login required” modules.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/fapi2/IssuerExpiredRequestUriTest.java Moves/aligns expired request_uri test into fapi2 package and updates expected error message.
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/AbstractVciConformanceTest.java Makes base VCI test class public, adds plan constants, and adds issuer-initiated interaction hook.
tests/conformance/src/test/java/org/keycloak/tests/conformance/runner/ModuleRun.java Adds a typed carrier for “created + info” module state for interaction callbacks.
tests/conformance/src/test/java/org/keycloak/tests/conformance/runner/ConformanceApiClient.java Adds multi-module variant discovery, interaction support during WAITING, and selective IO retry semantics.
tests/conformance/src/test/java/org/keycloak/tests/conformance/runner/BrowserInteraction.java Expands browser flow scripting to support consent denial, multi-visit behavior, and callback-only errors.
tests/conformance/src/test/java/org/keycloak/tests/conformance/runner/BrowserFlow.java Adds match-limit and optional tasks with JSON mapping support for the suite browser runner.
tests/conformance/src/test/java/org/keycloak/tests/conformance/containers/OpenIdConformanceSuite.java Bumps default OIDF conformance suite image tag.
tests/conformance/src/test/java/org/keycloak/tests/conformance/AbstractConformanceTest.java Adds multi-module variant discovery helper and introduces interaction hook into execution flow.
tests/conformance/pom.xml Bumps suite image tag property and adds UI/OAuth test framework dependencies.

@VinodAnandan

Copy link
Copy Markdown
Contributor

@pskopek @rmartinc @ahus1 @jimmychakkalakal @graziang @thomasdarimont As part of the remediation for issue #50345, we've noticed that our nightly conformance tests (https://github.com/keycloak/keycloak/actions/workflows/conformance.yml) have been failing for the past week. This PR  add more conformance tests and also addresses that issue and should help restore the nightly conformance pipeline

When you have a chance, could you please take a look and review it? If everything looks good, we'd greatly appreciate getting it merged as soon as possible. Thank you!

Copilot AI review requested due to automatic review settings July 10, 2026 13:33

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review any files in this pull request.

@keycloak-github-bot

Copy link
Copy Markdown

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.adapter.servlet.SAMLServletAdapterTest#employeeSigPostNoIdpKeyTestKeyIdAsKeyNameInKeyInfo

Keycloak CI - Adapter IT Strict Cookies

org.opentest4j.AssertionFailedError: expected: <true> but was: <false>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
	at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
...

Report flaky test

@keycloak-github-bot keycloak-github-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unreported flaky test detected, please review

Copilot AI review requested due to automatic review settings July 10, 2026 16:52

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@keycloak-github-bot keycloak-github-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unreported flaky test detected, please review

@keycloak-github-bot

Copy link
Copy Markdown

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.forms.RegisterWithUserProfileTest#testAttributeInputTypes

Keycloak CI - Forms IT (chrome)

org.openqa.selenium.NoSuchElementException: 
no such element: Unable to locate element: {"method":"link text","selector":"Register"}
  (Session info: chrome=150.0.7871.46)
For documentation on this error, please visit: https://www.selenium.dev/documentation/webdriver/troubleshooting/errors#no-such-element-exception
Build info: version: '4.28.1', revision: '73f5ad48a2'
...

Report flaky test

@forkimenjeckayang forkimenjeckayang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few comments @dominikschlosser
Let me know WYT..

Comment on lines +45 to +50
return discoverModuleVariants(
HAIP_PLAN,
ISSUER_INITIATED,
"oid4vci-1_0-issuer-happy-flow",
ConformanceResult.PASSED,
BrowserInteraction.LOGIN)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This selects only the happy-flow module for issuer_initiated, leaving 19 other issuer-initiated SD-JWT behavior rows from the pinned release-v5.1.45 HAIP plan without a JUnit invocation: 17 plain rows plus the encrypted unknown-configuration and unsupported-encryption rows. AbstractVCIIssuerTestModule declares VCIAuthorizationCodeFlowVariant for these modules and switches from performAuthorizationFlow() to waitForCredentialOffer() for issuer_initiated; no flow-applicability annotation excludes the affected subclasses. Consequently, their issuer-initiated behavior is not exercised by the wallet-initiated wrappers.

Could we discover and execute every applicable VCI module with ISSUER_INITIATED (grouping modules with the same expected result/browser interaction is fine), and add a coverage assertion/report that fails if any of the 18 plain or three encrypted issuer-initiated rows disappears ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I reproduced the pinned release-v5.1.45 plan and confirmed your numbers. But I don't think adding the rest adds real coverage:

  • The 40 FAPI2 rows are not specific to credential-offers so they basically do the same for wallet_initiated and issuer_initiated
  • The Keycloak paths that are actually unique to issuer_initiated (AIA offer creation, serving credential_offer_uri, and offer-state validation/removal in OID4VCIssuerEndpoint) are already exercised by the happy-flow test
  • The remaining 17 error modules test credential-endpoint errors that occur after authorization and are flow-agnostic (same argument as for FAPI2)

The only rows that genuinely exercise something new under issuer_initiated are the multi-request ones (happy-flow-additional-requests, batch-issuance, happy-flow-multiple-clients), and those hit the same offer-state limitation as the encrypted case.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I agree that duplicating the FAPI2 rows and the one-request credential-endpoint error cases under both authorization-code-flow selections would add limited regression value, so I am narrowing that part of my comment.
One case still looks distinct and is not blocked by #50889: oid4vci-1_0-issuer-batch-issuance sends multiple proofs in a single credential request. Keycloak iterates over those proofs and only removes the offer state after the complete response has been built (OID4VCIssuerEndpoint, lines 10101067). It therefore should not encounter the “second credential request” failure.
Could you add an ISSUER_INITIATED invocation for the batch module, or share the failure showing a different blocker?
I am happy to treat the additional-request and other genuinely multi-request cases as tracked by #50889.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added IssuerInitiatedBatchIssuanceTest

Comment on lines +51 to +55
// TODO: include the encrypted variant once Keycloak keeps the credential offer state for the
// lifetime of the authorized session. Keycloak removes the offer state after the first
// successful issuance (OID4VCIssuerEndpoint), so the encrypted variant's second credential
// request (encryption + DEFLATE compression check) fails with "No credential offer state".
.filter(module -> "plain".equals(module.moduleVariant().get("vci_credential_encryption")));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The TODO correctly identifies a current Keycloak limitation, but this filter turns that known failure into an absent JUnit row. The pinned HAIP plan publishes both plain and encrypted issuer-initiated happy-flow rows. The encrypted row makes a second credential request, using the same authorized flow with a fresh nonce/proof, to verify encryption plus zip=DEF; filtering it lets CI remain green while the PR claims complete HAIP SD-JWT coverage and closes the missing-coverage issue.

The production fix also needs to preserve credential-offer replay protection: OID4VCIssuerEndpoint currently requires the offer state during a credential request and removes it after the first successful issuance, while an existing test expects the offer URI to be unusable after consumption. Simply retaining the current state would therefore be insufficient.

Please either (1) separate offer-URI consumption from the authorization state needed by subsequent credential requests, add a product integration test proving that the offer URI remains single-use while two requests from the same authorized issuer-initiated flow succeed, and remove this filter; or (2) link a blocking product issue, represent this row as an explicit unsupported/skipped case, and narrow the PR.

In either case, please make plan coverage reporting fail or explicitly account for every generated row so future variants cannot disappear through an unreported stream filter. WDYT ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#50889 created

We don't have any mechanism for "plan coverage" right now as those tests dont have the goal to report if Keycloak is fully conformant. The main reason to have them in CI is to prevent regressions.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, #50889 captures the product limitation and the requirement to keep the offer URI single-use. I agree that a general plan-coverage reporting mechanism is outside the purpose of these regression tests, so I will not insist on that here.
Could you add #50889 to this TODO so the filtered encrypted variant is traceable from the source? With that cleanup and the issuer-initiated batch case discussed above, this concern is resolved for me.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@forkimenjeckayang forkimenjeckayang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mposolda mposolda self-assigned this Aug 3, 2026

@mposolda mposolda left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dominikschlosser @forkimenjeckayang Thanks for the PR and review!

Question: I am not 100% sure if changes from #51042 affect the setup of HAIP conformance tests in this PR? Can you please confirm if the testsuite still passing (ideally by rebase this PR to latest main) and eventually fix the setup if not?

@Awambeng

Awambeng commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

@dominikschlosser @forkimenjeckayang Thanks for the PR and review!

Question: I am not 100% sure if changes from #51042 affect the setup of HAIP conformance tests in this PR? Can you please confirm if the testsuite still passing (ideally by rebase this PR to latest main) and eventually fix the setup if not?

Hi @mposolda,

I created this ticket #51399 to address the conformance test failures related to the changes introduced by #51042.

The changes in #51042 affects the HAIP conformance test setup because key-attestation x5c validation now relies on trust-material IdPs instead of the server truststore. The current conformance setup does not provide the required X.509 trust material, which might cause the key attestation validation to fail.

Copilot AI review requested due to automatic review settings August 7, 2026 07:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 47 out of 47 changed files in this pull request and generated no new comments.

Suppressed comments (1)

tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerInitiatedRestOfferTest.java:29

  • This blocker is stale: #48188 was closed by merged PR #50576, and OID4VCIssuerEndpoint now leaves authorization-code offers unbound (targetClientId = null). Keeping this new test disabled therefore leaves the REST-created issuer-initiated path untested; either implement and enable it now or remove the dead test until it can be added.

@VinodAnandan

Copy link
Copy Markdown
Contributor

This PR has already been reviewed and approved by the SIG member and has been open for over a month. As @mposolda is currently on holiday, @vaceksimon , @rmartinc : could you please help review and merge the PR when you get a chance? Your help would be greatly appreciated.

@dominikschlosser

Copy link
Copy Markdown
Contributor Author

Tests all run successfully after rebasing main (including @Awambeng changes to key-attestation)

@VinodAnandan VinodAnandan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


// Fallbacks for running outside Maven, where the defaults are set by the pom properties of the same name
private static final String DEFAULT_IMAGE_TAG = "release-v5.1.44";
private static final String DEFAULT_IMAGE_TAG = "release-v5.2.1";

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
private static final String DEFAULT_IMAGE_TAG = "release-v5.2.1";
private static final String DEFAULT_IMAGE_TAG = "release-v5.2.2";

Comment thread tests/conformance/pom.xml Outdated

<properties>
<keycloak.conformance.imageTag>release-v5.1.44</keycloak.conformance.imageTag>
<keycloak.conformance.imageTag>release-v5.2.1</keycloak.conformance.imageTag>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<keycloak.conformance.imageTag>release-v5.2.1</keycloak.conformance.imageTag>
<keycloak.conformance.imageTag>release-v5.2.2</keycloak.conformance.imageTag>

…sd-jwt)

Signed-off-by: Dominik Schlosser <dominik.schlosser@gmail.com>
Copilot AI review requested due to automatic review settings August 10, 2026 15:52

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 53 out of 53 changed files in this pull request and generated no new comments.

Suppressed comments (3)

tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/VciAiaCredentialOffer.java:85

  • Extracting credential_offer_uri by split(\"credential_offer_uri=\") is incorrect when the href contains additional query parameters (you’ll return the value plus trailing &...) or if the parameter appears later in the query string. Parse the URL as a URI, extract the query parameters, and return only the decoded value for credential_offer_uri.
            WebElement offerLink = wait.until(
                    ExpectedConditions.presenceOfElementLocated(By.id("credential-offer-uri-link")));
            String offer = offerLink.getDomAttribute("href");
            String[] parts = offer == null ? new String[0] : offer.split("credential_offer_uri=");
            if (parts.length < 2) {
                throw new IllegalStateException("No credential_offer_uri in AIA credential offer: " + offer);
            }
            return URLDecoder.decode(parts[1], StandardCharsets.UTF_8);

tests/conformance/src/test/java/org/keycloak/tests/conformance/AbstractConformanceTest.java:94

  • This helper assumes names is non-empty (names.get(0)). Add an explicit guard (e.g., throw IllegalArgumentException) when names is empty to avoid an IndexOutOfBoundsException with a less actionable stack trace.
    protected Stream<ConformanceModuleVariant> discoverModuleVariants(String plan, Map<String, String> planVariant,
            List<String> names, ConformanceResult expectedResult, BrowserInteraction browserInteraction) {
        ConformanceModuleVariant template = new ConformanceModuleVariant(plan, planVariant, names.get(0), Map.of(),
                expectedResult, browserInteraction);
        Map<String, List<Map<String, String>>> discovered = OpenIdConformanceSuite.instance().client()
                .discoverModuleVariants(plan, planVariant, names, suiteConfig(template));
        return names.stream().flatMap(name -> discovered.get(name).stream()
                .map(moduleVariant -> new ConformanceModuleVariant(plan, planVariant, name, moduleVariant,
                        expectedResult, browserInteraction)));
    }

tests/conformance/src/test/java/org/keycloak/tests/conformance/runner/ConformanceApiClient.java:154

  • The exception message drops important context (the full endpoint path, alias, and query). Include requestPath in the error message so failures are easier to diagnose from CI logs.
    public void visitTestEndpoint(String alias, String path, String query) {
        String requestPath = "/test/a/" + alias + "/" + path + (query == null || query.isEmpty() ? "" : "?" + query);
        HttpResponse<String> response = send(request(requestPath).GET().build(), false);
        if (response.statusCode() / 100 != 2) {
            throw new IllegalStateException("Suite test endpoint " + path + " returned HTTP "
                    + response.statusCode() + ": " + response.body());
        }
    }

@mposolda mposolda left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dominikschlosser @forkimenjeckayang Thanks for the patience and all the work and review on the PR!

@mposolda
mposolda merged commit 871c2d0 into keycloak:main Aug 12, 2026
96 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[OID4VCI] Implement missing HAIP conformance tests Nightly Conformance tests are failing

7 participants