[OpenID4VCI] Implement all HAIP conformance tests (wallet_initiated, sd-jwt) - #50577
Conversation
789971d to
b0c4a10
Compare
There was a problem hiding this comment.
Pull request overview
This PR expands the Keycloak conformance test suite for OpenID4VCI to cover the HAIP (High Assurance Interoperability Profile) plan with SD-JWT VC, including both wallet-initiated and issuer-initiated variants, and adds supporting test-runner capabilities to drive suite “WAITING” states.
Changes:
- Extend the conformance realm/server configuration for HAIP requirements (DPoP binding, PAR/profile+policy enforcement, stricter TLS 1.2 cipher suites, randomized time claims).
- Add many new HAIP/FAPI2 conformance test classes (issuer wallet-initiated, issuer-initiated happy flow, and negative cases).
- Enhance the conformance runner to support multi-module variant discovery, per-module interaction callbacks, and richer browser-flow scripting.
Reviewed changes
Copilot reviewed 48 out of 48 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/VciTestSigningKey.java | Places generated keystore under a realm-scoped keystores base dir for java-keystore provider validation. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/VciConformanceRealmConfig.java | Adds HAIP realm/server settings (DPoP, profiles/policies, TLS ciphers, time-claim strategy) and an app client for AIA. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/VciAttesterKey.java | Makes the attester key helper public for cross-package test reuse. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/VciAiaCredentialOffer.java | Automates AIA login to mint and extract an issuer-initiated credential offer URI. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerUnsupportedEncryptionAlgorithmTest.java | Adds HAIP issuer negative test for unsupported encryption algorithm handling. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerUnsignedMetadataTest.java | Adds issuer metadata (unsigned) conformance coverage. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerUnknownCredentialIdentifierTest.java | Adds issuer negative test for unknown credential identifier handling. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerUnknownCredentialConfigurationTest.java | Adds issuer negative test for unknown credential configuration handling. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerSignedMetadataTest.java | Moves/aligns signed metadata test into issuer package and common HAIP plan constants. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerMissingProofTest.java | Adds issuer negative test for missing proof handling. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerMismatchedClientAttestationPopKeyTest.java | Adds issuer negative test for mismatched client-attestation PoP key. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerKeyAttestationTest.java | Moves/aligns key attestation test into issuer package and common HAIP plan constants. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerInvalidNonceTest.java | Adds issuer negative test for invalid nonce handling. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerInvalidKeyAttestationSignatureTest.java | Adds issuer negative test for invalid key attestation signature. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerInvalidJwtProofSignatureTest.java | Adds issuer negative test for invalid JWT proof signature. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerInvalidClientAttestationSignatureTest.java | Adds issuer negative test for invalid client attestation signature. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerInvalidClientAttestationPopSignatureTest.java | Adds issuer negative test for invalid client attestation PoP signature. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerInitiatedRestOfferTest.java | Adds (disabled) issuer-initiated REST offer test scaffold pending upstream behavior change. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerInitiatedHappyFlowTest.java | Adds issuer-initiated flow support, including filtering to the plain encryption variant for now. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerHappyFlowTest.java | Moves/aligns wallet-initiated happy flow test into issuer package and common HAIP plan constants. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerHappyFlowSkipNotificationTest.java | Adds coverage for “skip notification” happy flow module. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerHappyFlowMultipleClientsTest.java | Adds coverage for “multiple clients” happy flow module. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerHappyFlowAdditionalRequestsTest.java | Adds coverage for “additional requests” happy flow module. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerClientAttestationPopWrongAudTest.java | Adds issuer negative test for wrong audience in client-attestation PoP. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerClientAttestationNoSubTest.java | Adds issuer negative test for client attestation without sub. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerClientAttestationExpInPastTest.java | Adds issuer negative test for client attestation exp in the past. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerBatchIssuanceTest.java | Adds issuer batch-issuance module coverage with realm override for batch size. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerAccessTokenInQueryTest.java | Adds issuer negative test for access token in query handling. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/fapi2/IssuerUserRejectsAuthenticationTest.java | Adds FAPI2 “user rejects authentication/consent” module coverage. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/fapi2/IssuerRefreshTokenTest.java | Adds FAPI2 refresh token module coverage. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/fapi2/IssuerHolderOfKeyRequiredTest.java | Adds FAPI2 HoK-required module coverage. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/fapi2/IssuerFapi2UnsignedRequestWithoutParTest.java | Adds FAPI2 negative module for unsigned request without PAR. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/fapi2/IssuerFapi2StateOutsideRequestObjectTest.java | Adds FAPI2 negative module for state only outside request object (no PAR). |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/fapi2/IssuerFapi2ReuseRequestUriTest.java | Adds FAPI2 negative module for reusing consumed request_uri. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/fapi2/IssuerFapi2RequestUriReusePriorTest.java | Adds module asserting request_uri reuse prior to auth completion succeeds. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/fapi2/IssuerFapi2RequestUriForDifferentClientTest.java | Adds module asserting request_uri is client-bound and rejected for different client. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/fapi2/IssuerFapi2PkceRejectedTest.java | Adds grouped modules for PKCE enforcement/rejection behavior. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/fapi2/IssuerFapi2NoBrowserTest.java | Adds grouped “no browser required” FAPI2 modules. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/fapi2/IssuerFapi2LoginTest.java | Adds grouped FAPI2 “login required” modules. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/fapi2/IssuerExpiredRequestUriTest.java | Moves/aligns expired request_uri test into fapi2 package and updates expected error message. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/AbstractVciConformanceTest.java | Makes base VCI test class public, adds plan constants, and adds issuer-initiated interaction hook. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/runner/ModuleRun.java | Adds a typed carrier for “created + info” module state for interaction callbacks. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/runner/ConformanceApiClient.java | Adds multi-module variant discovery, interaction support during WAITING, and selective IO retry semantics. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/runner/BrowserInteraction.java | Expands browser flow scripting to support consent denial, multi-visit behavior, and callback-only errors. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/runner/BrowserFlow.java | Adds match-limit and optional tasks with JSON mapping support for the suite browser runner. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/containers/OpenIdConformanceSuite.java | Bumps default OIDF conformance suite image tag. |
| tests/conformance/src/test/java/org/keycloak/tests/conformance/AbstractConformanceTest.java | Adds multi-module variant discovery helper and introduces interaction hook into execution flow. |
| tests/conformance/pom.xml | Bumps suite image tag property and adds UI/OAuth test framework dependencies. |
b0c4a10 to
8ba7479
Compare
|
@pskopek @rmartinc @ahus1 @jimmychakkalakal @graziang @thomasdarimont As part of the remediation for issue #50345, we've noticed that our nightly conformance tests (https://github.com/keycloak/keycloak/actions/workflows/conformance.yml) have been failing for the past week. This PR add more conformance tests and also addresses that issue and should help restore the nightly conformance pipeline When you have a chance, could you please take a look and review it? If everything looks good, we'd greatly appreciate getting it merged as soon as possible. Thank you! |
8ba7479 to
f9171ff
Compare
Unreported flaky test detectedIf the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR. org.keycloak.testsuite.adapter.servlet.SAMLServletAdapterTest#employeeSigPostNoIdpKeyTestKeyIdAsKeyNameInKeyInfoKeycloak CI - Adapter IT Strict Cookies |
0d8cf83 to
2696da7
Compare
Unreported flaky test detectedIf the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR. org.keycloak.testsuite.forms.RegisterWithUserProfileTest#testAttributeInputTypesKeycloak CI - Forms IT (chrome) |
forkimenjeckayang
left a comment
There was a problem hiding this comment.
A few comments @dominikschlosser
Let me know WYT..
| return discoverModuleVariants( | ||
| HAIP_PLAN, | ||
| ISSUER_INITIATED, | ||
| "oid4vci-1_0-issuer-happy-flow", | ||
| ConformanceResult.PASSED, | ||
| BrowserInteraction.LOGIN) |
There was a problem hiding this comment.
This selects only the happy-flow module for issuer_initiated, leaving 19 other issuer-initiated SD-JWT behavior rows from the pinned release-v5.1.45 HAIP plan without a JUnit invocation: 17 plain rows plus the encrypted unknown-configuration and unsupported-encryption rows. AbstractVCIIssuerTestModule declares VCIAuthorizationCodeFlowVariant for these modules and switches from performAuthorizationFlow() to waitForCredentialOffer() for issuer_initiated; no flow-applicability annotation excludes the affected subclasses. Consequently, their issuer-initiated behavior is not exercised by the wallet-initiated wrappers.
Could we discover and execute every applicable VCI module with ISSUER_INITIATED (grouping modules with the same expected result/browser interaction is fine), and add a coverage assertion/report that fails if any of the 18 plain or three encrypted issuer-initiated rows disappears ?
There was a problem hiding this comment.
Thanks! I reproduced the pinned release-v5.1.45 plan and confirmed your numbers. But I don't think adding the rest adds real coverage:
- The 40 FAPI2 rows are not specific to credential-offers so they basically do the same for wallet_initiated and issuer_initiated
- The Keycloak paths that are actually unique to issuer_initiated (AIA offer creation, serving credential_offer_uri, and offer-state validation/removal in OID4VCIssuerEndpoint) are already exercised by the happy-flow test
- The remaining 17 error modules test credential-endpoint errors that occur after authorization and are flow-agnostic (same argument as for FAPI2)
The only rows that genuinely exercise something new under issuer_initiated are the multi-request ones (happy-flow-additional-requests, batch-issuance, happy-flow-multiple-clients), and those hit the same offer-state limitation as the encrypted case.
There was a problem hiding this comment.
Thanks, I agree that duplicating the FAPI2 rows and the one-request credential-endpoint error cases under both authorization-code-flow selections would add limited regression value, so I am narrowing that part of my comment.
One case still looks distinct and is not blocked by #50889: oid4vci-1_0-issuer-batch-issuance sends multiple proofs in a single credential request. Keycloak iterates over those proofs and only removes the offer state after the complete response has been built (OID4VCIssuerEndpoint, lines 1010–1067). It therefore should not encounter the “second credential request” failure.
Could you add an ISSUER_INITIATED invocation for the batch module, or share the failure showing a different blocker?
I am happy to treat the additional-request and other genuinely multi-request cases as tracked by #50889.
There was a problem hiding this comment.
Added IssuerInitiatedBatchIssuanceTest
| // TODO: include the encrypted variant once Keycloak keeps the credential offer state for the | ||
| // lifetime of the authorized session. Keycloak removes the offer state after the first | ||
| // successful issuance (OID4VCIssuerEndpoint), so the encrypted variant's second credential | ||
| // request (encryption + DEFLATE compression check) fails with "No credential offer state". | ||
| .filter(module -> "plain".equals(module.moduleVariant().get("vci_credential_encryption"))); |
There was a problem hiding this comment.
The TODO correctly identifies a current Keycloak limitation, but this filter turns that known failure into an absent JUnit row. The pinned HAIP plan publishes both plain and encrypted issuer-initiated happy-flow rows. The encrypted row makes a second credential request, using the same authorized flow with a fresh nonce/proof, to verify encryption plus zip=DEF; filtering it lets CI remain green while the PR claims complete HAIP SD-JWT coverage and closes the missing-coverage issue.
The production fix also needs to preserve credential-offer replay protection: OID4VCIssuerEndpoint currently requires the offer state during a credential request and removes it after the first successful issuance, while an existing test expects the offer URI to be unusable after consumption. Simply retaining the current state would therefore be insufficient.
Please either (1) separate offer-URI consumption from the authorization state needed by subsequent credential requests, add a product integration test proving that the offer URI remains single-use while two requests from the same authorized issuer-initiated flow succeed, and remove this filter; or (2) link a blocking product issue, represent this row as an explicit unsupported/skipped case, and narrow the PR.
In either case, please make plan coverage reporting fail or explicitly account for every generated row so future variants cannot disappear through an unreported stream filter. WDYT ?
There was a problem hiding this comment.
#50889 created
We don't have any mechanism for "plan coverage" right now as those tests dont have the goal to report if Keycloak is fully conformant. The main reason to have them in CI is to prevent regressions.
There was a problem hiding this comment.
Thanks, #50889 captures the product limitation and the requirement to keep the offer URI single-use. I agree that a general plan-coverage reporting mechanism is outside the purpose of these regression tests, so I will not insist on that here.
Could you add #50889 to this TODO so the filtered encrypted variant is traceable from the source? With that cleanup and the issuer-initiated batch case discussed above, this concern is resolved for me.
2696da7 to
a7c77be
Compare
a7c77be to
e7a97ed
Compare
e7a97ed to
ba9a909
Compare
mposolda
left a comment
There was a problem hiding this comment.
@dominikschlosser @forkimenjeckayang Thanks for the PR and review!
Question: I am not 100% sure if changes from #51042 affect the setup of HAIP conformance tests in this PR? Can you please confirm if the testsuite still passing (ideally by rebase this PR to latest main) and eventually fix the setup if not?
Hi @mposolda, I created this ticket #51399 to address the conformance test failures related to the changes introduced by #51042. The changes in #51042 affects the HAIP conformance test setup because key-attestation x5c validation now relies on trust-material IdPs instead of the server truststore. The current conformance setup does not provide the required X.509 trust material, which might cause the key attestation validation to fail. |
ba9a909 to
d952166
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 47 out of 47 changed files in this pull request and generated no new comments.
Suppressed comments (1)
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/issuer/IssuerInitiatedRestOfferTest.java:29
- This blocker is stale: #48188 was closed by merged PR #50576, and
OID4VCIssuerEndpointnow leaves authorization-code offers unbound (targetClientId = null). Keeping this new test disabled therefore leaves the REST-created issuer-initiated path untested; either implement and enable it now or remove the dead test until it can be added.
|
This PR has already been reviewed and approved by the SIG member and has been open for over a month. As @mposolda is currently on holiday, @vaceksimon , @rmartinc : could you please help review and merge the PR when you get a chance? Your help would be greatly appreciated. |
|
Tests all run successfully after rebasing main (including @Awambeng changes to key-attestation) |
There was a problem hiding this comment.
More information: https://gitlab.com/openid/conformance-suite/-/releases/release-v5.2.2
|
|
||
| // Fallbacks for running outside Maven, where the defaults are set by the pom properties of the same name | ||
| private static final String DEFAULT_IMAGE_TAG = "release-v5.1.44"; | ||
| private static final String DEFAULT_IMAGE_TAG = "release-v5.2.1"; |
There was a problem hiding this comment.
| private static final String DEFAULT_IMAGE_TAG = "release-v5.2.1"; | |
| private static final String DEFAULT_IMAGE_TAG = "release-v5.2.2"; |
|
|
||
| <properties> | ||
| <keycloak.conformance.imageTag>release-v5.1.44</keycloak.conformance.imageTag> | ||
| <keycloak.conformance.imageTag>release-v5.2.1</keycloak.conformance.imageTag> |
There was a problem hiding this comment.
| <keycloak.conformance.imageTag>release-v5.2.1</keycloak.conformance.imageTag> | |
| <keycloak.conformance.imageTag>release-v5.2.2</keycloak.conformance.imageTag> |
…sd-jwt) Signed-off-by: Dominik Schlosser <dominik.schlosser@gmail.com>
d952166 to
34e509d
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 53 out of 53 changed files in this pull request and generated no new comments.
Suppressed comments (3)
tests/conformance/src/test/java/org/keycloak/tests/conformance/vci/VciAiaCredentialOffer.java:85
- Extracting
credential_offer_uribysplit(\"credential_offer_uri=\")is incorrect when thehrefcontains additional query parameters (you’ll return the value plus trailing&...) or if the parameter appears later in the query string. Parse the URL as aURI, extract the query parameters, and return only the decoded value forcredential_offer_uri.
WebElement offerLink = wait.until(
ExpectedConditions.presenceOfElementLocated(By.id("credential-offer-uri-link")));
String offer = offerLink.getDomAttribute("href");
String[] parts = offer == null ? new String[0] : offer.split("credential_offer_uri=");
if (parts.length < 2) {
throw new IllegalStateException("No credential_offer_uri in AIA credential offer: " + offer);
}
return URLDecoder.decode(parts[1], StandardCharsets.UTF_8);
tests/conformance/src/test/java/org/keycloak/tests/conformance/AbstractConformanceTest.java:94
- This helper assumes
namesis non-empty (names.get(0)). Add an explicit guard (e.g., throwIllegalArgumentException) whennamesis empty to avoid anIndexOutOfBoundsExceptionwith a less actionable stack trace.
protected Stream<ConformanceModuleVariant> discoverModuleVariants(String plan, Map<String, String> planVariant,
List<String> names, ConformanceResult expectedResult, BrowserInteraction browserInteraction) {
ConformanceModuleVariant template = new ConformanceModuleVariant(plan, planVariant, names.get(0), Map.of(),
expectedResult, browserInteraction);
Map<String, List<Map<String, String>>> discovered = OpenIdConformanceSuite.instance().client()
.discoverModuleVariants(plan, planVariant, names, suiteConfig(template));
return names.stream().flatMap(name -> discovered.get(name).stream()
.map(moduleVariant -> new ConformanceModuleVariant(plan, planVariant, name, moduleVariant,
expectedResult, browserInteraction)));
}
tests/conformance/src/test/java/org/keycloak/tests/conformance/runner/ConformanceApiClient.java:154
- The exception message drops important context (the full endpoint path, alias, and query). Include
requestPathin the error message so failures are easier to diagnose from CI logs.
public void visitTestEndpoint(String alias, String path, String query) {
String requestPath = "/test/a/" + alias + "/" + path + (query == null || query.isEmpty() ? "" : "?" + query);
HttpResponse<String> response = send(request(requestPath).GET().build(), false);
if (response.statusCode() / 100 != 2) {
throw new IllegalStateException("Suite test endpoint " + path + " returned HTTP "
+ response.statusCode() + ": " + response.body());
}
}
mposolda
left a comment
There was a problem hiding this comment.
@dominikschlosser @forkimenjeckayang Thanks for the patience and all the work and review on the PR!
Closes #50694
Closes #50695
@thomasdarimont @mposolda @tdiesler @VinodAnandan