CertificateAuthentication: CustomTrustStore/CustomRootTrust is not applied to self-signed client certificates #69763

Summary

CertificateAuthenticationHandler applies the configured custom trust (CustomTrustStore together with ChainTrustValidationMode = CustomRootTrust) only to certificates it classifies as chained. Certificates classified as self-signed are validated under a separate policy that never consults that configuration. When AllowedCertificateTypes includes SelfSigned, the trust options therefore do not describe a single, uniform trust boundary, and nothing tells the operator.

What is wrong

• Broken invariant: when an app explicitly selects custom root trust and supplies a populated store, that store should be the set of roots the handler trusts for every certificate type it accepts. Today it applies to one type only.
• Origin: the self-signed branch in BuildChainPolicy predates CustomTrustStore. The store and trust mode were added later (Allow opt-out from client certificate root checking #15029) to support chained certificates issued by a private CA, and the self-signed branch was not revisited. No test covers the combination of a self-signed certificate with CustomRootTrust.
• Silent: the combination is accepted without a log message or validation error.
• Classifier: the self-signed/chained decision is made by IsSelfSigned(), which compares subject and issuer names. It selects which validation policy runs, so the fix should not depend on its precision.

Why it matters (defense in depth)

• Correctness and least surprise: the platform documents X509ChainPolicy.CustomTrustStore as "a collection of certificates replacing the default certificate trust". The handler's XML docs describe CustomTrustStore as "trusted components of the certificate chain" used in chain validation. Neither says self-signed certificates are exempt, so a reasonable reader expects the store to apply.
• Hardening rationale: the documentation already requires apps that accept self-signed certificates to pin them (thumbprint or issuer) in their own validation, see https://learn.microsoft.com/aspnet/core/security/authentication/certauth. Making the framework honor an explicitly configured trust store for self-signed certificates adds a second layer that does not rely on that app-side step being present.
• Scope: non-default configuration only. The default is CertificateTypes.Chained with system trust.

Affected code

• src/Security/Authentication/Certificate/src/CertificateAuthenticationHandler.cs:204-252 - BuildChainPolicy
• src/Security/Authentication/Certificate/src/CertificateAuthenticationHandler.cs:228-233 - self-signed branch: sets AllowUnknownCertificateAuthority and IgnoreEndRevocationUnknown, does not apply CustomTrustStore or TrustMode
• src/Security/Authentication/Certificate/src/CertificateAuthenticationHandler.cs:234-242 - chained branch: the only place CustomTrustStore and ChainTrustValidationMode are applied
• src/Security/Authentication/Certificate/src/CertificateAuthenticationHandler.cs:118-146 - ValidateCertificateAsync: classification and chain.Build
• src/Security/Authentication/Certificate/src/X509CertificateExtensions.cs:19-24 - IsSelfSigned (name comparison)
• src/Security/Authentication/Certificate/src/CertificateAuthenticationOptions.cs:21-27 - AllowedCertificateTypes docs
• src/Security/Authentication/Certificate/src/CertificateAuthenticationOptions.cs:29-32 - CustomTrustStore docs
• src/Security/Authentication/Certificate/src/CertificateAuthenticationOptions.cs:38-46 - ChainTrustValidationMode docs
• src/Security/Authentication/Certificate/src/CertificateTypes.cs:17-25 - SelfSigned and All docs
• src/Security/Authentication/test/CertificateTests.cs:52-260 - existing self-signed tests (no custom-root coverage)
• src/Security/Authentication/test/CertificateTests.cs:319-386 - existing custom-root tests (chained only)

Recommended fix

Deliver in two tiers. Tier 1 is non-breaking and can ship first, including to servicing branches. Tier 2 changes behavior and needs an owner decision on compatibility.

Tier 1: documentation, signal, test coverage (non-breaking)

1. Docs: in the AllowedCertificateTypes and CustomTrustStore sections of the certificate authentication article, state that self-signed certificates are not evaluated against CustomTrustStore/CustomRootTrust and must be pinned by thumbprint or public key in OnCertificateValidated or an authorization policy. Move the pinning guidance out of the sample walkthrough into those sections. Mirror the note in the XML docs on AllowedCertificateTypes, CustomTrustStore, ChainTrustValidationMode, and CertificateTypes.SelfSigned.
2. Startup signal: log one warning per scheme, at options configuration time, when AllowedCertificateTypes includes SelfSigned and ChainTrustValidationMode is CustomRootTrust with a non-empty CustomTrustStore. Do not throw; the combination is legitimate for apps that accept private-CA chains and pin self-signed certificates themselves. Do not log per request.
3. Tests: add tests that pin current behavior for a self-signed certificate under CustomRootTrust so any later change is deliberate.

Tier 2: apply custom root trust to self-signed certificates (behavior change)

• Selected semantics: define useCustomRoot = ChainTrustValidationMode == CustomRootTrust && CustomTrustStore.Count > 0.
  • When useCustomRoot is true, apply CustomTrustStore and TrustMode to every certificate, and do not set AllowUnknownCertificateAuthority for self-signed certificates. A self-signed certificate then validates only if it is present in the store.
  • When useCustomRoot is false, keep today's behavior unchanged.
  • Keep the existing self-signed handling for revocation (NoCheck, IgnoreEndRevocationUnknown), the ExtraStore additions, the client-auth EKU check, and the validity-period check.
  • The non-empty guard is required: with an empty store and CustomRootTrust, every self-signed certificate would fail.
• Platform validation (Windows, .NET 10):
  • With CustomRootTrust, a populated store, and AllowUnknownCertificateAuthority not set, a self-signed certificate validates only when it is in the store and otherwise fails with UntrustedRoot.
  • If AllowUnknownCertificateAuthority stays set, the store has no effect on a self-signed certificate, so the flag must be dropped in this mode.
  • Linux and macOS chain engines still need to be checked.
• Independent of IsSelfSigned: in useCustomRoot mode, trust no longer depends on the classification, so tightening the classifier is optional and separate.

Alternatives considered

• Throw at startup for the combination: rejected, it breaks legitimate setups that pin self-signed certificates in app code.
• Tighten IsSelfSigned only: rejected, it does not change trust semantics for a valid self-signed certificate.
• Always apply trust to self-signed certificates, or drop AllowUnknownCertificateAuthority unconditionally: rejected, it breaks all self-signed users on system trust.
• Dedicated new option for self-signed trust: viable, but it is new public API (API review required) and heavier than reusing the existing options.
• Docs only: insufficient on its own because the failure mode is silent.

Compatibility and versioning

• Tier 2 breaks apps that use CustomRootTrust with a populated store and AllowedCertificateTypes including SelfSigned and rely on self-signed certificates outside the store being accepted today.
• Owner decision needed: (a) opt-in first through an AppContext switch or a new option (a new public option requires the API review process and a PublicAPI.Unshipped.txt entry), then make it the default in the next major with a breaking-change article; or (b) make it the default in the next major only.
• Servicing branches: Tier 1 only.

Acceptance criteria

• Tier 1 docs and XML docs state that self-signed certificates are not evaluated against CustomTrustStore/CustomRootTrust and need app-side pinning.
• A single warning per scheme is logged for the self-signed plus populated-custom-root combination, and nothing is logged for other combinations.
• Tests cover self-signed certificates under CustomRootTrust (certificate in store, certificate not in store, empty store) and the unchanged behavior under system trust.
• If Tier 2 is accepted: with useCustomRoot, a self-signed certificate outside the store is rejected and one inside the store is accepted; behavior without useCustomRoot is unchanged.
• Tier 2 behavior is verified on Windows, Linux, and macOS.
• The compatibility decision, any API review for a new option, and the breaking-change documentation are recorded before Tier 2 ships.