Add expired_tls_hotel_reservation problem - #784

Real-World Failure

An expired TLS certificates are an extremely common cause of system outages. Unlike most failures, they are completely predictable since every certificate has a know expiry date; however, they still manage to take down major services. I previously mentioned in issue #763 some large cases where this has happened.

How I Simulated The Failure

The fault is injected into the Hotel Reservation app on its hotel-reservation namespace. The simulation required two additional Kubernetes resources:

1. The Expired Certificate
   • The fault injector generates a new certificate at injection time using Python's cryptography library. At first I thought to use OpenSSL to generate the certificate however that library is strict about the date you set and wouldn't allow me to create an already expire certificate.
   • I also initially thought to write the certificate to the disk to avoid complex base64 string encoding, but I changed my implementation to base64 simply to keep everything local.
2. Frontend Ingress
   • I added a new Kubernetes Ingress resource (frontend-ingress) which is applied to the namespace. It references the expired secret via the standard Kubernetes TLS Ingress configuration.

Like I stated above, I used Pythons cryptography package to generate the expired x509 certificate, and to parse the tls.crt PEM bytes from the Kubernetes Secret in order to read the not_valid_after_utc.
In order to use it I ran:

uv add cryptography

If not installed, both the inject_fault and the oracle's _get_cert_expiry methods log an ImportError with the fix command rather than producing a confusing traceback.

Problem Runtime

After the inject_fault() runs, the cluster has the following observable state:
Kubernetes resources:

kubectl get ingress -n hotel-reservation
NAME               CLASS   HOSTS         ADDRESS   PORTS     AGE
frontend-ingress   nginx   hotel.local             80, 443   Xs
 
kubectl describe ingress frontend-ingress -n hotel-reservation
# Shows: TLS terminates with secret "expired-frontend-cert"
# Annotation: nginx.ingress.kubernetes.io/ssl-redirect: "true"
 
kubectl get secret expired-frontend-cert -n hotel-reservation
NAME                    TYPE                DATA   AGE
expired-frontend-cert   kubernetes.io/tls   2      Xs

Certificate:

kubectl get secret expired-frontend-cert -n hotel-reservation \
  -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -dates
notBefore=<10 days ago>
notAfter=<1 day ago>

Pod health:

kubectl get pods -n hotel-reservation
# All pods show Running/Ready

HTTPS Behaviour:

• Requests to https://hotel.local result in a failed TLS handshake
• the NGINX Ingress controller logs chow certificate validation errors
• HTTP requests are redirected to HTTPS (because of ssl-redirect: "true" in frontend-ingress) so there are no unencrypted fallback paths. Any path will be affected.

The challenge for an AI agent is that at first glance everything seems to be working:

1. kubectl get pods -A shows all green
2. Application logs show no errors at the app layer since TLS is terminated at the Ingress.
3. Service/Deployment configs don't show anything abnormal

Oracle Behavior

Before I get into Agent behavior, I want to talk about the new Oracle I had to design in order for this failure to work as expected. The default MitigationOracle checks pod health, which in this case is always fine since the expired TLS certificate does not affect pod health.
Therefore I made ExpiredTlsMitigationOracle to inspect the frontend Ingress at submission time and checks the expiry of any TLS secret it references. It will accept any of the following as valid mitigations: deleting the Ingress, deleting the secret, removing the TLS block from the Ingress, or replacing the secret with a valid certificate.
It will reject the submission if any referenced secret still contains an expired certificate.
I tested it in both cases:

• submit() with failure active: success: false, "Secret 'expired-frontend-cert' still contains a certificate that expired on <timestamp>"
• submit() after removing the Ingress and secret: success: true, "frontend-ingress has been deleted — mitigation accepted"

The Agent Behavior

Human Agent (me):

While I may have had some insider information about the problem, I was able to diagnose the issue and resolve it. I ran the following command to diagnose the failure:

kubectl get ingress -n hotel-reservation

This showed me the frontend-ingress referencing the expired-frontend-cert you could then inspect the cert to verify it is expired. Then I ran:

kubectl delete ingress frontend-ingress -n hotel-reservation
kubectl delete secret expired-frontend-cert -n hotel-reservation

This way I deleted the Ingress and secret to mitigate the failure.

AI Agent (Stratus, gemini/gemini-3.1-flash-lite)

The AI agent failed both stages across multiple attempts. During diagnosis, it checked the Jaeger for telemtry traces, found none, and submitted an incorrect root cause about services not sending data to Jaeger. Then, during mitigation, the agent read the frontend logs and found connection refused errors from the frontend trying to reach Consul. These errors are normal and go away on their own. It spent all 20 steps investing this error and never ran kubectl get ingress -n hotel-reservation
Note: Testing was also attempted with gemini/gemini-2.5-flash but hit the free-tier rate limit (5 RPM, 20 RPD). gemini/gemini-3.1-flash-lite was used for all completed runs.

Additional Change

Outside of the actual problem, I noticed that whenever I pressed enter without a prompt in the CLI, it would crash. I added a check that way anytime you press enter, whether on accident or not, the CLI doesn't just crash. If that was a feature and not a bug let me know I will remove that addition.

/validate-problem expired_tls_hotel_reservation