Problems derived from real-world postmortem reports (UIUC++ SRSE) #779

This is a running document to track the status of evaluation tasks.

Finished!

1. Add service wrong pod selection benchmark for Hotel Reservation #757 @jackj6441
2. Add admission_webhook_outage_hotel_reservation problem #758 @samiamjidkhan
3. Add init_container_dependency_hang problem #762 @JamesWeng0428
4. Add Cloudflare-Style WAF Regex CPU Saturation #773 @RedwanulKarim612
5. Add admission webhook TLS mismatch problem #777 @mohamedharake
6. Add node conntrack exhaustion hotel reservation problem #768 @munimthahmid
7. Add pod_cidr_exhaustion_hotel_reservation benchmark problem #774 @MohammadElsharqawy
8. A Kafka poison-pill head-of-line block problem #790 @TheDeadcoder
9. Add expired_tls_hotel_reservation problem #784 @gAngelov24
10. Add Problem: Kubelet Eviction Threshold Misconfig #800 @parthokunda
11. New Problem | HPA missing effective CPU request  #820 @MABDz
12. Add internalTrafficPolicy: Local fault simulation for astronomy-shop #805 @sharyash81
13. Add cronjob_sidecar_blocks_completion_hotel_reservation problem #814 @selen0phile
14. Add dev_shm_exhaustion_hotel_reservation problem #829 @SuMayaBee
15. Add priority preemption cascade problem #821 @thromel
16. ephemeral_port_range_hotel_reservation benchmark problem #831 @sukanya1426
17. Add mutating_webhook_resource_limits problem #847 @keremakcali
18. Add file_descriptor_exhaustion problem #832 @TejasShukla2007
19. Add: secret_rotation_stale_env_credentials_astronomy_shop problem #851 @Ipetr14
20. Finalizer deadlock controller clean #873 @ermias19
21. Add cumulative_admission_webhook_timeout_hotel_reservation problem #886 @riyadomf
22. Add psa_restricted_blocks_recreation problem #891 @TamimEhsan
23. Add Calico route-reflector label drift problem #828 @thromel
24. Add node_clock_drift Problem  #894 @haani-maybe
25. Add problem: feature_flag_latent_bug_hotel_reservation #791 @MohammadElsharqawy
26. Add nightly rebalance problem, RFC 862 #901 @iamyeezuschrist
27. Kafka Producer Leak Causing Broker OOM on Astronomy Shop #935 @vsmart-06
28. Add cfs_cpu_throttling_hotel_reservation problem #807 @varuni7
29. postgres_lock_contention_product_catalog problem (astronomy-shop) #927 @saihariG
30. feat(benchmark): Add stale_hostaliases_dns_poisoning_astronomy_shop problem #960 @Talhaasif7
31. postgres_integer_overflow_primary_key problem (astronomy-shop) #1005 @saihariG

PR Closed (Let's try again!)

• Add: unit_mismatch_gomemlimit_astronomy_shop problem #992 @sidbhanushali
• feat(problem): add limitrange_quota_exhaustion_hotel_reservation problem #913 @kabirshahriar468
• Add database connection pool exhaustion problem for Hotel Reservation #907 @Joshi2502
• Add runAsNonRoot security context fault for hotel-reservation #888 @talha-asgher
• [Problem RFC] Missing Image Pull Secrets in hotel-reservation app #883 @W-OK-E
• Add - native_sidecar_crashloop_hotel_reservation` #881 @labid-al-nahiyan-avro
• Add hotel reservation page-cache thrash problem #866 @alpblkba
• add: coredns_saturation_ndots problem #865 @Mr-NOBY
• add EndpointSlice stale-after-rollout fault in hotelReservation #864 @irfanAbir1231
• Fix/finalizer deadlock mitigation #858 @ermias19
• Add readiness probe missing binary fault simulation #856 @ziad-osama04
• GitOps Mass Deletion — Skyscanner 2021 Global Outage Simulation #855 @abhishekkumar1101
• Add: connection_pool_exhaustion_hotel_reservation problem #848 @Neilblaze
• Add: oversized_config_artifact_hotel_reservation problem #845 @Neilblaze
• Add secret_wrong_key_hotel_reservation problem #839 @haisumharoon
• Add: incomplete rollout version skew problem for hotel reservation app #838 @Neilblaze
• Add low_memory_limit_hotel_reservation: OOMKilled CrashLoop from a too-low container memory limit #836 @TahseenSust
• Add PodDisruptionBudget-blocks-node-drain problem #835 @abdallahade1
• Add service protocol mismatch problem #825 @afridi044
• Add node_routing_table_wipeout problem #818 @keremakcali
• Add external readiness gate stuck Hotel Reservation problem #801 @MrAnayDongre
• feat: implement ephemeral storage pod eviction failure scenarios #798 @aashutoshav
• Add ImagePullBackOff problem variants for Hotel Reservation #776 @varuni7
• feat: add cascading OOM hotel reservation failure scenario #772 @Michael-mjk
• Add pdb_block_hotel_reservation problem #764 @arnavk23
• Enable Top-of-Rack Router Failure problem #761 @arnavk23

RFC Received

• [Problem RFC] PostgreSQL identity sequence exhaustion blocks product-review writes #971 @saihariG
• [Problem RFC] Invalid GOMEMLIMIT unit suffix crash-loops Go services #970 @sidbhanushali
• [Problem RFC] Pod-Local DNS Poisoning: Stale hostAliases Shadows In-Cluster Resolution #943 @Talhaasif7
• [Problem RFC] Scaling Deadlock: Unmet topologySpreadConstraint Causes HPA Scale-up Failure #938 @Talhaasif7
• [Problem RFC] Node OOMKill via Rogue CronJob leading to Cascading Failures #930 @Talhaasif7
• [Problem RFC] A missing index on product-catalog's catalog.products forces sequential scans and degrades every read (astronomy-shop) #928 @Ashish301101
• [Problem RFC] A Stuck 'ACCESS EXCLUSIVE' lock on product-catalog's table blocks all the reads (astronomy-shop) #925 @saihariG
• [Problem RFC] Overlay network MTU mismatch — large packets silently dropped while small requests succeed #921 @Ashish301101
• [Problem RFC] Consul service-registry failure silently breaks discovery (Roblox 2021-style) in Hotel Reservation #889 @Md-Arif-Hasan
• [Problem RFC] Pod Security Admission "restricted" label silently blocks pod recreation #887 implemented by Add psa_restricted_blocks_recreation problem #891 @TamimEhsan
• Missing Geo Service Endpoint in Hotel Reservation #884 @nabila-sheona
• [Problem RFC] ResourceQuota exhaustion blocks pod creation #880 @Md-Arif-Hasan
• [Problem RFC] Kubernetes Rolling Update Stalls Due to Pod Count ResourceQuota #879 @labid-al-nahiyan-avro
• [Problem RFC] ConfigMap subPath silent staleness #878 @riyadomf
• [Problem RFC] Enterprise Proxy Misconfiguration Blocks Internal Service Communication #868 @shashwht
• [Problem RFC] PagerDuty-Style Kafka Producer Leak Exhausts Broker Memory #867 @vsmart-06
• [Problem RFC] OOMs from nightly rebalancer #862 @iamyeezuschrist
• [Problem RFC] Startup probe misconfiguration kills astronomy-shop ad service before JVM initializes #859 @TamimEhsan
• [Problem RFC] topologySpreadConstraint with missing node label silently deadlocks pod scheduling #854 @haisumharoon
• [Problem RFC] Missing imagePullSecret Dependency Fault #852 @W-OK-E
• [Problem RFC] Endpoint-deregistration race on pod termination drops requests during rollouts ("graceful shutdown" 502s) #850 @TahseenSust
• [Problem RFC] HPA scaleTargetRef misconfiguration silently disables autoscaling #849 @W-OK-E
• [Problem RFC] A database connection cap set too low starves a service's connection pool #846 @Neilblaze
• [Problem RFC] Hotel Reservation - service with no running pods still appears healthy #843 @wbth
• [Problem RFC] A regenerated config artifact outgrows the consumer's compiled capacity #842 @Neilblaze
• [Problem RFC] Stuck Finalizer Deadlock Causing Persistent Resource Termination #841 @ermias19 implemented by Fix/finalizer deadlock mitigation #858
• [Problem RFC] Node-label selector drift breaks a critical DaemonSet (Reddit Pi-Day mechanism) #837 @TahseenSust
• [Problem RFC] Incomplete rollout leaves a previous build serving behind a live Service #834 @Neilblaze
• [Problem RFC] PodDisruptionBudget blocks node drain #833 @abdallahade1 implemented by Add PodDisruptionBudget-blocks-node-drain problem #835
• [PROBLEM RFC] Add grpc pick first load balancer pin hotel reservation problem #830 @AshhKetchup
• [Problem RFC] Calico route-reflector selector drift partitions cross-node pod networking #827 @thromel by Add Calico route-reflector label drift problem #828
• [Problem RFC] Node Clock Drift Silently, Breaks TLS While Pods Stay Running #826 @haani-maybe
• [Problem RFC] CPU Throttling Due to CFS Limits #824 @haani-maybe
• [Problem RFC] Worker Crashes with "No Space Left on Device" Despite a Healthy Node Disk #823 @SuMayaBee implemented by Add dev_shm_exhaustion_hotel_reservation problem #829
• [Problem RFC] Stale Secret Environment Variable After Database Credential Rotation #822 @Ipetr14 implemented by Add: secret_rotation_stale_env_credentials_astronomy_shop problem #851
• [Problem RFC] A bad value in a shared ConfigMap takes down several services #819 @NahinNilav
• [Problem RFC] Co-located backup scan evicts OS page cache and slows hotel search #817 @alpblkba
• [Problem RFC] Services Fail to Connect After Node Kernel Parameter Misconfiguration #816 @sukanya1426
• [Problem RFC] HPA Control Loop Fails Due to Missing CPU Requests #812 @MABDz implemented by New Problem | HPA missing effective CPU request  #820
• [Problem RFC] CoreDNS saturation via ndots:5 DNS query amplification #811 @Mr-NOBY
• [Problem RFC] File descriptor exhaustion #806 @TejasShukla2007 implemented by Add file_descriptor_exhaustion problem #832
• [Problem RFC] internalTrafficPolicy: Local silently drops in-cluster traffic on nodes without a local backend pod #803 @sharyash81 implemented by Add internalTrafficPolicy: Local fault simulation for astronomy-shop #805
• [Problem RFC] PriorityClass Preemption Displaces a Critical Service #795 @munimthahmid
• [Problem RFC] Node Routing Table Wipeout #794 @keremakcali implemented by Add node_routing_table_wipeout problem #818
• [Problem RFC] CrashLoopBackOff caused by misconfigured memory limit (OOMKilled) in hotel-reservation #781, @yangjasmine400-cmd
• [Problem RFC] Cloudflare 2019-style WAF regex CPU exhaustion #769, done by Add Cloudflare-Style WAF Regex CPU Saturation #773 @RedwanulKarim612
• [Problem RFC] Expired TLS/SSL Certificate #763, implemented by Add expired_tls_hotel_reservation problem #784 @gAngelov24