[Feature] Use Moka's Time‑To‑Idle (TTI) instead of the eviction thread in the gateway #15248
Description
Checklist
- If this is not a feature request but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed.
- Please use English. Otherwise, it will be closed.
Motivation
If I understand correctly, the current implementation clears all KV‑cache entries after a configured interval.
I think we could leverage Moka's TTI for a more precise eviction policy. For example, with a TTI of five minutes, any KV‑cache entry that hasn’t been accessed (via get or insert) within that period will be automatically removed. This aligns better with what we intend to do. See the example: https://docs.rs/moka/0.12.11/moka/future/struct.Cache.html#example-time-based-expirations
What do you think?
Related resources
No response