DeepSeek V4 Flash seems to currently be the 192GB VRAM king (though I never managed to test MiMo V2.5 due to needing to limit context to 90K and in theory 200K being possible).

However it's currently stuck in endless churn in vLLM:

• [New Model][Nvidia] Add SM12x support for DeepSeek V4 Flash with essential fixes vllm-project/vllm#41834
• [Roadmap] DeepSeek V4 vllm-project/vllm#40902 (no plan for GPUs before Hopper)
• Enable DeepSeek V4 and GLM-5.1 on SM120 vllm-project/vllm#43477

Kernels and forks all over the place:

• https://github.com/aidendle94/vllm/tree/DSV4
• https://github.com/aidendle94/flashinfer

And DeepSeek doesn't plan to support SM120:

• https://github.com/deepseek-ai/DeepGEMM
• https://github.com/deepseek-ai/FlashMLA
• https://github.com/deepseek-ai/TileKernels

And FlashAttention has no MLA kernel and no plan soon:

• Support for NVIDIA Blackwell (sm_120) architectures — any planned release date? Dao-AILab/flash-attention#1987

This requires:

• Mixed precision Fp8/MXFP4 support
• MLA kernel
• DSA kernel (DeepSeek Attention)
• NSA kernel / Lightning Indexer
• mHC kernel (manifold constrained Hyper-Connections)