Skip to content

features

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History
 
 

features

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
quantization
quantization
 
 
speculative_decoding
speculative_decoding
 
 
README.md
README.md
 
 
automatic_prefix_caching.md
automatic_prefix_caching.md
 
 
batch_invariance.md
batch_invariance.md
 
 
custom_arguments.md
custom_arguments.md
 
 
custom_logitsprocs.md
custom_logitsprocs.md
 
 
disagg_encoder.md
disagg_encoder.md
 
 
disagg_prefill.md
disagg_prefill.md
 
 
interleaved_thinking.md
interleaved_thinking.md
 
 
lora.md
lora.md
 
 
mooncake_connector_usage.md
mooncake_connector_usage.md
 
 
multimodal_inputs.md
multimodal_inputs.md
 
 
nixl_connector_compatibility.md
nixl_connector_compatibility.md
 
 
nixl_connector_usage.md
nixl_connector_usage.md
 
 
prompt_embeds.md
prompt_embeds.md
 
 
reasoning_outputs.md
reasoning_outputs.md
 
 
sleep_mode.md
sleep_mode.md
 
 
structured_outputs.md
structured_outputs.md
 
 
tool_calling.md
tool_calling.md
 
 

README.md

Features

Compatibility Matrix

The tables below show mutually exclusive features and the support on some hardware.

The symbols used have the following meanings:

  • ✅ = Full compatibility
  • 🟠 = Partial compatibility
  • ❌ = No compatibility
  • ❔ = Unknown or TBD

!!! note Check the ❌ or 🟠 with links to see tracking issue for unsupported feature/hardware combination.

Feature x Feature

<style> td:not(:first-child) { text-align: center !important; } td { padding: 0.5rem !important; white-space: nowrap; } th { padding: 0.5rem !important; min-width: 0 !important; } th:not(:first-child) { writing-mode: vertical-lr; transform: rotate(180deg) } </style>
Feature CP APC LoRA SD CUDA graph pooling enc-dec logP prmpt logP async output multi-step mm best-of beam-search prompt-embeds
CP ✅
APC ✅ ✅
LoRA ✅ ✅ ✅
SD ✅ ✅ ❌ ✅
CUDA graph ✅ ✅ ✅ ✅ ✅
pooling 🟠* 🟠* ✅ ❌ ✅ ✅
enc-dec ❌ ❌ ❌ ❌ ✅ ✅ ✅
logP ✅ ✅ ✅ ✅ ✅ ❌ ✅ ✅
prmpt logP ✅ ✅ ✅ ✅ ✅ ❌ ✅ ✅ ✅
async output ✅ ✅ ✅ ❌ ✅ ❌ ❌ ✅ ✅ ✅
multi-step ❌ ✅ ❌ ❌ ✅ ❌ ❌ ✅ ✅ ✅ ✅
mm ✅ ✅ 🟠^ ❔ ✅ ✅ ✅ ✅ ✅ ✅ ❔ ✅
best-of ✅ ✅ ✅ ❌ ✅ ❌ ✅ ✅ ✅ ❔ ❌ ✅ ✅
beam-search ✅ ✅ ✅ ❌ ✅ ❌ ✅ ✅ ✅ ❔ ❌ ❔ ✅ ✅
prompt-embeds ✅ ✅ ✅ ❌ ✅ ❌ ❌ ✅ ❌ ❔ ❔ ❌ ❔ ❔ ✅

* Chunked prefill and prefix caching are only applicable to last-token or all pooling with causal attention.
^ LoRA is only applicable to the language backbone of multimodal models.

Feature x Hardware

Feature Volta Turing Ampere Ada Hopper CPU AMD Intel GPU
CP ❌ ✅ ✅ ✅ ✅ ✅ ✅ ✅
APC ❌ ✅ ✅ ✅ ✅ ✅ ✅ ✅
LoRA ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅
SD ✅ ✅ ✅ ✅ ✅ ❌ ✅ ✅
CUDA graph ✅ ✅ ✅ ✅ ✅ ❌ ✅ ❌
pooling ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅
enc-dec ✅ ✅ ✅ ✅ ✅ ✅ ❌ ✅
mm ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅
prompt-embeds ✅ ✅ ✅ ✅ ✅ ✅ ❔ ✅
logP ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅
prmpt logP ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅
async output ✅ ✅ ✅ ✅ ✅ ❌ ❌ ✅
multi-step ✅ ✅ ✅ ✅ ✅ ❌ ✅ ✅
best-of ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅
beam-search ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅

!!! note For information on feature support on Google TPU, please refer to the TPU-Inference Recommended Models and Features documentation.