/kind feature
Describe the solution you'd like
The controller should not require hard-coded names to match objects exactly in order to run.
If a user does not want to use prefill for example, the LLMInferenceServiceConfig object named "kserve-config-llm-prefill-worker-data-parallel" should not need to exist in order for the controller to run.
Also, the controller should be able to discover these objects dynamically rather than only based on the name of these objects.
Anything else you would like to add:
|
// Configuration template names for different LLM deployment patterns |
|
// These configs are automatically applied based on the service configuration |
|
const ( |
|
configPrefix = "kserve-" |
|
// Single node deployment template |
|
configTemplateName = configPrefix + "config-llm-template" |
|
// Disaggregated prefill/decode templates |
|
configDecodeTemplateName = configPrefix + "config-llm-decode-template" |
|
configPrefillTemplateName = configPrefix + "config-llm-prefill-template" |
|
// Pipeline parallel worker configurations |
|
configDecodeWorkerPipelineParallelName = configPrefix + "config-llm-decode-worker-pipeline-parallel" |
|
configWorkerPipelineParallelName = configPrefix + "config-llm-worker-pipeline-parallel" |
|
configPrefillWorkerPipelineParallelName = configPrefix + "config-llm-prefill-worker-pipeline-parallel" |
|
// Data parallel worker configurations |
|
configWorkerDataParallelName = configPrefix + "config-llm-worker-data-parallel" |
|
configDecodeWorkerDataParallelName = configPrefix + "config-llm-decode-worker-data-parallel" |
|
configPrefillWorkerDataParallelName = configPrefix + "config-llm-prefill-worker-data-parallel" |
|
// Router and scheduler configurations |
|
configRouterSchedulerName = configPrefix + "config-llm-scheduler" |
|
configRouterRouteName = configPrefix + "config-llm-router-route" |
|
) |
Links to the design documents:
[Optional, start with the short-form RFC template to outline your ideas and get early feedback.]
[Required, use the longer-form design doc template to specify and discuss your design in more detail]