First run the setup script in the /app directory to download (if necessary) the required model:

./setup.sh -model <model-name>

For example:

./setup.sh -model EliovpAI/Deepseek-R1-0528-Qwen3-8B-FP8-KV -loc <download-or-cached-location-of-model>

or

./setup.sh -model amd/Llama-3.1-8B-Instruct-FP8-KV <download-or-cached-location-of-model>

! This will depend on which model has been agreed for usage with Paiton.

To serve the AMD/Llama-3.1-8B-Instruct-FP8-KV model, use the following command:

python3 /app/wrap_vllm_entrypoint.py \
    --model /app/Llama-3.1-8B-Instruct-FP8-KV/ \
    --served-model-name amd/Llama-3.1-8B-Instruct-FP8-KV \
    --compilation-config '{cudagraph_mode": 0, "cudagraph_capture_sizes": []}' \
    --kv-cache-dtype fp8

To serve the EliovpAI/Deepseek-R1-0528-Qwen3-8B-FP8-KV model, use the following command:

python3 /app/wrap_vllm_entrypoint.py \
    --model /app/Deepseek-R1-0528-Qwen3-8B-FP8-KV/ \
    --served-model-name EliovpAI/Deepseek-R1-0528-Qwen3-8B-FP8-KV \
    --compilation-config '{cudagraph_mode": 0, "cudagraph_capture_sizes": []}' \
    --kv-cache-dtype fp8

• --model: Path to the model files
• --served-model-name: Name identifier for the served model
• --compilation-config: JSON configuration for CUDA graph optimization
  • use_cudagraph: Disabled for compatibility with Paiton to use VLLMs faster Async Engine
  • cudagraph_capture_sizes: Empty array for no capture sizes, again for compatibility with Paiton
• --max-model-len: Maximum sequence length (4096 tokens)
• --kv-cache-dtype fp8: Use FP8 precision for key-value cache (memory optimization)

• FP8 Key-Value Cache: Reduces memory usage while maintaining accuracy
• Paiton Integration: AMD-optimized inference framework
• vLLM Backend: High-performance serving with efficient request handling
• Custom Entry Point: Integrates Paiton models with vLLM serving

To benchmark the model performance using the ShareGPT dataset:

vllm bench serve --backend vllm --model EliovpAI/Deepseek-R1-0528-Qwen3-8B-FP8-KV --dataset-name sharegpt --dataset-path /app/vllm/benchmarks/ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 1024 --random-range-ratio 1.0 --percentile-metrics ttft,tpot,itl,e2el --sharegpt-output-len 256 --port 8888 --max-concurrency 128 --request-rate=inf

• --backend vllm: Use vLLM as the serving backend
• --model: Model identifier for benchmarking
• --dataset-name sharegpt: Use ShareGPT dataset for testing
• --dataset-path: Path to the ShareGPT dataset file
• --num-prompts 1024: Number of prompts to test
• --random-range-ratio 1.0: Use full range of dataset
• --percentile-metrics: Metrics to measure:
  • ttft: Time to first token
  • tpot: Time per output token
  • itl: Inter-token latency
  • e2el: End-to-end latency
• --sharegpt-output-len 256: Generate 256 tokens per response

The benchmark will provide detailed performance metrics including:

• Throughput: Requests per second
• Latency: Response time measurements
• Memory Usage: GPU memory consumption
• Token Generation Speed: Tokens per second

1. FP8 Key-Value Cache:

   • Reduces memory footprint by ~50% compared to FP16
   • Maintains model quality with minimal accuracy loss
   • Optimized for AMD GPUs

2. Paiton Integration:

   • AMD-optimized inference kernels
   • Efficient memory management
   • Hardware-specific optimizations

3. vLLM Serving:

   • High-throughput request handling
   • Efficient batching and scheduling
   • RESTful API interface

Once the server is running, you can interact with it using the OpenAI-compatible API:

import openai

client = openai.OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="dummy"
)

response = client.chat.completions.create(
    model="amd/Llama-3.1-8B-Instruct-FP8-KV",
    messages=[
        {"role": "user", "content": "Hello, how are you?"}
    ],
    max_tokens=256,
    temperature=0.7
)

print(response.choices[0].message.content)

• FP8 KV cache reduces memory usage significantly
• Efficient memory allocation through Paiton
• Dynamic memory management in vLLM

• Request batching for higher throughput
• Efficient scheduler with configurable steps
• Hardware-specific optimizations for AMD GPUs

• Optimized token generation pipeline
• Efficient attention computation
• Reduced memory bandwidth requirements

1. Out of Memory Errors:

   • Reduce --max-model-len
   • Ensure sufficient GPU memory
   • Check FP8 compatibility

2. Model Loading Issues:

   • Verify model path is correct
   • Check model file integrity
   • Ensure Paiton compatibility

3. Performance Issues:

   • Adjust --num-scheduler-steps
   • Monitor GPU utilization
   • Check compilation configuration

• GPU Memory Usage: Monitor with rocm-smi
• Throughput: Requests per second
• Latency: P50, P95, P99 latencies
• Token Generation Speed: Tokens per second

The server provides health endpoints:

• GET /health: Basic health check
• GET /v1/models: List available models
• GET /metrics: Performance metrics (if enabled)