nprof

A profiler for Ray-based distributed inference on heterogeneous devices.

This project relies on the following major dependencies (installed via instructions below:

• Python: 3.13
• Ray >= 2.49.x
• GPU Stack:
  • Torch >= 2.8
  • Transformers >= 4.56.x
  • Accelerate >= 1.10.x
• Monitoring: Grafana >= 10.2.x
• Node.js: 22.20.0 (npm 10.9.3 tested)

First, set up the Python virtual environment and install dependencies.

# 1. Create and activate virtual environment
python -m venv .venv
source .venv/bin/activate

# 2. Install Python dependencies
# Note: Use requirements-rocm.txt instead for AMDGPU.
pip install -r requirements.txt

# 3. Make utility scripts executable
chmod +x scripts/*

Install the frontend dependencies.

cd webui
npm install
cd ..

Launch the Ray cluster with Prometheus attached.

# Launch Ray cluster
scripts/run_ray.sh

# Launch Grafana
scripts/run_grafana.sh

# Check active nodes
ray list nodes

You can view the Ray Dashboard at http://127.0.0.1:8265.

Start the web interface (SvelteKit). Run this command in a separate terminal window.

# Inside /webui directory
cd webui
npm run dev
cd ..

Access the WebUI at http://localhost:5173.

Execute the inference script across the cluster.

python inference/demo.py

To stop the Ray cluster and clean up resources:

scripts/clear_ray.sh

• Qwen3 Repository
• Qwen3-4B-Instruct Model
• Hugging Face Accelerate Profiler Guide
• PyTorch Profiler Recipe
• Ray Documentation
• Ray Serve Tutorials
• Ray Cluster Metrics