Skip to content
This repository was archived by the owner on Jun 13, 2025. It is now read-only.
This repository was archived by the owner on Jun 13, 2025. It is now read-only.

[Bug]: Failure to loading lora adapter crashes the whole vllm instance #7

Open
Open
[Bug]: Failure to loading lora adapter crashes the whole vllm instance#7
Labels
bugSomething isn't working

Description

@yx-lamini
yx-lamini
opened on Jan 30, 2025

What

Start vllm:

HF_TOKEN="<FILL_IN_YOUR_HUGGINGFACE_TOKEN>"
LLAMAFACTORY_LORA_DIR="<FILL_IN_LLAMAFACTORY_LORA_OUTPUT>"
docker run --runtime nvidia --gpus all -v ${LLAMAFACTORY_LORA_DIR}:/eda-test/lora-test --env "HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}" --env "VLLM_ALLOW_RUNTIME_LORA_UPDATING=True"  -p 8000:8000 --ipc=host   powerml/inference-engine-nvidia:v0.6.5-000   --model meta-llama/Meta-Llama-3.1-8B-Instruct --dtype=half --enable_lora --max-model-len 192

Then send lora inference request, note /eda-test is incorrect:

curl http://localhost:8000/v1/completions -H "Content-Type: application/json" -d '{
        "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
        "prompt": "San Francisco is a",
        "lora_request": {
            "lora_name": "test_adapter",
            "lora_path": "/eda-test"
        }
    }'

The stack trace from vllm:

INFO:     172.17.0.1:54434 - "POST /v1/completions HTTP/1.1" 500 Internal Server Error
ERROR 01-30 09:48:33 engine.py:135] RuntimeError('Error in model execution (input dumped to /tmp/err_execute_model_input_20250130-094833.pkl): Loading lora /eda-test failed')
ERROR 01-30 09:48:33 engine.py:135] Traceback (most recent call last):
ERROR 01-30 09:48:33 engine.py:135]   File "/usr/local/lib/python3.12/dist-packages/vllm/lora/worker_manager.py", line 95, in _load_adapter
ERROR 01-30 09:48:33 engine.py:135]     lora = self._lora_model_cls.from_local_checkpoint(
ERROR 01-30 09:48:33 engine.py:135]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 01-30 09:48:33 engine.py:135]   File "/usr/local/lib/python3.12/dist-packages/vllm/lora/models.py", line 215, in from_local_checkpoint
ERROR 01-30 09:48:33 engine.py:135]     with open(lora_config_path) as f:
ERROR 01-30 09:48:33 engine.py:135]          ^^^^^^^^^^^^^^^^^^^^^^
ERROR 01-30 09:48:33 engine.py:135] FileNotFoundError: [Errno 2] No such file or directory: '/eda-test/adapter_config.json'
ERROR 01-30 09:48:33 engine.py:135]
ERROR 01-30 09:48:33 engine.py:135] The above exception was the direct cause of the following exception:
ERROR 01-30 09:48:33 engine.py:135]
ERROR 01-30 09:48:33 engine.py:135] Traceback (most recent call last):
ERROR 01-30 09:48:33 engine.py:135]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner_base.py", line 116, in _wrapper
ERROR 01-30 09:48:33 engine.py:135]     return func(*args, **kwargs)
ERROR 01-30 09:48:33 engine.py:135]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 01-30 09:48:33 engine.py:135]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner.py", line 1624, in execute_model
ERROR 01-30 09:48:33 engine.py:135]     self.set_active_loras(model_input.lora_requests,
ERROR 01-30 09:48:33 engine.py:135]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner.py", line 1342, in set_active_loras
ERROR 01-30 09:48:33 engine.py:135]     self.lora_manager.set_active_adapters(lora_requests, lora_mapping)
ERROR 01-30 09:48:33 engine.py:135]   File "/usr/local/lib/python3.12/dist-packages/vllm/lora/worker_manager.py", line 137, in set_active_adapters
ERROR 01-30 09:48:33 engine.py:135]     set_active_adapters_worker(requests, mapping, self._apply_adapters,
ERROR 01-30 09:48:33 engine.py:135]   File "/usr/local/lib/python3.12/dist-packages/vllm/adapter_commons/utils.py", line 52, in set_active_adapters_worker
ERROR 01-30 09:48:33 engine.py:135]     apply_adapters_func(requests)
ERROR 01-30 09:48:33 engine.py:135]   File "/usr/local/lib/python3.12/dist-packages/vllm/lora/worker_manager.py", line 197, in _apply_adapters
ERROR 01-30 09:48:33 engine.py:135]     self.add_adapter(lora)
ERROR 01-30 09:48:33 engine.py:135]   File "/usr/local/lib/python3.12/dist-packages/vllm/lora/worker_manager.py", line 206, in add_adapter
ERROR 01-30 09:48:33 engine.py:135]     lora = self._load_adapter(lora_request)
ERROR 01-30 09:48:33 engine.py:135]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 01-30 09:48:33 engine.py:135]   File "/usr/local/lib/python3.12/dist-packages/vllm/lora/worker_manager.py", line 108, in _load_adapter
ERROR 01-30 09:48:33 engine.py:135]     raise RuntimeError(f"Loading lora {lora_path} failed") from e
ERROR 01-30 09:48:33 engine.py:135] RuntimeError: Loading lora /eda-test failed
ERROR 01-30 09:48:33 engine.py:135]
ERROR 01-30 09:48:33 engine.py:135] The above exception was the direct cause of the following exception:
ERROR 01-30 09:48:33 engine.py:135]
ERROR 01-30 09:48:33 engine.py:135] Traceback (most recent call last):
ERROR 01-30 09:48:33 engine.py:135]   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 133, in start
ERROR 01-30 09:48:33 engine.py:135]     self.run_engine_loop()
ERROR 01-30 09:48:33 engine.py:135]   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 196, in run_engine_loop
ERROR 01-30 09:48:33 engine.py:135]     request_outputs = self.engine_step()
ERROR 01-30 09:48:33 engine.py:135]                       ^^^^^^^^^^^^^^^^^^
ERROR 01-30 09:48:33 engine.py:135]   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 214, in engine_step
ERROR 01-30 09:48:33 engine.py:135]     raise e
ERROR 01-30 09:48:33 engine.py:135]   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 205, in engine_step
ERROR 01-30 09:48:33 engine.py:135]     return self.engine.step()
ERROR 01-30 09:48:33 engine.py:135]            ^^^^^^^^^^^^^^^^^^
ERROR 01-30 09:48:33 engine.py:135]   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/llm_engine.py", line 1405, in step
ERROR 01-30 09:48:33 engine.py:135]     outputs = self.model_executor.execute_model(
ERROR 01-30 09:48:33 engine.py:135]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 01-30 09:48:33 engine.py:135]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/gpu_executor.py", line 88, in execute_model
ERROR 01-30 09:48:33 engine.py:135]     output = self.driver_worker.execute_model(execute_model_req)
ERROR 01-30 09:48:33 engine.py:135]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 01-30 09:48:33 engine.py:135]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 343, in execute_model
ERROR 01-30 09:48:33 engine.py:135]     output = self.model_runner.execute_model(
ERROR 01-30 09:48:33 engine.py:135]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 01-30 09:48:33 engine.py:135]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 01-30 09:48:33 engine.py:135]     return func(*args, **kwargs)
ERROR 01-30 09:48:33 engine.py:135]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 01-30 09:48:33 engine.py:135]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner_base.py", line 152, in _wrapper
ERROR 01-30 09:48:33 engine.py:135]     raise type(err)(
ERROR 01-30 09:48:33 engine.py:135] RuntimeError: Error in model execution (input dumped to /tmp/err_execute_model_input_20250130-094833.pkl): Loading lora /eda-test failed
INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [1]

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions