Steps to reproduce
Create a service (the same configuration as in #4240 issue):
type: service
name: qwen38-27b-agentic-smg
groups:
- replicas: 1
python: "3.12"
commands:
- pip install smg
- smg launch --host 0.0.0.0 --port 8000 --enable-igw --policy cache_aware --model-path Qwen/Qwen3.8-27B
router:
type: sglang
resources:
gpu: 0
- replicas: 1
image: vllm/vllm-openai:qwen38
commands:
- pip install smg-grpc-servicer
- python3 -m vllm.entrypoints.grpc_server --host 0.0.0.0 --port 8000 --model Qwen/Qwen3.8-27B-FP8
resources:
gpu: H100:80GB
port: 8000
model:
type: chat
name: Qwen/Qwen3.8-27B
format: openai
prefix: /v1
Actual behaviour
There are two errors in the server logs:
Parent instance <RunModel at 0x78af662b9e40> is not bound to a Session; deferred load operation of attribute 'run_name' cannot proceed – occurs once
Could not fetch server_info for worker http://10.0.0.3:8000: ReadError('') – repeats until the worker is ready
Expected behaviour
No response
dstack version
0.22.1
Server logs
DEBUG dstack._internal.server.background.pipeline_tasks.base:357 Processing service_router_worker_sync item
648bb027-2a5c-447e-80c7-a8e0b391bc37
ERROR dstack._internal.server.background.pipeline_tasks.base:361 Unexpected exception when processing item
Traceback (most recent call last):
File "/home/def/dev/dstack/src/dstack/_internal/server/background/pipeline_tasks/base.py", line 359, in start
await self.process(item)
File "/home/def/dev/dstack/src/dstack/_internal/server/utils/tracing.py", line 26, in wrapper
result = await f(*args, **kwargs)
File "/home/def/dev/dstack/src/dstack/_internal/server/background/pipeline_tasks/service_router_worker_sync.py", line
269, in process
await sync_router_workers_for_run_model(run_for_sync)
File "/home/def/dev/dstack/src/dstack/_internal/server/services/runs/router_worker_sync.py", line 643, in
sync_router_workers_for_run_model
fmt(run_model),
File "/home/def/dev/dstack/src/dstack/_internal/server/services/logging.py", line 26, in fmt
return fmt_entity("run", model.id, model.run_name)
File "/home/def/dev/dstack/.venv/lib/python3.10/site-packages/sqlalchemy/orm/attributes.py", line 569, in __get__
return self.impl.get(state, dict_) # type: ignore
File "/home/def/dev/dstack/.venv/lib/python3.10/site-packages/sqlalchemy/orm/attributes.py", line 1096, in get
value = self._fire_loader_callables(state, key, passive)
File "/home/def/dev/dstack/.venv/lib/python3.10/site-packages/sqlalchemy/orm/attributes.py", line 1129, in
_fire_loader_callables
return callable_(state, passive)
File "/home/def/dev/dstack/.venv/lib/python3.10/site-packages/sqlalchemy/orm/strategies.py", line 585, in __call__
return strategy._load_for_state(state, passive)
File "/home/def/dev/dstack/.venv/lib/python3.10/site-packages/sqlalchemy/orm/strategies.py", line 543, in
_load_for_state
raise orm_exc.DetachedInstanceError(
sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <RunModel at 0x78af662b9e40> is not bound to a Session; deferred
load operation of attribute 'run_name' cannot proceed (Background on this error at: https://sqlalche.me/e/20/bhk3)
DEBUG dstack._internal.server.background.pipeline_tasks.base:364 Processed service_router_worker_sync item
648bb027-2a5c-447e-80c7-a8e0b391bc37 in 0.022
----------------------------------
ERROR dstack._internal.server.services.runs.router_worker_sync:421 Could not fetch server_info for worker http://10.0.0.3:8000:
ReadError('')
Traceback (most recent call last):
File "/home/def/dev/dstack/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 101, in
map_httpcore_exceptions
yield
File "/home/def/dev/dstack/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 394, in
handle_async_request
resp = await self._pool.handle_async_request(req)
File "/home/def/dev/dstack/.venv/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 256, in
handle_async_request
raise exc from None
File "/home/def/dev/dstack/.venv/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 236, in
handle_async_request
response = await connection.handle_async_request(
File "/home/def/dev/dstack/.venv/lib/python3.10/site-packages/httpcore/_async/connection.py", line 103, in
handle_async_request
return await self._connection.handle_async_request(request)
File "/home/def/dev/dstack/.venv/lib/python3.10/site-packages/httpcore/_async/http11.py", line 136, in
handle_async_request
raise exc
File "/home/def/dev/dstack/.venv/lib/python3.10/site-packages/httpcore/_async/http11.py", line 106, in
handle_async_request
) = await self._receive_response_headers(**kwargs)
File "/home/def/dev/dstack/.venv/lib/python3.10/site-packages/httpcore/_async/http11.py", line 177, in
_receive_response_headers
event = await self._receive_event(timeout=timeout)
File "/home/def/dev/dstack/.venv/lib/python3.10/site-packages/httpcore/_async/http11.py", line 217, in _receive_event
data = await self._network_stream.read(
File "/home/def/dev/dstack/.venv/lib/python3.10/site-packages/httpcore/_backends/anyio.py", line 32, in read
with map_exceptions(exc_map):
File "/home/def/.local/share/uv/python/cpython-3.10-linux-x86_64-gnu/lib/python3.10/contextlib.py", line 153, in
__exit__
self.gen.throw(typ, value, traceback)
File "/home/def/dev/dstack/.venv/lib/python3.10/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
raise to_exc(exc) from exc
httpcore.ReadError
Additional information
No response
Steps to reproduce
Create a service (the same configuration as in #4240 issue):
Actual behaviour
There are two errors in the server logs:
Parent instance <RunModel at 0x78af662b9e40> is not bound to a Session; deferred load operation of attribute 'run_name' cannot proceed– occurs onceCould not fetch server_info for worker http://10.0.0.3:8000: ReadError('')– repeats until the worker is readyExpected behaviour
No response
dstack version
0.22.1
Server logs
DEBUG dstack._internal.server.background.pipeline_tasks.base:357 Processing service_router_worker_sync item 648bb027-2a5c-447e-80c7-a8e0b391bc37 ERROR dstack._internal.server.background.pipeline_tasks.base:361 Unexpected exception when processing item Traceback (most recent call last): File "/home/def/dev/dstack/src/dstack/_internal/server/background/pipeline_tasks/base.py", line 359, in start await self.process(item) File "/home/def/dev/dstack/src/dstack/_internal/server/utils/tracing.py", line 26, in wrapper result = await f(*args, **kwargs) File "/home/def/dev/dstack/src/dstack/_internal/server/background/pipeline_tasks/service_router_worker_sync.py", line 269, in process await sync_router_workers_for_run_model(run_for_sync) File "/home/def/dev/dstack/src/dstack/_internal/server/services/runs/router_worker_sync.py", line 643, in sync_router_workers_for_run_model fmt(run_model), File "/home/def/dev/dstack/src/dstack/_internal/server/services/logging.py", line 26, in fmt return fmt_entity("run", model.id, model.run_name) File "/home/def/dev/dstack/.venv/lib/python3.10/site-packages/sqlalchemy/orm/attributes.py", line 569, in __get__ return self.impl.get(state, dict_) # type: ignore File "/home/def/dev/dstack/.venv/lib/python3.10/site-packages/sqlalchemy/orm/attributes.py", line 1096, in get value = self._fire_loader_callables(state, key, passive) File "/home/def/dev/dstack/.venv/lib/python3.10/site-packages/sqlalchemy/orm/attributes.py", line 1129, in _fire_loader_callables return callable_(state, passive) File "/home/def/dev/dstack/.venv/lib/python3.10/site-packages/sqlalchemy/orm/strategies.py", line 585, in __call__ return strategy._load_for_state(state, passive) File "/home/def/dev/dstack/.venv/lib/python3.10/site-packages/sqlalchemy/orm/strategies.py", line 543, in _load_for_state raise orm_exc.DetachedInstanceError( sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <RunModel at 0x78af662b9e40> is not bound to a Session; deferred load operation of attribute 'run_name' cannot proceed (Background on this error at: https://sqlalche.me/e/20/bhk3) DEBUG dstack._internal.server.background.pipeline_tasks.base:364 Processed service_router_worker_sync item 648bb027-2a5c-447e-80c7-a8e0b391bc37 in 0.022 ---------------------------------- ERROR dstack._internal.server.services.runs.router_worker_sync:421 Could not fetch server_info for worker http://10.0.0.3:8000: ReadError('') Traceback (most recent call last): File "/home/def/dev/dstack/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions yield File "/home/def/dev/dstack/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 394, in handle_async_request resp = await self._pool.handle_async_request(req) File "/home/def/dev/dstack/.venv/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 256, in handle_async_request raise exc from None File "/home/def/dev/dstack/.venv/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 236, in handle_async_request response = await connection.handle_async_request( File "/home/def/dev/dstack/.venv/lib/python3.10/site-packages/httpcore/_async/connection.py", line 103, in handle_async_request return await self._connection.handle_async_request(request) File "/home/def/dev/dstack/.venv/lib/python3.10/site-packages/httpcore/_async/http11.py", line 136, in handle_async_request raise exc File "/home/def/dev/dstack/.venv/lib/python3.10/site-packages/httpcore/_async/http11.py", line 106, in handle_async_request ) = await self._receive_response_headers(**kwargs) File "/home/def/dev/dstack/.venv/lib/python3.10/site-packages/httpcore/_async/http11.py", line 177, in _receive_response_headers event = await self._receive_event(timeout=timeout) File "/home/def/dev/dstack/.venv/lib/python3.10/site-packages/httpcore/_async/http11.py", line 217, in _receive_event data = await self._network_stream.read( File "/home/def/dev/dstack/.venv/lib/python3.10/site-packages/httpcore/_backends/anyio.py", line 32, in read with map_exceptions(exc_map): File "/home/def/.local/share/uv/python/cpython-3.10-linux-x86_64-gnu/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/def/dev/dstack/.venv/lib/python3.10/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions raise to_exc(exc) from exc httpcore.ReadErrorAdditional information
No response