Dashboard panels fail to load simultaneously #846
Description
Description:
Multiple dashboard panels fail to load at the same time, even though each panel loads correctly when viewed individually. Grafana appears to not send some queries to ClickHouse when rendering the entire dashboard.
The issue seems to be related to how Grafana batches or throttles queries when multiple panels are rendered at once. Each panel works in isolation, suggesting the queries themselves are valid, but the system fails under combined load.
Key observations from logs:
In Grafana UI, panels show: Query execution failed: Unexpected error
Grafana logs:
logger=context userId=1 orgId=1 uname=admin t=2026-01-21T11:06:59.377516695Z level=info msg="Request Completed" method=GET path=/api/datasources/proxy/uid/PDEE91DDB90597936/ status=499 remote_addr=172.22.254.221 time_ms=34 duration=34.613047ms size=0 referer="https://grafana.pgw.internal/d/beueev132i3ggf/health-on-clickhouse?from=now-6h&orgId=1&to=now" handler=/api/datasources/proxy/uid/:uid status_source=downstream
logger=plugin.context t=2026-01-21T11:07:01.366730304Z level=warn msg="Could not create user agent" error="invalid user agent format"
logger=plugin.context t=2026-01-21T11:07:01.388783856Z level=warn msg="Could not create user agent" error="invalid user agent format"
logger=plugin.context t=2026-01-21T11:07:01.414983793Z level=warn msg="Could not create user agent" error="invalid user agent format"
logger=plugin.context t=2026-01-21T11:07:01.455126679Z level=warn msg="Could not create user agent" error="invalid user agent format"
logger=plugin.context t=2026-01-21T11:07:01.477803362Z level=warn msg="Could not create user agent" error="invalid user agent format"
logger=plugin.context t=2026-01-21T11:07:01.498109591Z level=warn msg="Could not create user agent" error="invalid user agent format"
logger=plugin.context t=2026-01-21T11:07:01.523216782Z level=warn msg="Could not create user agent" error="invalid user agent format"
logger=plugin.context t=2026-01-21T11:07:01.546203671Z level=warn msg="Could not create user agent" error="invalid user agent format"
logger=plugin.context t=2026-01-21T11:07:01.568825792Z level=warn msg="Could not create user agent" error="invalid user agent format"
logger=plugin.context t=2026-01-21T11:07:01.591535768Z level=warn msg="Could not create user agent" error="invalid user agent format"
logger=plugin.context t=2026-01-21T11:07:01.609274211Z level=warn msg="Could not create user agent" error="invalid user agent format"
logger=plugin.context t=2026-01-21T11:07:01.630196269Z level=warn msg="Could not create user agent" error="invalid user agent format"
logger=context userId=1 orgId=1 uname=admin t=2026-01-21T11:07:01.67213073Z level=info msg="Request Completed" method=GET path=/api/datasources/proxy/uid/PDEE91DDB90597936/ status=499 remote_addr=172.22.254.221 time_ms=7 duration=7.042584ms size=0 referer="https://grafana.pgw.internal/d/beueev132i3ggf/health-on-clickhouse?from=now-6h&orgId=1&to=now" handler=/api/datasources/proxy/uid/:uid status_source=downstream
logger=authn.service t=2026-01-21T11:07:08.591038583Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2026-01-21T11:07:08.591147943Z level=warn msg=Unauthorized error="user token not found" remote_addr=192.168.242.103 traceID=
logger=context userId=0 orgId=0 uname= t=2026-01-21T11:07:08.591473976Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=192.168.242.103 time_ms=1 duration=1.211845ms size=40 referer= handler=/api/live/ws status_source=server
Environment:
- Grafana version: 10.4.18-security-01
- ClickHouse Grafana plugin version: v3.4.8
- ClickHouse version: 25.3.4.190
Expected behavior:
All panels should load simultaneously when opening the dashboard.