Skip to content

Support OTEL Metrics, stabalize service map render (stream based), some other minor fixes#7

Open
VertigoOne1 wants to merge 11 commits into
metafab:mainfrom
VertigoOne1:main
Open

Support OTEL Metrics, stabalize service map render (stream based), some other minor fixes#7
VertigoOne1 wants to merge 11 commits into
metafab:mainfrom
VertigoOne1:main

Conversation

@VertigoOne1

Copy link
Copy Markdown
Contributor

Hi

I kindly offer this to add support for OTEL Metrics (nothing fancy), a stream based service map render pipeline so it stops flashing and throwing 160 requests a second, and some other minor fixes.

image

i did not wire the "premium" features, phase 3. Otherwise, excellent work btw, nice stack and only uses like 250Mb RAM with a fairly heavy flow.

ciao

VertigoOne1 and others added 4 commits June 25, 2026 09:37
Logs: replace the full-list refetch with SSE snapshot/append delta
streaming so the logs table no longer flashes/remounts on every update
(loading state only on first load). Vite dev allows host.docker.internal
so containerised OTLP exporters aren't 403'd.

Metrics (OTLP, Phases 1-2): implement /v1/metrics ingest (protobuf + JSON,
gzip), an in-memory store with per-series point rings and a seq-cursor SSE
delta protocol, and a Metrics tab:
- Gauge & Sum line charts (uPlot), live-appending and flicker-free
- Histogram & exponential-histogram distribution bars + time heatmap
- Summary quantile lines
- Server-side rate computation with counter-reset detection (raw/rate toggle)
- Series filtering with a top-N cap
- OTEL_GUI_MAX_METRICS / OTEL_GUI_MAX_METRIC_POINTS retention knobs
- docs/metrics/ brief + phase 1-3 plans

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the Service Map's HTTP polling (~133 req/s of full rebuilds) with the
SSE snapshot contract used by logs/metrics, and fix the layout reflow that made
the graph visibly "flash" whenever the topology changed.

Server:
- packages/core: persistent, never-pruned cumulative aggregate
  (accumulateSpan / projectServiceMap) that resolves cross-service edges
  out-of-order (SERVER parents arrive after children), dedups per span, and
  windows edge latency while keeping call/error counts cumulative
- traceStore: aggregate folded in on ingest, serviceMapSeq cursor + memoised
  projection; reset only on clearTraces; rebuilt on replaceAllTraces
- new /api/service-map/stream SSE endpoint (map-count / map-snapshot)

Client:
- serviceMap.svelte.ts store (snapshot-on-connect, first-load-only loading,
  REST fallback)
- createStickyLayout: persistent per-node placements so placed nodes never
  move; new nodes append and the canvas grows (expand-only)
- +page.svelte: drop fetchServiceMap + 2s interval; keep <ServiceMap> mounted

Validated live: 0 polls (was 798/6s), existing nodes never reposition, counts
update in place, new services appear without moving incumbents, no flash.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ADME

- broaden the intro to traces, logs, and metrics
- describe the snapshot+delta SSE streaming shared by all live views, the
  flicker-free render fixes, and the cumulative expand-only service map
- add a Metrics feature bullet, "Sending Metrics" section, and screenshot
- document OTEL_GUI_MAX_METRICS / OTEL_GUI_MAX_METRIC_POINTS and the new
  metrics + *_/stream API endpoints
- add the `c` (Metrics tab) shortcut and uPlot to the tech stack

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Browser-based OTLP exporters issue cross-origin requests to /v1/* and
/api/*. Anything beyond a simple request (e.g. application/x-protobuf or
custom headers) triggers a preflight OPTIONS that SvelteKit did not
answer, and every response was missing Access-Control-Allow-Origin, so
the browser discarded it.

Add a server hook that answers CORS preflights and attaches CORS headers
to OTLP ingest and read API responses, with an OTEL_GUI_CORS_ORIGIN knob
(default "*", or a comma-separated allow-list). Policy logic lives in a
pure, unit-tested module.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@metafab

metafab commented Jun 30, 2026

Copy link
Copy Markdown
Owner

Hi @VertigoOne1, I'm glad you enjoy the tool!
Thank you for the feedback on the memory consumption and for this PR. I plan to review it before the end of the week.

VertigoOne1 and others added 5 commits July 2, 2026 11:08
Opening a trace was extremely slow and eventually got stuck. Two causes:

1. Row clicks used window.location.href — a full-page reload that tore
   down and rebuilt the whole app (and every SSE stream) on each open.
   Switched Traces/Logs/Metrics row navigation to client-side goto().

2. The list page held four separate SSE streams (traces/logs/metrics/
   service-map) open permanently, consuming 5 of the browser's ~6
   connections-per-origin, so reloads stalled behind sockets that were
   slow to release. Multiplexed all four into a single /api/stream
   endpoint consumed via a shared sseClient singleton; steady-state
   connections drop from 5 to 2 (measured). Removed the four old
   per-stream endpoints. Logs/Metrics tabs mount after the connect-time
   snapshot, so they now seed the initial list via REST (guarded so an
   in-flight seed can't clobber a concurrent clear/delete snapshot).

Also adds navigation/accessibility hooks: data-testid + data-*-id on
list rows, keyboard-activatable rows, tab/tabpanel ARIA wiring, and
status/alert roles on loading and error states.

Tests: sseClient unit tests; /api/stream connect + delta + clear tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Feat/cors support - single sse stream
Deep, in-memory search across traces, logs, and metrics — matching message
bodies, span names, status, events, and every attribute — exposed three ways:

- Store: searchLogs/searchTraces/searchMetrics/searchAll + listServices on the
  in-memory trace store, plus reusable match primitives in @otel-gui/core.
- MCP: localhost-only JSON-RPC endpoint at POST /mcp (search, search_*, get_*,
  list_services, get_service_map, and a write-only flush tool). Loopback-guarded
  by default; OTEL_GUI_MCP_ALLOW_REMOTE=1 relaxes it for Docker/proxy.
- UI: GET /api/search backs the per-tab search boxes, which now query the server
  deep-search (with a client-side substring fallback) and show matchedIn hints.

Add POST /api/flush + a global Flush button + MCP flush tool to clear all
buffers (traces, logs, metrics, service map) for clean-state captures.

Document all missing env vars in .env.example (PORT, MAX_METRICS,
MAX_METRIC_POINTS, MCP_ENABLED, MCP_ALLOW_REMOTE, license PEM/clock-skew).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@metafab

metafab commented Jul 9, 2026

Copy link
Copy Markdown
Owner

@VertigoOne1 This PR is becoming large and mixes multiple subjects.
Could you please split it into multiple smaller PRs (one for metrics, one for the service map, one for fixes...)?

VertigoOne1 and others added 2 commits July 13, 2026 11:07
The multiplexed /api/stream and per-metric SSE endpoints enqueued encoded
snapshots (encoder.encode -> controller.enqueue) without ever checking
controller.desiredSize. A web ReadableStream's enqueue never blocks, so when
a consumer drained slower than ingestion produced (the traces sub-stream
re-sends the full list on every change), the encoded Uint8Arrays piled up
unbounded in the stream's internal queue. Off-heap arrayBuffers grew ~1.4MB/s
with the datastore static at cap, eventually OOM-killing the container (heapUsed
stayed flat while external/arrayBuffers climbed to 2GB+).

Flush only when desiredSize > 0; otherwise mark dirty and let the stream's
pull() callback re-flush the moment the consumer catches up. This caps the
outbound queue at ~one flush without dropping updates -- pending deltas wait
in the bounded store, not the socket queue. Full snapshots (traces/map/metric)
are re-read fresh on the next flush.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Brings in upstream metafab/main (8899c62): prefer log.record.uid as the
stored log ID when present, and URL-encode dynamic IDs to keep routes safe
when IDs contain reserved characters (/, ?, :).

Conflicts (Logs.svelte, Traces.svelte): our fork navigates row clicks via
client-side goto() to avoid tearing down every SSE stream on each open
(the 6-connection-per-origin stall fix); upstream switched to
window.location.href but added encodeURIComponent(). Resolved by keeping
client-side goto() AND wrapping the id in encodeURIComponent() — both the
no-reload behaviour and the URL-safety fix are preserved. Row keydown
handlers (ours) retained.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@VertigoOne1

Copy link
Copy Markdown
Contributor Author

@VertigoOne1 This PR is becoming large and mixes multiple subjects. Could you please split it into multiple smaller PRs (one for metrics, one for the service map, one for fixes...)?

I'll see what i can do! There was a lot of plumbing changes to improve memory consumption, the last fix was to correct a memory leak on the SSE side that would eventually crash out docker-engine.

@VertigoOne1

Copy link
Copy Markdown
Contributor Author

Agreed — done. I've split this into a stack of smaller PRs, rebuilt cleanly from main (the history here had later commits rewriting earlier ones, so it's partitioned by feature rather than cherry-picked):

  1. feat: configurable CORS support for OTLP ingest and read API #10CORS support (standalone, ~250 lines)
  2. fix: unify SSE streaming on one connection + incremental log deltas #11Unified SSE stream + navigation fix (standalone; the streaming infrastructure the rest builds on — includes the backpressure/off-heap memory fix)
  3. feat/otel-metricsOTLP metrics support (stacked on fix: unify SSE streaming on one connection + incremental log deltas #11; ~7k lines, the bulk of it metrics UI + tests). PR to follow once fix: unify SSE streaming on one connection + incremental log deltas #11 lands.
  4. feat/service-map-sseService map over SSE + memory fixes (stacked on 3). PR to follow.

The MCP endpoint / deep search / flush work also in this branch will come as a final separate PR after the others, since it touches all three signals.

Each branch passes test, check, and lint independently. I'll keep this PR open until the replacements are all up, then close it.

@metafab2

Copy link
Copy Markdown
Collaborator

Thanks a lot. I'm currently reviewing these PRs :)

@metafab2

Copy link
Copy Markdown
Collaborator

@VertigoOne1 #10 and #11 have been merged. You can prepare the next one 😉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants