A self-hosted, AI-native event observability platform. Ingest arbitrary JSON events at scale, store them in ClickHouse, and query them via HTTP API or natural language through MCP — all without a predefined schema.

Client
  │
  ▼
POST /v1/ingest/:dataset    ← HTTP Ingest API (Go)
  │
  ▼
Kafka                        ← Write buffer (absorbs spikes)
  │
  ▼
Batch Worker (Go)            ← Consumer → ClickHouse writer
  │
  ▼
ClickHouse                   ← Columnar storage (MergeTree)
  │
  ▼
GET /v1/query/:dataset       ← Query API (Go)
  │
  ▼
MCP Tools (Go)               ← AI agent interface

• Schema-less ingest — Send any JSON, no pre-defined schema required
• Kafka-buffered writes — Absorb traffic spikes, decouple ingest from storage
• ClickHouse storage — Columnar, partitioned, fast time-range aggregations
• Query API — Filter by dataset, time range, pagination with limit/offset
• MCP integration — AI agents (Claude, Cursor, etc.) can query logs in natural language
• Graceful shutdown — Drains Kafka offsets + ClickHouse buffer on SIGINT/SIGTERM

• Go 1.25+
• Docker + Docker Compose

make dev
# or: docker compose -f docker-compose.dev.yml up -d

This starts Kafka (single broker, KRaft mode) and ClickHouse.

go run ./cmd/api/main.go

Override port at runtime:

go run ./cmd/api/main.go -port=9090

go run ./cmd/worker/main.go

curl -s http://localhost:8080/healthz
# {"status":"ok"}

curl -X POST http://localhost:8080/v1/ingest/my-app \
  -H "Content-Type: application/json" \
  -d '[
    {"level": "error", "message": "DB connection failed", "service": "auth"},
    {"level": "info",  "message": "User logged in",       "service": "auth"}
  ]'

Response: 202 Accepted on success, 503 Service Unavailable if Kafka is unreachable.

curl -s "http://localhost:8080/v1/query/my-app?from=2026-01-01T00:00:00Z&limit=10"

Query parameters:

GET /healthz
→ 200 {"status":"ok"}

POST /v1/ingest/:dataset
Body: JSON array of event objects
→ 202 Accepted
→ 503 Service Unavailable (Kafka unavailable)

GET /v1/query/:dataset
Params: from, to, limit, offset
→ 200 JSON array of enriched events

POST /mcp
Body: MCP JSON-RPC message

The /mcp endpoint serves the Model Context Protocol over streamable HTTP transport. Connect any MCP-compatible client (Claude Desktop, Cursor, etc.) to query logs using natural language.

Available tools:

logman/
├── cmd/
│   ├── api/main.go           # HTTP API server entrypoint
│   └── worker/main.go        # Kafka consumer + ClickHouse writer
├── internal/
│   ├── clickhouse/            # ClickHouse writer, batch processor, query
│   ├── config/                # Config loading (koanf, env-based)
│   ├── domain/                # Core domain types (LogEvent, EnrichedEvent)
│   ├── handler/               # HTTP handlers (ingest, query) + MCP tools
│   ├── kafkaclient/           # Kafka consumer + publisher
│   ├── migrations/            # ClickHouse schema migrations
│   └── server/                # Gin HTTP server setup + graceful shutdown
├── assets/
│   ├── architecture.png       # System architecture diagram
│   └── kafka_architecture.png # Kafka write path diagram
├── docker-compose.dev.yml     # Local infrastructure (Kafka + ClickHouse)
├── Makefile                   # Dev workflow shortcuts
└── .env                       # Environment variables

All configuration is loaded from environment variables prefixed with APP_. A .env file at the project root is loaded automatically.

Why Kafka instead of writing directly to ClickHouse? ClickHouse performs best with large batch inserts. Kafka absorbs ingest spikes and lets the worker control insert cadence — prevents write amplification under load.

Why MergeTree and not ReplacingMergeTree? Events are append-only. MergeTree is the fastest and simplest choice. Deduplication would add overhead with no benefit.

Why schema-on-write? Accepting any JSON without a pre-defined schema is the core value prop. Events are stored with their original JSON payload for maximum flexibility.

Why MCP? The Model Context Protocol lets AI agents (Claude, Cursor, GitHub Copilot, etc.) query logs using natural language. This turns LogMan into an AI-native observability tool.