Kippu

Kippu (記譜 — "to write down a record") is a small Go service that stores Grok-style parsing templates in Redis and executes them on demand. Send it a raw log line; get back a typed map[string]any of fields.

curl -s localhost:8080/v1/execute -H 'content-type: application/json' -d '{
  "pattern": "%{IP:client.ip}:%{NUMBER:client.port:int}",
  "input":   "10.0.0.1:8080",
  "named_captures_only": true,
  "typed":   true
}'
# {"matched":true,"fields":{"client.ip":"10.0.0.1","client.port":8080}}

• Why Kippu?
• Features
• Quick start
• Configuration
• API reference
• Grok primer
• Architecture
• Development
• Deployment
• Roadmap
• Contributing
• License
• Acknowledgements

Grok is a fantastic way to turn unstructured text into structured fields, but every project that uses it ends up with the same scattered concerns: where do you keep the patterns, how do you share them across services, and how do you let non-Logstash callers run them?

Kippu solves the small, sharp version of that problem:

• A single source of truth for Grok templates, addressable by stable IDs.
• A language-agnostic JSON API, so any service can extract fields without bundling a Grok library.
• A stateless /v1/execute for quick iteration, plus stored templates for production use.
• Fast cold-start, single binary, no database to operate beyond Redis.

It is intentionally not a logging pipeline, a SIEM, or a query engine. It is the smallest useful thing in front of Grok.

• Stored templates with full CRUD and consistent JSON errors.
• Custom and nested pattern definitions on a per-template basis.
• Typed parsing — pattern hints like %{NUMBER:port:int} return native types.
• Ad-hoc /v1/execute endpoint for testing patterns without persisting them.
• Health endpoint that verifies Redis connectivity.
• Container-first: distroless multi-stage image, non-root runtime, sub-20 MB.
• Configured entirely through environment variables.
• GitHub Actions CI with tests, race detector, lint, and GHCR image publishing.

git clone https://github.com/darkfadr/kippu.git
cd kippu
docker compose up --build

In another terminal, create a template and run it:

# 1) Create
curl -s localhost:8080/v1/templates \
  -H 'content-type: application/json' \
  -d '{
    "name": "nginx-host-port",
    "pattern": "%{IP:client.ip}:%{NUMBER:client.port:int}",
    "description": "Match an IPv4 host:port pair from an nginx upstream log",
    "named_captures_only": true
  }' | tee /tmp/kippu-template.json

# 2) Execute against an input string
ID=$(jq -r .id /tmp/kippu-template.json)
curl -s "localhost:8080/v1/templates/$ID/execute" \
  -H 'content-type: application/json' \
  -d '{"input":"10.0.0.1:8080","typed":true}'
# {"matched":true,"fields":{"client.ip":"10.0.0.1","client.port":8080}}

You will need Go 1.24+ and a Redis instance.

# Start Redis (any way you like)
docker run --rm -p 6379:6379 redis:7-alpine

# Run the API
export REDIS_URL=redis://localhost:6379/0
go run ./cmd/api

All configuration is supplied via environment variables. The same values live in .env.example.

• APP_ENV — development (default) or production. Controls log format (text vs. JSON).
• LOG_LEVEL — debug, info (default), warn, error.
• HTTP_ADDR — listen address, default :8080.
• HTTP_READ_TIMEOUT — request read timeout, default 10s.
• HTTP_WRITE_TIMEOUT — response write timeout, default 15s.
• HTTP_IDLE_TIMEOUT — keep-alive idle timeout, default 60s.
• HTTP_MAX_BODY_BYTES — max request body size, default 1048576 (1 MiB).
• SHUTDOWN_TIMEOUT — graceful shutdown budget, default 15s.
• REDIS_URL — Redis connection string, default redis://localhost:6379/0. Use rediss:// for TLS or unix:///path/to/socket for a unix socket. The URL carries username, password, host, port, and DB number, so it works with managed Redis providers (Upstash, Redis Cloud, Render, Fly, Heroku, …) out of the box.
• REDIS_KEY_PREFIX — prefix for all keys, default kippu.

All endpoints accept and return JSON. Errors share a single envelope:

{
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "name is required",
    "details": { "field": "name" }
  }
}

Returns service status and Redis reachability. Returns 503 if Redis is unreachable.

{
  "status": "ok",
  "checks": { "redis": "ok" },
  "service": "kippu",
  "time": "2026-05-12T06:55:12.123456Z"
}

Create a template.

{
  "name": "nginx-host-port",
  "pattern": "%{IP:client.ip}:%{NUMBER:client.port:int}",
  "description": "Optional, max 2048 chars",
  "patterns": { "FOO": "bar" },
  "named_captures_only": true
}

Response 201:

{
  "id": "1b27e3f5-…",
  "name": "nginx-host-port",
  "pattern": "%{IP:client.ip}:%{NUMBER:client.port:int}",
  "description": "Optional, max 2048 chars",
  "patterns": { "FOO": "bar" },
  "named_captures_only": true,
  "created_at": "2026-05-12T06:55:12.123456Z",
  "updated_at": "2026-05-12T06:55:12.123456Z"
}

List templates ordered by creation time (newest first).

Query params: page (default 1), page_size (default 20, max 100).

{
  "data": [ /* templates */ ],
  "pagination": {
    "page": 1,
    "page_size": 20,
    "total_items": 142,
    "total_pages": 8
  }
}

Fetch a single template.

Partially update a template. Send only the fields you want to change:

{ "pattern": "%{IPORHOST:host}", "named_captures_only": true }

Delete a template. Idempotent — returns 204 No Content whether or not the template existed.

Execute a stored template against an input string.

{ "input": "10.0.0.1:8080", "typed": true }

Response:

{
  "matched": true,
  "fields": { "client.ip": "10.0.0.1", "client.port": 8080 }
}

Stateless execution — handy for trying patterns without persisting them.

{
  "pattern": "%{IP:ip}",
  "input": "192.168.1.1 - hello",
  "named_captures_only": true,
  "typed": false
}

Grok patterns are named regex fragments combined with %{PATTERN_NAME:capture_name[:type]} interpolations. For example:

%{IP:client.ip} - %{USER:user} \[%{HTTPDATE:ts}\] "%{WORD:method} %{URIPATHPARAM:path} HTTP/%{NUMBER:http.version}" %{NUMBER:status:int} %{NUMBER:bytes:int}

Kippu uses elastic/go-grok, which ships a comprehensive default pattern set (IPs, dates, syslog, URIs, paths…). You can extend that set per template via the patterns field:

{
  "name": "nginx-host",
  "pattern": "%{NGINX_HOST}",
  "patterns": {
    "NGINX_HOST":         "(?:%{IP:destination.ip}|%{NGINX_NOTSEPARATOR:destination.domain})(:%{NUMBER:destination.port:int})?",
    "NGINX_NOTSEPARATOR": "\"[^\\t ,:]+\""
  },
  "named_captures_only": true
}

When typed: true is set on execute, capture suffixes like :int and :float convert values to native types in the response. With named_captures_only: true, only patterns with an explicit capture name appear in the output.

flowchart TD
  Client["API Client"] --> HTTP["Go HTTP Server (chi)"]
  HTTP --> Handlers["Template Handlers"]
  Handlers --> Service["Template Service"]
  Service --> RedisStore["Redis Store"]
  Service --> GrokEngine["elastic/go-grok"]
  RedisStore --> Redis["Redis"]

Redis key layout (under REDIS_KEY_PREFIX, default kippu):

• kippu:template:<id> — JSON-encoded template.
• kippu:template:name:<lc-name> — name → id uniqueness index.
• kippu:templates — sorted set, score = created_at µs since epoch.

Source layout:

cmd/api/                  process entrypoint, dependency wiring, graceful shutdown
internal/config/          environment parsing and validation
internal/apierror/        consistent JSON error envelope helpers
internal/httpapi/         chi router, middleware, handlers, JSON I/O
internal/templates/       domain model, service, validation, Grok engine
internal/store/memory/    in-memory store used in tests
internal/store/redisstore Redis implementation of templates.Store

make help            # list available targets
make run             # go run ./cmd/api
make test            # go test -race -count=1 ./...
make test-integration  # also exercise the Redis-backed store (needs Redis)
make fmt             # gofmt -s -w .
make vet             # go vet ./...
make build           # produce ./bin/kippu
make ci              # fmt-check + vet + test (what CI runs)

Integration tests for the Redis store are gated by KIPPU_TEST_REDIS=1 and read REDIS_URL (defaults to redis://127.0.0.1:6379/0). They use isolated key prefixes and clean up after themselves, so they are safe to run against a shared dev Redis.

Images are published to GHCR on every push to main and every v* tag.

docker pull ghcr.io/darkfadr/kippu:latest

docker run --rm -p 8080:8080 \
  -e REDIS_URL=redis://host.docker.internal:6379/0 \
  -e REDIS_KEY_PREFIX=kippu \
  ghcr.io/darkfadr/kippu:latest

The image is built from distroless/static:nonroot, so there is no shell and the process runs as a non-root user by default.

• API authentication (bearer tokens / API keys).
• Prometheus metrics endpoint.
• Batch /v1/execute for high-throughput callers.
• Pattern import/export bundles.
• Optional per-template pattern caching / pre-compilation.

Issues and PRs are welcome. Quick guidelines:

1. Open an issue first for anything larger than a small fix.
2. Add tests for behavioral changes; make ci should pass.
3. Keep PRs focused — one logical change per PR.
4. Follow the repository style (gofmt -s, go vet ./...).

MIT © Kippu contributors.

• elastic/go-grok — the Grok engine that does the real work.
• redis/go-redis — the Redis client.
• go-chi/chi — the HTTP router.