Feed logdrain a stream of noisy log lines and it learns their templates incrementally - returning a cluster id per line, with no batch training and no model files.

GET /api/v1/servers/409/metrics 200 12ms   ┐
GET /api/v1/servers/410/metrics 200  9ms   ├─►  GET /api/v1/servers/<*>/metrics <*> <*>
GET /api/v1/servers/873/metrics 503 41ms   ┘
request 550e8400-…-446655440000 done 88ms  ┐
request 6ba7b810-…-00c04fd430c8 done 51ms  ├─►  request <uuid> done <*>
request 7c9e6679-…-e07fc1f90ae7 done 75ms  ┘

5,000,000 noisy lines → 8 templates in ~2 s on a single core. Memory tracks the number of templates, not the volume of input.

add(line) returns a cluster id instantly - created, generalized, or matched. No batch step.

/servers/409/foo clusters to /servers/<*>/foo, not <*>. URLs and structured logs stay readable.

UUIDs, IPs, emails, JWTs → named placeholders before clustering - built-in or custom.

Cluster multi-line traces on their first line and keep the full trace as a per-cluster suffix.

extract() recovers the variable values of a line; attach de-duplicated labels per cluster.

Feed a parsed timestamp per line (add_at, or the CLI's --time-key) for true event-time first/last-seen and rates, independent of processing wall-clock.

Exact per-template counts (no sampling, no top-N) with snapshot/restore to a pluggable backend - in-memory, file, or Redis/Kafka behind features.

add(&self) from many threads; a sharded, mostly-lock-free hot path (~1-2M lines/sec per core).

Collapse millions of lines into a handful of templates, ranked by volume, to see what's actually happening.

Alert on the Created signal, the first time a never-seen log shape appears (new errors, attacks, regressions).

Rank distinct stack-trace failures by frequency, so you triage the top one instead of scrolling.

Store (template, count) instead of raw lines, or sample by template - a Vector/Cribl-style stage.

Turn unstructured logs into (template_id, params) for dashboards, ML features, or alerting.

Compress millions of lines into a few hundred templates + counts so an LLM or incident-triage agent can reason over the whole picture - feed the catalog and the Created signal, not raw lines.

Embed it in a log shipper to summarize at the source and cut egress and ingest cost.

cargo add logdrain                  # the library
cargo install logdrain-cli          # the `logdrain` CLI, on your $PATH

Reads files (or stdin), one record per line, and prints the templates it found:

$ logdrain --path-delimiters / --masks uuid,ipv4,email access.log
ID  SIZE  TEMPLATE
 1     5  GET /api/v1/servers/<*>/metrics <*> <*>
 2     4  POST /api/v1/users/<*>/login <*> from <ipv4>
 3     4  request <uuid> completed in <*>
 4     4  signup for <email> from <ipv4>
 5     4  cache miss for key <*>
 …          (30 lines → 9 templates)

cat app.log | logdrain --format json --sort size   # JSON, biggest clusters first
cat events.ndjson | logdrain --key event.message    # pull a field out of JSON lines
logdrain --format jsonl app.log | jq -c '{template, size}'   # stream one object/line into jq, Vector, Kafka, ...

createdAt is observation time, not log time. The createdAt / updatedAt fields (and the derived linesPerMinute / rate_per_min) are the machine's wall-clock when logdrain first/last saw each template - logdrain does not parse timestamps from your records by default. Streaming a live feed they approximate event time; replaying a historical file they reflect when you ran the command and how fast you fed it, not the original event times or rate.

For true event time, pass --time-key <field> (with optional --time-format). logdrain then parses that timestamp out of each record and reports eventFirstSeen / eventLastSeen / eventRatePerMinute - which reflect when events actually happened, even when replaying historical logs:

logdrain --key msg --time-key ts app.ndjson --format jsonl   # epoch or RFC 3339, auto-detected
logdrain --key m --time-key t --time-format '%Y-%m-%d %H:%M:%S' app.ndjson

By default the CLI treats one line as one record. That already covers the common cases:

• Plaintext logs (one event per line).
• NDJSON - one JSON object per line; use --key <field> to pull the message out.
• A stack trace escaped inside a JSON field - since the whole event is still one physical line, --key stack --first-line-only splits on the embedded \n and clusters on the first line.

When a single record spans multiple physical lines (raw console stack traces, pretty-printed JSON), pick one of two options - which one depends on whether the stream already marks record boundaries:

1. --record-separator <SEP> - when records are explicitly framed. Use it when a known delimiter sits between records (or you control the producer and can emit one): a blank line, a NUL byte, a sentinel string. Exact and unambiguous. Escapes \n \r \t \0 \\ are interpreted.

logdrain --record-separator '\n\n' app.log     # records separated by a blank line
producer | logdrain --record-separator '\0'     # NUL-framed stream

2. --multiline-start <REGEX> - when there is no delimiter. Use it for ordinary logs you don't control, where a record begins with a recognizable line (a timestamp or level) and continuation lines (stack frames) follow. A line matching the regex starts a new record; every non-matching line is appended to it.

logdrain --multiline-start '^\d{4}-\d{2}-\d{2}' app.log   # records start at a timestamp
logdrain --multiline-start '^(ERROR|WARN|INFO)' --first-line-only app.log

Rule of thumb: if you can point at a character that separates records, use --record-separator. If the boundary is only implied by what the first line looks like (the usual case for tracebacks), use --multiline-start. The two are mutually exclusive. Embedding the library? Skip both - you decide record boundaries and pass each whole record (newlines and all) to add().

First-line mode (--first-line-only, or .first_line_only(true)) changes how much of a record is used to cluster. Off (default), the whole record is tokenized; on, the record is split at the first newline and only the first line drives clustering and the template - the rest is kept as a per-cluster suffix (from the first occurrence). It only matters for records that span multiple lines.

That's exactly what stack traces need: the frames differ every time, but the first line (exception type + message) is stable, so all occurrences collapse into one template (with masks applied, so per-request ids / IPs become placeholders) while one full trace is preserved as the suffix. The example below uses the library (each trace passed to add() as one record); from the CLI, assemble multi-line traces with --multiline-start or --record-separator (see Input model):

$ cargo run -p logdrain --example stacktrace
6 traces  ->  3 distinct failures

#1  x3  ERROR NullPointerException req <uuid> at com/acme/svc/OrderHandler.process
        at OrderHandler.process(OrderHandler.java:142)
        at Dispatcher.run(Dispatcher.java:88)
        at java.base/Thread.run(Thread.java:829)

#2  x2  ERROR SQLTimeoutException from <ipv4> at com/acme/db/ConnectionPool.acquire
        at ConnectionPool.acquire(ConnectionPool.java:64)
        at OrderHandler.load(OrderHandler.java:71)

#3  x1  WARN RetryableException at com/acme/net/HttpClient.call
        at HttpClient.call(HttpClient.java:33)

use logdrain::{builtin_masks, Miner};

let miner = Miner::builder()
    .path_delimiters(&['/'])
    .masks([builtin_masks::uuid(), builtin_masks::ipv4()])
    .build()
    .unwrap();

miner.add("GET /api/v1/servers/409/metrics from 10.0.0.1");
miner.add("GET /api/v1/servers/410/metrics from 10.0.0.2");

for c in miner.clusters() {
    println!("#{} x{}  {}", c.id(), c.size(), c.template());
    // => #1 x2  GET /api/v1/servers/<*>/metrics from <ipv4>
}

// Classify a new line and recover its variable parts.
let (_id, params) = miner.extract("GET /api/v1/servers/777/metrics from 10.0.0.9").unwrap();
assert_eq!(params, ["777"]); // <ipv4> is a named placeholder, not a wildcard

use logdrain::{FilePersistence, Miner};

let store = FilePersistence::new("templates.bin");
miner.save_state(&store).unwrap();        // atomic write (tmp + fsync + rename)

let restored = Miner::builder().build().unwrap();
restored.load_state(&store).unwrap();     // rebuilds the tree from the snapshot

logdrain = { version = "0.3", features = ["redis"] }  # or "kafka", or both

// Redis: latest snapshot stored under one key.
let store = logdrain::RedisPersistence::new("redis://127.0.0.1/", "logdrain:snapshot")?;

// Kafka: snapshot written to the tail of a (compacted, single-partition) topic.
let store = logdrain::KafkaPersistence::new("localhost:9092", "logdrain-snapshots");

miner.save_state(&store)?;
miner.load_state(&store)?;

Numbers from one mid-range Linux box - they vary with hardware and load, so treat them as orders of magnitude and re-run on your own target.

Reproduce: cargo bench -p logdrain --bench add, or cargo run --release -p logdrain --example scaling.

Available: logdrain (library) and logdrain-cli, published and tested, with Redis and Kafka persistence backends behind cargo features. Roadmap: an HTTP service (draind), an S3 persistence backend, and hierarchical/distributed aggregation.

Issues and pull requests welcome. Please keep the standard checks green:

cargo test --workspace
cargo clippy --workspace --all-targets -- -D warnings
cargo fmt --all --check

Dual-licensed under Apache-2.0 or MIT, at your option. Minimum supported Rust version: 1.85.