logdrain: Fast, Embeddable Log-Template Mining in Rust

2026-06-28

Every team running services at scale eventually hits the same wall: the logs become noise. Millions of lines an hour, almost all of them minor variations of a few hundred underlying shapes. You can’t read them, and grepping only works once you already know what you’re looking for.

I just open-sourced logdrain to attack that problem at the source: it mines the templates behind your logs, online, in Rust, as an embeddable library and a CLI.

What it does

Feed logdrain a stream of log lines and it learns their templates incrementally, returning a cluster id for each line. No batch training step, no model files. Just add(line).

GET /api/v1/servers/409/metrics 200 12ms   ┐
GET /api/v1/servers/410/metrics 200  9ms   ├─>  GET /api/v1/servers/<*>/metrics <*> <*>
GET /api/v1/servers/873/metrics 503 41ms   ┘
request 550e8400-…-446655440000 done 88ms  ┐
request 6ba7b810-…-00c04fd430c8 done 51ms  ├─>  request <uuid> done <*>
request 7c9e6679-…-e07fc1f90ae7 done 75ms  ┘

In one demo, 5,000,000 noisy lines collapse to 8 templates in about 2 seconds on a single core. Memory tracks the number of templates, not the volume of input.

From the CLI it looks like this:

$ logdrain --path-delimiters / --masks uuid,ipv4,email access.log
ID  SIZE  TEMPLATE
 1     5  GET /api/v1/servers/<*>/metrics <*> <*>
 2     4  POST /api/v1/users/<*>/login <*> from <ipv4>
 3     4  request <uuid> completed in <*>
 …          (30 lines → 9 templates)

A quick mental model: Drain

logdrain is built on Drain (He et al., 2017), a deterministic, fixed-depth prefix-tree algorithm for online log parsing. The intuition is simple:

Tokenize a line.
Walk a short prefix tree keyed first by token count, then by the first few tokens.
At the leaf, compare against existing templates by similarity; if one is close enough, merge into it (replacing differing positions with a <*> wildcard); otherwise start a new template.

No embeddings, no learned model. Just a tree and a similarity threshold. That’s what makes it fast and predictable. It’s the same family of algorithm behind log-pattern features in tools like Grafana Loki.

Watch one get built

The fastest way to get it is to watch four lines arrive at a fresh miner (default settings: tree depth 4 → two prefix levels, similarity threshold 0.4, numeric tokens treated as wildcards during descent).

1. user 42 logged in. Four tokens, so it drops into the “4-token” shard. We walk the first two tokens to find a leaf: user, then 42. Since 42 is numeric, the descent follows the <*> branch. The leaf is empty, so the line becomes a new template:

4-token shard
└─ user
   └─ <*>
      └─ #1  user 42 logged in      (size 1)

2. user 99 logged in. Same shard, same leaf. Now we compare against template #1, position by position: user=user, 42≠99, logged=logged, in=in → 3 of 4 match (0.75 ≥ 0.4), so it merges. The single differing slot is generalized to a wildcard:

4-token shard
└─ user
   └─ <*>
      └─ #1  user <*> logged in     (size 2)   ← "42" and "99" collapsed to <*>

3. user 7 logged in. Same leaf again. Against user <*> logged in every position matches (the <*> absorbs 7), so the template is untouched; only the count moves:

      └─ #1  user <*> logged in     (size 3)

4. disk full on sda. Also four tokens, but the first two (disk, full) walk a different branch, so it can’t collide with the user … family. New leaf, new template:

4-token shard
├─ user
│  └─ <*>
│     └─ #1  user <*> logged in     (size 3)
└─ disk
   └─ full
      └─ #2  disk full on sda       (size 1)

That’s the whole algorithm: token count → a couple of prefix tokens → similarity at the leaf. No backtracking, no all-pairs comparison: each line touches one short path, which is why it stays fast however many templates pile up. (Masking and path-aware tokenization, below, run before this step; they shape the tokens the tree ever sees.)

Try it on your own logs

Here is that same algorithm running in your browser, in a few hundred lines of JavaScript rather than Rust. Paste your own log lines over the examples and watch them collapse to templates as you type. Raise the similarity threshold to split the templates into narrower ones, or toggle the masks, and watch the grouping shift.

similarity threshold 0.4 (higher = stricter, more templates) masks (uuid, ip, email)

0 lines collapsed to 0 templates

#	size	template

The Rust version does much more than this toy: path-aware tokenization, persistent state, stack-trace clustering, exact counts, event-time rates. But the shape of the idea is exactly what you are looking at.

What logdrain adds

A bare Drain implementation gets you the skeleton. The things that make it actually useful in production are what I focused on:

Path-preserving tokenization. /servers/409/foo clusters to /servers/<*>/foo, not a useless <*>. URLs and structured logs stay readable instead of collapsing into mush.
Configurable masks. A regex pre-pass turns high-cardinality tokens (UUIDs, IPs, emails, JWTs) into named placeholders (<uuid>, <ip>) before clustering, so they don’t fragment your templates.
Stack-trace clustering. Multi-line exceptions cluster on their first line, keeping the full trace as a per-cluster suffix, so a flood of stack traces becomes a ranked list of distinct failures.
Exact and persistent. Every template keeps an exact count (no sampling, no top-N truncation), and you can snapshot/restore the whole catalog to a pluggable backend.
Event-time aware. Feed it the timestamp you parsed from each line (--time-key, or add_at in the library) and it tracks real first/last-seen and per-template rates - so even when you replay a week-old file, the rates reflect when events actually happened, not how fast you piped the file.

Stack-trace clustering is the one I reach for most. Point logdrain at a stream of exceptions and it collapses them into a ranked list of distinct failures, and because masks run before the first-line split, per-request ids and IPs in that line become placeholders, so the same failure groups into one template no matter those values:

$ cargo run -p logdrain --example stacktrace
6 traces  ->  3 distinct failures

#1  x3  ERROR NullPointerException req <uuid> at com/acme/svc/OrderHandler.process
        at OrderHandler.process(OrderHandler.java:142)
        at Dispatcher.run(Dispatcher.java:88)
        at java.base/Thread.run(Thread.java:829)

#2  x2  ERROR SQLTimeoutException from <ipv4> at com/acme/db/ConnectionPool.acquire
        at ConnectionPool.acquire(ConnectionPool.java:64)
        at OrderHandler.load(OrderHandler.java:71)

#3  x1  WARN RetryableException at com/acme/net/HttpClient.call
        at HttpClient.call(HttpClient.java:33)

One thing worth being explicit about: the example above drives the library, which takes a whole trace as a single add() call: you decide where a record starts and ends. The CLI is line-oriented by default (one line = one record), which is perfect for plaintext and NDJSON, including a stack trace escaped inside a JSON field (--key stack --first-line-only splits on the embedded \n). When a record genuinely spans multiple physical lines, the CLI gives you two ways to reassemble it, and which one you reach for depends on the stream:

--record-separator <SEP> when records are explicitly framed (a blank line, a NUL byte, a sentinel you control): logdrain --record-separator '\n\n' app.log.
--multiline-start <REGEX> when there is no delimiter (the usual case for raw tracebacks), where a record begins with a timestamp or level line and the frames follow: logdrain --multiline-start '^\d{4}-\d{2}-\d{2}' app.log.

Rule of thumb: if you can point at a character that separates records, use the separator; if the boundary is only implied by what the first line looks like, use the start pattern.

The part I had the most fun with: concurrency in Rust

Miner::add takes &self, not &mut self, so any number of threads can mine into the same miner at once. That one signature is where Rust earned its keep, because it turns “is this concurrent code actually correct?” from a code-review argument into a compile error.

The shape of the problem

The template tree shards by token count. Each shard is an Arc<RwLock<Shard>> living in a DashMap, and every cluster body is an Arc<RwLock<ClusterInner>>. The first version took the shard’s write lock on every add: trivially correct, and it serialized every line that landed in the same shard. Under real traffic that’s most of them.

But the overwhelmingly common case is boring: a line matches an existing template and nothing structural changes; you just bump a counter. Only the rare case (creating a new template, growing a branch) actually needs exclusive access. So the lock you take should depend on what the line turns out to be, not on the fact that you’re writing at all.

Read-then-upgrade

The hot path takes the shard’s read lock, finds the matching cluster, and updates it through a shared reference, with no exclusive lock anywhere:

// recording a hit, under a shared (&) reference:
self.size.fetch_add(1, Ordering::Relaxed);
self.updated_at_ms.store(now_ms(), Ordering::Relaxed);
self.last_used.store(tick, Ordering::Relaxed);

Those fields are AtomicU64. The reason this is sound (and the reason Rust lets you mutate them through &self) is encoded in the type: AtomicU64::fetch_add takes &self, not &mut self, because an atomic read-modify-write is safe from many threads at once. The code only escalates to the shard’s write lock when it genuinely has to create a cluster or extend the tree. Readers run in parallel; writers are the exception.

I picked Ordering::Relaxed on purpose. These are independent tallies with no cross-variable invariant to protect, so paying for SeqCst fences on the hottest line in the program would be pure waste. Rust forces you to name the ordering at every atomic operation, which is mildly annoying for a day and then permanently makes you reason about exactly what each one guarantees: the opposite of the “sprinkle volatile and hope” experience in some other languages.

What the compiler did for me

Send/Sync are inferred and then enforced. Miner is Sync only because every field is: the DashMaps, the Arc<RwLock<…>>s, the atomics. The first time I tried to park a non-Sync value on the hot path, it simply didn’t compile. There’s no “I’m pretty sure this is thread-safe”; the bound holds or it doesn’t.
The borrow checker caught a real lifetime bug. Rebuilding the tree on restore, I held a view that borrowed a cluster’s tokens while that same cluster was being moved into the map: a use-after-move that would be a heisenbug in C++. Here it was a red squiggle; I scoped the borrow in a block and moved on. It never got the chance to become a runtime failure.
The hot path is allocation-free. Tree descent walks borrowed &str slices (with Arc<str>: Borrow<str>, so a lookup never allocates an owned key) and a stack-allocated SmallVec for the prefix. The common “this line matches” case touches the heap zero times.

The payoff: moving the hit path off the write lock and onto atomics, plus going allocation-free, took 8-thread throughput from ~2.2× to ~3.4× over single-threaded. Sub-linear (the ceiling is contention on the hot shared counters), but real, honest scaling.

Trusting it

Two things let me actually believe the code above. First, a concurrency test hammers a single shard from many threads with std::thread::scope and asserts not a single update is lost. Second (and this is the one I’d recommend to anyone writing Rust), cargo-mutants, a mutation tester, caught a bug my unit tests sailed past: the LRU recency key was millisecond-resolution wall-clock, so when many hits landed in the same millisecond, eviction order was effectively random under load. The fix was a monotonic tick (one more AtomicU64), which made eviction both deterministic and testable. That’s a 3am-in-prod class of bug, and a tool found it on my laptop by deleting a line and noticing nothing failed.

Performance, honestly

Numbers from one mid-range Linux box; they vary a lot with hardware and load, so treat them as orders of magnitude:


Throughput (single thread)	~1–2M lines/sec
Throughput (8 threads)	multi-M lines/sec (~3.3× scaling)
`add` latency (steady / cold)	~0.3 µs / ~1.8 µs
Memory	bounded by template count, not input

It’s sub-linear across cores (the ceiling is contention on hot shared template counters), but it’s well into “fast enough for real volumes” territory. You can reproduce it with cargo bench and the bundled scaling example.

Using it as a library

The CLI is handy, but logdrain is built to embed. The whole surface is a few methods on Miner:

use logdrain::{builtin_masks, Miner, UpdateType};

let miner = Miner::builder()
    .path_delimiters(&['/'])
    .masks([builtin_masks::uuid(), builtin_masks::ipv4()])
    .build()?;

// Feed lines from anywhere: a file, a socket, your logging pipeline, a Kafka consumer.
for line in source {
    let res = miner.add(&line);            // cluster id + Created / TemplateChanged / None
    if res.update == UpdateType::Created {
        // a never-seen log shape just appeared: alert, page, or wake an agent
    }
}

// Read the catalog back, or pull the variable parts out of a line.
for c in miner.clusters() {
    println!("{:>7} x  {}", c.size(), c.template());
}
let (_id, params) = miner
    .extract("GET /api/v1/servers/777/metrics from 10.0.0.9")
    .unwrap(); // -> ["777"]  (the <ipv4> stays a named placeholder, not a wildcard)

add(&self) takes a shared reference, so many threads can mine into the same Miner at once. Everything you’d reach for in a pipeline is on that type: masks, path-aware tokenization, first-line stack-trace mode, event-time via add_at, and exact per-template counts.

Keeping state across restarts

The miner is purely in-memory; nothing is written unless you ask. save_state serializes the whole current catalog and hands the bytes to a backend; load_state rebuilds it:

let store = logdrain::FilePersistence::new("templates.bin");
miner.save_state(&store)?;   // atomic write: tmp file -> fsync -> rename

// next run:
let miner = Miner::builder().build()?;
miner.load_state(&store)?;   // back to where you left off, templates and counts intact

Persistence is a tiny synchronous trait: save(&[u8]) and load() -> Option<Vec<u8>>. Beyond the in-memory and file backends, Redis and Kafka ship behind cargo features:

// feature = "redis": latest snapshot under a single key
let store = logdrain::RedisPersistence::new("redis://127.0.0.1/", "logdrain:snapshot")?;

// feature = "kafka": snapshot on the tail of a compacted, single-partition topic
let store = logdrain::KafkaPersistence::new("localhost:9092", "logdrain-snapshots");

Two things to internalize about persistence here:

Writes are caller-driven. There’s no background flush or write-per-line; you decide when to checkpoint (a timer, on shutdown, every N lines), which keeps the hot path free of I/O.
Each snapshot is the whole state, and the whole state is tiny. Memory and the snapshot are bounded by the number of templates, not the volume of input (a few hundred templates after billions of lines), so a full checkpoint is cheap to write and to ship.

One clarification so the mental model is right: these backends are a state store, not a log source. They hold the learned catalog so a process can resume where it left off; they are not a stream you mine from. (Persistence is a library concern; the CLI doesn’t snapshot.)

Where it earns its keep

A few concrete places I reach for it:

incident triage. Production is throwing errors and your log UI is a wall of 200k near-identical lines. Pipe the last hour through logdrain --multiline-start '^\d{4}' --first-line-only and you get a ranked list of distinct failures with counts; the top one is usually the thing paging you. Scrolling becomes reading.
Catching a brand-new failure mode. Wire the Created signal into alerting: the first time a never-seen log shape appears (a new exception after a deploy, a probe from an attacker, a dependency starting to time out), you hear about it immediately, instead of waiting for it to grow loud enough to show up in aggregate dashboards.
Cutting the observability bill. Drop logdrain into a shipping pipeline as a reduction stage: forward (template, count) plus a sample per template instead of every raw line. When a million lines collapse to a few hundred templates, ingest and egress costs fall with them: the Vector/Cribl “reduce” idea, but pattern-aware.
Dashboards from logs that never had a schema. extract() turns each line into (template_id, params), so you can chart error rate or latency by template: real structure pulled out of free text, with --time-key giving you true event-time rates.
A compression layer in front of an AI agent. Hand a model a few hundred templates with counts instead of a million raw lines (see below).

Feeding logs to AI agents

SRE-AI, LLM-based log analysis and incident-triage agents share an awkward problem: you can’t hand a model a million raw log lines. It blows the context window, runs up the token bill, and most of those lines are near-duplicates anyway.

logdrain is a natural front-end for that workflow. It compresses a flood of logs into a few hundred templates with exact counts (often 100,000×+), giving an agent a compact, structured view it can actually reason over:

Put the template catalog (templates + counts + sample params) in the prompt instead of raw lines; the whole “what is this system doing right now” picture fits in context.
Stable template ids are durable handles an agent can reference, correlate, and track over time, instead of re-embedding raw text on every pass.
The Created signal is a cheap trigger: wake the agent only when a genuinely new log shape appears, rather than streaming everything to an LLM.
extract() yields (template_id, params): structured, tool-friendly input for function-calling agents.

Concretely: picture an on-call agent that stays quiet until logdrain emits a Created event, then pulls the current template catalog, notices that payment gateway timeout <*> just went from zero to 4,000/min, and posts a ranked root-cause hypothesis to the incident channel, without ever ingesting a single raw log line. The deterministic engine does the cheap 99% (collapsing noise into structure), so the expensive model only ever looks at the 1% that matters. logdrain isn’t an AI product; it’s the compression and routing layer that makes AI log analysis affordable.

Try it

cargo add logdrain          # the library
cargo install logdrain-cli  # the `logdrain` CLI

On the roadmap: an HTTP service, Redis/S3 persistence, and hierarchical aggregation (logdrain feeding logdrain across a fleet).

Code and docs: github.com/vnvo/logdrain.

If you work with logs at scale, give it a stream of your noise and tell me what breaks.

Diagrams and interactive components in this post were produced with AI.