Flare — Log Anomaly Detection

Live Pipeline Demo

Randomly samples blocks from the HDFS dataset and streams them through the full detection pipeline at 5 lines/sec. Each run is different.

Lines:

Drain3 Template Mining

Raw log lines are parsed and clustered into parameterized templates — e.g. Receiving block <*> src: <*> dest: <*>. This reduces millions of unique messages to a handful of structural patterns.

→

Isolation Forest Detection

Each block gets a feature vector: how often each template appeared. Isolation Forest flags blocks whose template distribution looks unusual — e.g. blocks with IOException/ERROR templates but no completion event.

→

DBSCAN Clustering

Anomalous blocks are grouped into incidents by feature similarity. Each incident is enriched with raw log lines, templates, and time range — ready for LLM summarization.

Benchmarked on the full HDFS dataset: 11.2M lines · 575K blocks · 16,838 anomalies → F1 = 0.642, Precision = 0.688, Recall = 0.601

Click Run Demo to watch the pipeline process log windows in real time.

Raw Prometheus Metrics

Live Pipeline Demo