High Level Architecture

Built for petabyte-scale. Engineered with the most performant modern stack.

Data Flow Diagram

Loading diagram...
🦀

Rust Backend

Core data processing, log parsing, and search coordination written entirely in Rust. Memory-safe, thread-safe concurrency processes millions of events per second.

  • Zero-cost abstractions for data parsing
  • Concurrent indexing pipelines
  • Predictable latency without GC pauses
⚛️

Next.js 16 Frontend

Dashboard and website built on Next.js 16 with App Router. Server Components for fast initial loads, Client Components for interactive visualizations.

  • React Server Components (RSC)
  • Playwright E2E UI Testing
  • Custom CSS Design System with Dark Mode
🔍

Quickwit

Distributed full-text search engine built on object storage. Sub-second search across petabytes with inverted index and columnar aggregation.

  • Object-storage native (S3 / MinIO)
  • Kafka source for real-time indexing
  • ANSI SQL query interface
📊

ClickHouse

Columnar OLAP database for analytics. Handles aggregations, time-series, and complex queries on billions of rows in milliseconds.

  • Columnar compression (10:1 ratio)
  • Vectorized query execution
  • Materialized views for real-time analytics
📡

Redpanda

Kafka-compatible event streaming. Handles raw logs, normalized events, enriched events, and detection alerts across the pipeline.

  • Kafka API compatible
  • Lower latency than Java Kafka
  • Topic-based event routing
🐳

Docker & Kubernetes

Full local development stack via Docker Compose. Production deployment with Kubernetes manifests, HPA, and health probes.

  • One-command local setup (make up)
  • K8s manifests with auto-scaling
  • Multi-stage Docker builds