Detection Is a Distributed Systems Problem

Walk into almost any security program and you will find the same center of gravity: the detections. The rules, the analytics, the ATT&CK coverage map, the backlog of new logic waiting to ship. That is where the attention goes, the hiring goes, and the vendor pitches land. It is also, in my experience building these systems, almost never where the failures that matter come from.

The detections are rarely the hard part. The hard part is the thing underneath them: a high-volume, real-time, stateful streaming platform that has to collect, normalize, enrich, and correlate telemetry from thousands of sources, under adversarial load, without dropping, duplicating, or reordering the events that a detection depends on. That platform is a distributed system. A demanding one. And the security field keeps describing it as plumbing.

The better recent writing has started to admit the pipeline matters. It talks about buffering, schema normalization, and retention, which is real progress. But it still frames the pipeline as a data-quality concern, a hygiene layer you get right so the detections can do the interesting work. That framing stops exactly where the engineering begins. Delivery semantics, event-time ordering, backpressure, windowed state, checkpoint recovery: these are not hygiene. They are the discipline that decides whether your detections fire correctly, on time, and at all. I learned this the way most people learn distributed systems, which is by operating one while it was under stress and watching where it bent.

Four stages, one streaming system

The canonical detection pipeline is drawn as four boxes: collect, normalize, enrich, correlate. Drawn that way, it looks like a data-processing chore. Look at each stage as a distributed-systems problem and a different picture appears, because each one hides a decision that the security framing never names.

Collect is a delivery-semantics decision. Every source, an endpoint agent, a cloud audit log, a network sensor, forces a choice you rarely see written down: at-least-once or exactly-once delivery. It is not academic. A dropped authentication event is a detection that silently never fires. A duplicated one, replayed after a buffer recovers, is a false-positive storm landing on an analyst at 3 a.m. Most pipelines quietly choose at-least-once because it is easier, then absorb the duplicates somewhere downstream, or fail to, and nobody connects the duplicate-alert pain back to the delivery choice made three systems upstream.

Normalize is a problem about time. The security framing treats normalization as mapping fields into a common schema, and it is that. But the harder half is temporal. There is the time an event happened, the time it was collected, and the time it was ingested, and they can differ by seconds or by hours. Clocks across your fleet are skewed. If your pipeline reasons about ingest time because that is what is easy to stamp, then any detection that depends on the order of events, which is most of the interesting ones, is reasoning about a fiction. Event time versus processing time is one of the foundational problems in stream processing, and detection lives right on top of it.

Enrich is a stateful join against moving data. Useful detections need context: which user owns this identity, which asset is this, is this IP on a current threat-intel list. Each of those is a join against a reference dataset that is itself changing, and that can lag or be unavailable at the moment the event flows through. Enrich an event against a stale asset inventory and the correlation that depends on it quietly draws the wrong conclusion. The enrichment source becomes a production dependency of your detection correctness, with its own freshness and failure modes, and almost nobody treats it with that seriousness.

Correlate is distributed stateful stream processing, full stop. This is where the real system lives, and where the rules-centric view is most misleading. The detections that matter are almost never single-event rules. They are patterns across events, across sources, across time: this process spawned that connection, which preceded this data movement, within a window. Computing that means holding state, windowing it, joining streams whose events arrive out of order, and deciding what to do when the event that completes a pattern shows up after the window you were going to close. That is partitioned state, watermarks, checkpointing, and recovery. It is the beating heart of frameworks like Flink, and it is the part no detection-rule syntax will save you from.

flowchart LR
    subgraph SRC["Sources"]
        direction TB
        S1["Endpoint agents"]
        S2["Cloud audit logs"]
        S3["Network sensors"]
        S4["Identity / app logs"]
    end

    subgraph COLLECT["Collect<br/><i>delivery semantics</i>"]
        B["Durable buffer<br/>backpressure · replay"]
    end

    subgraph NORM["Normalize<br/><i>event-time, schema</i>"]
        N["Common schema<br/>event-time stamping<br/>watermarks"]
    end

    subgraph ENRICH["Enrich<br/><i>stateful joins</i>"]
        E["Identity · asset inventory<br/>· threat intel"]
    end

    subgraph CORR["Correlate<br/><i>partitioned stateful processing</i>"]
        C["Windowed multi-stream joins<br/>checkpointed state"]
    end

    SRC --> COLLECT --> NORM --> ENRICH --> CORR --> D["Detections<br/>& response"]

    REF["Reference state<br/>identity, assets, intel"] -. moving data .-> ENRICH
    CORR -. late events reopen windows .-> CORR

Four stages, read as one streaming system. The labels under each stage are the distributed-systems decisions the rules-centric view never names.

The failures nobody files against the pipeline

Here is the tell that an organization is misattributing the problem. Go read the postmortems. The detection failures get filed against the detection: the rule was wrong, the threshold was off, the analytics model needs tuning. A surprising share of them are not detection failures at all.

The detection that fired three hours after the incident was not slow logic. It was telemetry that arrived late because a collector buffered through a network blip and nobody monitored freshness, so the rule evaluated against data that was not there yet.

The correlation that missed a textbook attack chain did not have a gap in its logic. Event B arrived before event A’s enrichment landed, the join ran against an un-enriched record, and the pattern never assembled. That is a watermark-and-ordering bug wearing a detection-gap costume.

The 40,000 duplicate alerts that buried the on-call during the actual breach were not a rule tuning issue. They were at-least-once delivery replaying a backed-up buffer at the worst possible moment, and a correlation layer with no idempotency to absorb it.

None of those get fixed by writing better detections. They get fixed by engineering the platform: freshness as a monitored SLO, event-time processing with an explicit watermark policy, idempotent correlation keys, backpressure that degrades gracefully instead of collapsing and replaying. Every one of those is a distributed-systems practice, and none of them is in the detection-engineering job description as it is usually written.

The adversarial case is a load problem

There is one property that makes a detection pipeline unlike almost any other data platform, and it is worth stating plainly because it changes the engineering priorities. For most streaming systems, peak load and the event of interest are independent. For a detection pipeline, they arrive together. The attack is the load spike. A credential-stuffing run, a lateral-movement burst, a data-exfiltration event: the thing you most need to detect is also the thing generating the telemetry flood that can knock your pipeline into backpressure, buffering, and eventual drop or replay.

This is the single most under-appreciated fact in detection architecture. Capacity and backpressure handling are not site-reliability niceties you get to later. They are security controls. A pipeline that silently sheds load under volume has a denial-of-service path straight through its detection coverage, and a sufficiently aware adversary will use it: generate enough noise to push the pipeline past its headroom, and the signal you would have caught gets dropped or delayed past relevance. If you have not load-tested your detection platform at several multiples of normal volume, you do not actually know what your coverage is during the exact moment it matters most.

sequenceDiagram
    participant A as Event A (t=0)
    participant B as Event B (t=1)
    participant P as Pipeline (event-time)
    participant W as Window + state
    participant D as Detection

    A->>P: arrives late (ingest t=5)
    B->>P: arrives early (ingest t=2)
    Note over P: order by EVENT time, not arrival
    B->>W: opens correlation window
    Note over W: watermark holds window<br/>for late arrivals
    A->>W: late event reopens / completes pattern
    W->>D: pattern assembled correctly
    Note over A,D: Ingest-time ordering would have missed this entirely

Two events in a classic attack chain, arriving out of order. Reason about event time with a watermark and the pattern assembles. Reason about ingest time, the easy default, and the detection silently never fires.

What this means for how you build and staff

If the preceding is right, then a conclusion follows that most security orgs have not absorbed: a detection program is a distributed-systems program with a security specialization, not a security program with a data pipeline attached. The scarcest skill on a mature detection team is not another analyst who can write another rule. It is someone who understands delivery semantics, event-time processing, partitioned state, and backpressure, and who can reason about the pipeline as the reliability-critical distributed system it is.

I say this as someone who came to detection from the systems side rather than the SOC side. At Versive I built a real-time machine-learning pipeline that processed large volumes of network telemetry to surface malicious activity, and the model was never the hard part. The pipeline was: getting telemetry in reliably, ordering it honestly, enriching it against moving context, and holding correlation state at volume without falling over. Later, building real-time decisioning platforms in regulated finance, the same lesson held in a different domain. The streaming substrate was the product. The decision logic rode on top of it and was only ever as good as the platform’s guarantees.

What this is really about

The reframe is the whole point. Detection is not a library of rules sitting on top of a log store. It is a real-time, stateful, distributed streaming system that happens to emit security decisions, and the quality of your detection is bounded by the quality of that system long before it is bounded by the cleverness of your rules.

There is a cost to taking this seriously, and it is honest to name it. Exactly-once semantics, event-time processing with watermarks, checkpointed correlation state, and real backpressure headroom all cost latency, infrastructure, and operational discipline. You are buying correctness and resilience under load with complexity and spend. In a low-stakes environment that trade may not be worth it. In anywhere that a missed or delayed detection is a genuine loss, it is not optional, and pretending the pipeline is plumbing is how you end up with a coverage map that quietly lies to you.

The organizations that will struggle with detection over the next few years are not the ones with the thinnest rule libraries. They are the ones who staffed and architected detection as a content problem and never built the distributed system underneath it. The ones who will pull ahead are the ones who realized, early, that they were running a streaming data platform the entire time.

I would value sharp disagreement on this, especially from people running detection at scale: where does the rules-centric view still earn its keep, and where has the pipeline turned out to be the thing that decides your coverage?

Narendra Loganathan is a distinguished-level engineer and architect with 25 years building large-scale distributed systems, real-time data platforms, security telemetry and detection pipelines, and more recently agentic AI systems, including real-time ML threat detection over network telemetry and real-time decisioning in regulated finance. He publishes open reference architectures at naren-chakraview.github.io.