Behavioral Detection versus Indicator Matching in Alert Pipelines

A detection rule is a predicate over a stream of events. The predicate’s expressiveness determines which attacker behaviors it can name, and its specificity determines how often it fires on legitimate activity. Those two properties trade against each other, and the trade is different for the two dominant rule families.

An indicator rule matches a concrete artifact: a file hash, a domain, an IP address, a registry path, a byte sequence in a request. Its strength is precision, since a 32-byte hash either matches or does not, and its weakness is that it describes one instance of a tool rather than the behavior the tool performs. A behavioral rule matches a relationship between events: process A created by process B, with a particular argument shape, followed by a network connection to an address outside an internal range. It survives recompilation and re-registration, and pays for that durability in ambiguity, because legitimate software sometimes does the same thing.

Why indicator matching decays

Hashes are exact for the file they were computed from and useless for any other file. Recompiling an implant with a changed build string changes the hash. Changing a single byte of padding changes the hash. Domain and address indicators decay more slowly but still decay: infrastructure rotates on a schedule the operator controls, and a domain is cheap to abandon. The half-life of an address indicator is measured in weeks at best.

This does not make indicators worthless. They are excellent when the artifact is invariant across the attacker’s variants: a distinctive certificate field, a specific User-Agent string that a framework emits by default, a named pipe that a known family uses. The error is architectural: an indicator list is a description of things already seen, and the ruleset’s effectiveness is bounded by the coverage of the last campaign.

The operational consequence is measurable. A ruleset of ten thousand hashes has high precision and low recall against a novel sample, and the false-positive rate is near zero because a random file does not collide with a hash. A ruleset of ten thousand behavioral rules has the opposite profile: recall depends on whether the behavior is genuinely unusual, and false positives accumulate in proportion to how common the behavior is in normal operation.

What a behavioral rule must express

The mechanism-level anchors are mostly about the relationship between an action and its context, and most rule languages reduce to a few primitives.

Lineage: the parent-child relationship. A web server’s worker process spawning a shell is anomalous because the worker has no legitimate reason to. Encoding that requires the rule to know the parent’s identity, which means the event stream must carry it. On Linux, the audit context inheritance described in process start records supplies it; without that inheritance, a rule can only match on the child.

Temporal sequence: A then B within a window. The window is the parameter that determines both usefulness and false-positive rate. A rule stating that a process wrote to a system directory and then started a service will fire constantly on software installation. Narrowing the window reduces false positives and begins to miss the real behavior when the attacker introduces delay.

Rate and aggregation: N occurrences in a window. Failed authentications per source address, file reads per process per minute. These rules are stateless in expression and stateful in execution, and the state must be reset at a defined point; an implementation that resets on calendar boundaries rather than on sliding windows is sensitive to events crossing the boundary.

Entropy or structural anomaly: very long command lines, arguments with a high proportion of non-alphanumeric characters, names that are visually similar to legitimate binaries. These are cheap to express and produce a noisy signal on their own, which makes them useful as one clause of a compound rule.

The rule as a cost function

Every rule produces some number of true positives and some number of false positives per unit time, and the analyst’s attention is the scarce resource being consumed. A rule that fires a hundred times a day with one genuine finding will not be investigated a hundred times a day; it will be tuned, muted, or ignored, and the genuine finding will be missed along with the rest.

This is the strongest argument for behavioral rules over indicator lists within a tuned environment and against both in an untuned one. An untuned behavioral ruleset in an environment with unfamiliar software produces so many alerts that the pipeline stops functioning. Sensitivity depends on knowing what normal looks like for that environment, which is why the same rule can be routine in one environment and the primary detector in another.

The measurable calibration is the ratio of alerts an analyst closes as benign to alerts that require investigation. A ruleset whose ratio is unworkable has too many clauses whose scope exceeds the environment; the fix is narrower rules for specific hosts or user populations, not a global threshold.

Writing the rule so it can be debugged

A detection rule is code and fails in the ways code fails: it does not compile, it compiles and matches nothing, or it matches the wrong thing. The third is the dangerous one, and the defense is a fixture set: known-good and known-bad event captures the rule is executed against.

import os

def rule_shell_from_server(ev, server_procs=frozenset({"nginx", "httpd", "tomcat"})):
    # Fires when a web server worker starts a shell.
    if ev["event"] != "execve":
        return False
    parent = ev.get("parent_comm")
    if parent not in server_procs:
        return False
    child = os.path.basename(ev["argv"][0]) if ev.get("argv") else ""
    return child in {"sh", "bash", "dash", "zsh"}

FIXTURES = [
    ({"event": "execve", "parent_comm": "nginx", "argv": ["/bin/sh", "-c", "id"]}, True),
    ({"event": "execve", "parent_comm": "nginx", "argv": ["/usr/bin/php-fpm"]}, False),
    ({"event": "open", "parent_comm": "nginx", "argv": []}, False),
]
assert all(rule_shell_from_server(e) is want for e, want in FIXTURES)

The fixture set does two jobs. It documents the rule’s intent in executable form, and it makes the rule’s expected firing behavior visible to whoever next modifies it. A rule without fixtures is a rule whose behavior is only known from the alerts it produces, which means the first time anyone learns it is broken is when it fails to fire.

Where behavioral detection runs out

Some behaviors are indistinguishable from legitimate ones at the event level. Reading a large number of files is what a backup agent does and what staging for exfiltration looks like; the same syscall pattern describes both. Distinguishing them requires knowing the process’s identity and history, which is why baseline-informed rules eventually dominate large environments, because the rule’s context expands from the event to the entity.

The second limit is visibility. A rule cannot match an event the sensor does not produce, and sensors differ sharply in coverage. Kernel-level instrumentation sees syscalls that userspace cannot; network sensors see traffic that kernel auditing cannot. Layering both, and treating the gaps as unknown rather than clean, is the basis for the reasoning in log destruction and forwarding gaps: a detector’s silence is only informative relative to what it was in a position to observe.

A third limit is the rule’s own execution model. A rule evaluated per event cannot express “the same account authenticated from two distant locations within a period shorter than travel allows”, because no single event contains the contradiction. Such rules need a state store keyed by entity, with the window held in that store rather than in the event. The store is then part of the detection system’s correctness surface: if it is evicted under memory pressure, the rule silently stops matching entity history and reverts to a weaker stateless approximation, which is the failure mode that makes long-window rules unreliable in practice.

Related to this is the distinction between a rule that detects and a rule that hunts. A hunting query is run against retained data after a hypothesis is formed, is allowed to be expensive, and is allowed to be imprecise because a human reads the output. A detection rule runs continuously, must be cheap enough to execute on every event, and must be precise enough that its output is worth reading. Confusing the two produces detection rules written in a query language designed for hunting, with the cost paid on every event and the precision tuned for a human reader who is not there.