Anomaly and Behavioral Detection

Signatures catch known threats. They are blind to novel malware, zero-days, and custom tooling that no one has fingerprinted yet. Anomaly and behavioral detection fills that gap by flagging traffic that deviates from normal.

The premise: even unknown attacks leave behavioral traces. An adversary may evade every signature yet still beacon on an odd interval, move data at an unusual hour, or scan internal hosts.

Anomaly detection requires a model of normal. You build a baseline over time, profiling things like:

• Which hosts talk to which, and on what ports
• Typical bytes-per-flow and connection durations
• Normal volumes by hour and day of week
• Expected protocols per segment

The baseline must be learned from clean traffic over a representative period; a baseline built during an active compromise bakes the attack into normal.

The simplest behavioral detections are statistical. Compute a metric, learn its normal distribution, and alert when a value falls far outside, for example beyond three standard deviations or a high percentile.

Examples: a workstation that normally sends 50 MB/day suddenly uploads 5 GB, or a host opening connections to thousands of distinct internal IPs in minutes.

Much behavioral analysis runs on flow records rather than full packets. A flow summarizes a conversation: source, destination, ports, protocol, byte and packet counts, and duration.

Flows (NetFlow, IPFIX, or Suricata/Zeek conn logs) are compact, work even on encrypted traffic, and scale to enterprise volumes. They are the backbone of traffic-volume and connection-pattern detection.

Command-and-control channels often beacon: the implant calls home at a regular interval. Even over TLS, where payload is hidden, the timing pattern is visible.

Detect it by analyzing the distribution of intervals between connections to a destination. Low variance (with optional jitter) plus small, similar request sizes is a strong C2 indicator. Tools like RITA specialize in this.

DNS is a rich behavioral signal because attackers abuse it for tunneling and C2.

• High-entropy / long domains suggest DGA (domain generation algorithms)
• Large query volume to one domain suggests DNS tunneling
• Unusual record types (TXT, NULL) in bulk are suspicious
• Newly registered domains carry elevated risk

Profiling query rates, name length, and entropy per domain catches data exfiltration that no payload signature would see.

You cannot read encrypted payloads, but you can fingerprint how the encryption is negotiated. JA3/JA3S hash the TLS ClientHello/ServerHello parameters, producing a fingerprint of the client and server stacks.

Malware families often share a JA3 because they use the same TLS library and settings. Profiling JA3 frequency lets you flag rare fingerprints or match known-bad ones, all without decryption.

Behavioral detection looks for actions attackers must take, regardless of tooling. High-value behaviors include:

• Lateral movement — a host suddenly authenticating to many peers
• Internal scanning — one source touching many ports/hosts
• Data staging/exfil — large outbound transfers to new destinations
• Protocol on wrong port — SSH on 443, HTTP on 53

These map cleanly to ATT&CK tactics and resist evasion better than payload signatures.

User and Entity Behavior Analytics (UEBA) applies machine learning to model per-entity normalcy and score deviations. Clustering and time-series models can surface subtle changes a fixed threshold misses.

ML adds power but also risk: opaque models, concept drift as the network changes, and adversarial evasion. Treat ML scores as one signal among many, not an oracle, and keep a human in the loop for high-impact decisions.

Zeek (formerly Bro) is a network analysis framework that turns traffic into rich, structured protocol logs rather than alerts. It is the workhorse behind much behavioral detection.

Zeek produces per-protocol logs (conn, dns, http, ssl, files, x509) and lets you script custom analyzers. Feeding these logs into a SIEM gives analysts the connection, DNS, and TLS metadata needed to baseline normal and spot deviations, even across encrypted traffic.

Anomaly detection's strength, catching the unknown, is also its weakness: not every anomaly is malicious. New business apps, patch days, backups, and seasonal traffic all look anomalous.

Manage this by:

• Risk-scoring and correlating multiple weak signals before alerting
• Enriching with asset and identity context
• Feeding anomalies to hunters rather than auto-blocking
• Continuously retraining baselines as the environment evolves

Apply behavioral reasoning to encrypted C2.

Behavioral and anomaly detection catches what signatures cannot:

• Detect deviations from a learned baseline of normal
• Use statistical thresholds on volume, fan-out, and timing
• Run analysis on flow records (NetFlow/Zeek), effective even on encrypted traffic
• Spot beaconing via interval regularity
• Profile DNS entropy/volume for DGA and tunneling
• Fingerprint TLS with JA3/JA3S without decrypting
• Hunt behaviors: lateral movement, scanning, exfil
• ML/UEBA adds power but needs human oversight and tuning

Next: tuning and deploying these sensors in the real network.