NOD: Uncovering Intense Attackers’ Behavior Through Nested Outlier Detection From SSH Logs
Proceedings of the Workshop on Attack Provenance, Reasoning, and Investigation for Security in the Monitored Environment (PRISM) 2026.
abstract
Persistent, high-volume SSH brute-force activity frequently overwhelms security operations, yet current defenses often treat network telemetry as a terminal artifact for post-hoc diagnosis rather than a source for upstream investigation. These approaches focus on absolute volume suppression and binary alerts, often failing to provide population-aware rankings that are necessary to prioritize high-risk, relative outliers. This work addresses these gaps by introducing Nested Outlier Detection (NOD), a two-stage framework that transforms raw network telemetry into structured behavioral strata. By progressively filtering routine noise, NOD isolates "outliers of outliers"; statistically extreme behaviors. NOD provides interpretability by mapping these outliers to three intuitive dimensions; volume, reach, and credential diversity; enabling population-level reasoning. This tiered approach reveals distinct attacker phenotypes characterized by high volume, broad target reach, and a variety of credentials. Evaluation on large-scale datasets demonstrates that NOD compresses millions of logs into compact, interpretable structures, shifting the defensive focus from per-source classification to the graded, population-level reasoning required for scalable triage and longitudinal threat analysis.