A rule that costs more than it catches
The client and every number in this exercise are fictional, made up for practice.
Background
A detection rule looks for remote administration tools being run on servers. It is one of the noisiest rules in a client’s security operations center. Most of its alerts come from a legitimate remote administration tool the client’s IT team uses every day. A detection engineer proposes an exclusion for that tool.
| Item | Value |
|---|---|
| Alerts | 1,800 |
| Alerts confirmed as malicious (true positives) | 27 |
| Analyst time to triage one alert | 6 minutes |
| False positives the proposed exclusion would remove | 70% |
| True positives the exclusion would remove (from last month’s data) | 0 |
Your task
- What is the rule’s precision, and how many analyst hours does it cost a month, before and after the exclusion?
- What is the risk of the exclusion, and how would you design it to keep that risk small?
- The client is assessed against SOC 2. What should the change leave behind as evidence?
One way to approach it
Before. Precision is 27 ÷ 1,800 = 1.5%. Triage costs 1,800 × 6 = 10,800 minutes = 180 hours a month, more than a full-time analyst, to find 27 real events.
After. False positives fall from 1,773 by 70% (about 1,241) to about 532. With the 27 true positives, that is about 559 alerts. Precision rises to 27 ÷ 559 ≈ 4.8%, and triage falls to 559 × 6 = 3,354 minutes ≈ 56 hours a month, saving about 124 hours.
The risk. Attackers use legitimate administration tools precisely because defenders exclude them. A broad exclusion (“ignore this tool everywhere”) creates a blind spot. Keep it narrow: the signed binary from the approved publisher, on the hosts and from the administrator accounts that actually use it, launched in the expected way. Add a separate, lower-volume detection for that tool used from unusual accounts, hosts or times. Review the exclusion on a schedule and whenever the IT team’s tooling changes. Zero true positives removed last month is encouraging but is one month of data; test the exclusion against past incidents and red-team activity before relying on it.
The evidence. The change should go through the client’s change management process like any other: a ticket with the reason (the alert volumes above), the exact logic of the exclusion, who reviewed and approved it, when it was deployed, and how it was tested. A note of the compensating detection and the review date completes the record. An assessor sampling detection changes should be able to see why the rule changed and that it was approved, not just that it changed.