0Pricing
Cyber Security Academy · Lesson

Testing and Tuning Detections

Reducing false positives.

Testing and Tuning Detections is a free Cyber Security Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Cyber Security Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

The False-Positive Problem

An untuned detection that fires on benign activity is worse than no detection. Analysts learn to ignore noisy alerts, and the one real attack drowns in the queue. This is alert fatigue, and it is how breaches slip past well-funded SOCs.

Tuning is the disciplined process of maximizing true positives while driving false positives toward zero, without blinding yourself to real threats.

True and False, Positive and Negative

Frame detection quality with four outcomes:

  • True positive (TP) — fires on real malicious activity
  • False positive (FP) — fires on benign activity
  • True negative (TN) — correctly stays silent on benign
  • False negative (FN) — misses real malicious activity

Tuning trades along this space. Loosening a rule cuts FNs but risks FPs; tightening does the reverse. The art is finding the balance the SOC can sustain.

Test Data: Known Good and Known Bad

You cannot tune blind. Build a corpus of known-bad samples (the rule must fire) and known-good samples (the rule must stay silent). In Sigma DaC these live as test cases beside the rule.

tests:
  - name: malicious_encoded_powershell
    log: { Image: 'powershell.exe', CommandLine: 'powershell -enc SQBFAFgA' }
    expect: match
  - name: legit_admin_script
    log: { Image: 'powershell.exe', CommandLine: 'powershell -File backup.ps1' }
    expect: no_match

Adversary Emulation

Generate real known-bad telemetry by safely executing the technique. Atomic Red Team provides small, documented tests mapped to ATT&CK that you run in a lab to confirm your rule actually fires.

Run the atomic, capture the logs, and assert the detection triggers. If it does not, your rule has a coverage gap, regardless of how good the YAML looks.

# Run an atomic test for T1059.001 (PowerShell)
Invoke-AtomicTest T1059.001 -TestNumbers 1

# Then confirm the SIEM detection fired for that host/time window

Baselining the Environment

Before deploying a rule to alert, run it in audit mode against historical and live data to learn what normally triggers it. This baseline reveals the legitimate processes, admins, and tools that would otherwise generate FPs.

What looks malicious in the abstract is often a backup job, a vulnerability scanner, or an RMM agent in your specific environment. Baselining surfaces these before they wake an analyst at night.

Tuning with Filters

The cleanest tuning keeps the malicious logic intact and subtracts known-good triggers via explicit filters. This is auditable: a reviewer sees exactly what was excluded and why.

Prefer narrow, justified exclusions over broadening the core logic, which can silently create blind spots.

detection:
  selection:
    Image|endswith: '\\wmic.exe'
    CommandLine|contains: 'process call create'
  filter_sccm:
    ParentImage|contains: '\\CcmExec'
  condition: selection and not filter_sccm

Beware Over-Filtering

Every exclusion is a hole an attacker can hide in. If you filter out all activity from ParentImage containing a tool name, an adversary who masquerades under that name evades you.

Guidelines:

  • Filter on the most specific attribute available (full path, signed-publisher, hash)
  • Document why each filter exists
  • Re-review filters periodically — environments change
  • Prefer raising a threshold over removing a condition

Thresholds and Aggregation

Some behaviors are only suspicious at volume. A single failed login is normal; fifty in a minute from one source is not. Use aggregation in the condition to alert on rate or count rather than each event.

Threshold tuning reduces noise dramatically for brute-force, scanning, and exfiltration patterns.

detection:
  selection:
    EventID: 4625
  timeframe: 1m
  condition: selection | count() by SourceIp > 30

Measuring and Iterating

Tuning is continuous, not one-time. Track per-rule metrics and revisit the worst offenders:

  • Alert volume per rule per day
  • FP rate from analyst dispositions
  • Precision = TP / (TP + FP)
  • Time-to-triage impact

A rule generating 200 alerts a week, all closed as benign, should be tuned, demoted to audit, or retired. Let data drive the decision.

Enrichment to Reduce Triage Load

Not all noise is solved by suppression. Often an alert is technically valid but needs context to disposition. Enrichment attaches that context automatically so analysts decide faster.

  • Asset criticality and owner
  • User role and whether the account is privileged
  • Threat-intel reputation of IPs/domains/hashes
  • Whether the host is in a maintenance window

Enriched alerts let you risk-score and prioritize, cutting effective noise even when raw alert volume is unchanged.

Regression Testing on Every Change

When you tune a rule, re-run its full test suite in CI. The known-bad tests guard against accidentally tuning out real detections; the known-good tests confirm the FP is fixed.

This closed loop is why DaC and tuning belong together: you can refactor aggressively because the tests catch any coverage you break.

pytest tests/windows/wmic_process_create_test.yml
# all known-bad cases still 'match'
# new known-good case now 'no_match'

Quick Check

Apply tuning judgment to a real tradeoff.

Recap

Tuning keeps detections trustworthy and actionable:

  • Unmanaged FPs cause alert fatigue and missed breaches
  • Reason with the TP/FP/TN/FN framework
  • Build known-good and known-bad test corpora
  • Use adversary emulation (Atomic Red Team) to prove rules fire
  • Baseline in audit mode before alerting
  • Tune with narrow, documented filters and thresholds, not broad exclusions
  • Measure FP rate and precision; iterate on the worst rules
  • Re-run regression tests on every change

You have completed the Sigma detection-engineering course.

Frequently asked questions

Is the “Testing and Tuning Detections” lesson free?

Yes — the full text of “Testing and Tuning Detections” is free to read here on the web, and the Cyber Security Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Cyber Security Academy course, upgrade to CoddyKit PRO.

What will I learn in “Testing and Tuning Detections”?

Reducing false positives. You practise Cyber Security Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Cyber Security Academy?

No prior experience is required. Cyber Security Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Testing and Tuning Detections” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Cyber Security Academy lesson?

Yes. Every Cyber Security Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Detection-as-Code Principles
  2. Writing Sigma Rules
  3. Mapping to MITRE ATT&CK
  4. Testing and Tuning Detections
← Back to Cyber Security Academy