Cyber Security Academy · Lesson

SIEM Architecture and Log Ingestion

Set up a Splunk or ELK stack pipeline to collect and index logs from multiple sources.

SIEM Architecture and Log Ingestion is a free Cyber Security Academy lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Cyber Security Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

What is a SIEM?

A SIEM (Security Information and Event Management) system collects, normalizes, correlates, and alerts on log data from across the environment. It provides a single pane of glass for security operations.

SIEM Core Functions

Key functions: log collection and normalization, real-time correlation and alerting, historical search and investigation, compliance reporting, and incident response workflow.

# SIEM capabilities:
# Collection: agents, syslog, API pulls
# Normalization: common event schema
# Correlation: rule-based and ML-based
# Alerting: threshold, pattern, anomaly
# Dashboards: SOC analyst views
# Reporting: compliance evidence

Popular SIEM Platforms

Commercial: Splunk, Microsoft Sentinel, IBM QRadar, Exabeam. Open source: ELK Stack (Elasticsearch, Logstash, Kibana) with security extensions, Wazuh, Graylog.

ELK Stack Architecture

ELK Stack: Elasticsearch (search/storage), Logstash (ingestion/parsing), Kibana (visualization/query). Filebeat and Winlogbeat are lightweight shippers that forward logs to Logstash or directly to Elasticsearch.

# ELK data flow:
# Log Source → Filebeat/Winlogbeat
#    → Logstash (parse, enrich, filter)
#    → Elasticsearch (index/store)
#    → Kibana (search/visualize/alert)

# Direct flow (simpler):
# Log Source → Filebeat → Elasticsearch → Kibana

Filebeat Configuration

Filebeat is a lightweight log shipper. Configure inputs (which log files to read) and output (Logstash or Elasticsearch endpoint). Enable specific modules for common log formats.

# filebeat.yml excerpt:
filebeat.inputs:
- type: log
  paths:
    - /var/log/auth.log
    - /var/log/apache2/access.log

output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  index: "filebeat-%{+yyyy.MM.dd}"

# Enable Apache module:
filebeat modules enable apache

Winlogbeat for Windows

Winlogbeat ships Windows Event Logs to Elasticsearch. Configure which event channels to collect: Security, System, Application, Sysmon, and PowerShell Operational.

# winlogbeat.yml
winlogbeat.event_logs:
  - name: Security
    event_id: 4624, 4625, 4648, 4672, 4769
  - name: Microsoft-Windows-Sysmon/Operational
  - name: Microsoft-Windows-PowerShell/Operational

output.elasticsearch:
  hosts: ["elasticsearch:9200"]

Logstash Parsing with Grok

Logstash uses Grok patterns to parse unstructured log lines into structured fields. Pre-built patterns exist for most common log formats (Apache, syslog, Nginx).

# Logstash pipeline for Apache logs:
filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  date {
    match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
  geoip {
    source => "clientip"
  }
}

Splunk Log Ingestion

Splunk ingests logs via Universal Forwarder (agent), HEC (HTTP Event Collector for APIs), or syslog. Splunk automatically extracts fields from known formats and uses props.conf/transforms.conf for custom parsing.

# Splunk Universal Forwarder config
# inputs.conf:
[monitor:///var/log/auth.log]
index = linux_security
sourcetype = linux_secure

# HEC (HTTP Event Collector)
curl -k "https://splunk:8088/services/collector/event" \
  -H "Authorization: Splunk YOUR_TOKEN" \
  -d '{"event": "Hello Splunk"}'

Sysmon for Enhanced Windows Logging

Sysmon (System Monitor) extends Windows logging with: process creation (Event 1), network connections (Event 3), file creation (Event 11), and registry modifications (Event 13). Essential for detection engineering.

# Install Sysmon with config
sysmon64 -accepteula -i sysmonconfig.xml

# Key Sysmon event IDs:
# 1:  Process Create
# 3:  Network connection
# 7:  Image loaded (DLL)
# 11: File created
# 13: Registry set value
# 22: DNS query

Index and Retention Management

Plan index lifecycle: hot (fast SSD, recent 7 days), warm (slower, 30 days), cold (archive, 90-365 days). Balance query performance against storage cost based on compliance requirements.

Data Volume Planning

Estimate log volume: a typical endpoint generates 1-5 GB/day of events; a busy web server 10-50 GB/day. Size Elasticsearch cluster accordingly and set retention policies before going live.

Quick Check

What does Sysmon Event ID 1 log?

Summary: SIEM Architecture

A SIEM collects and correlates logs at scale. ELK Stack provides an open-source path: Filebeat/Winlogbeat ship logs, Logstash parses them, Elasticsearch indexes them, Kibana visualizes and searches. Add Sysmon for rich Windows endpoint visibility. Plan data volumes and retention before deployment — retroactive resizing is painful.

Frequently asked questions

Is the “SIEM Architecture and Log Ingestion” lesson free?

Yes — the full text of “SIEM Architecture and Log Ingestion” is free to read here on the web, and the Cyber Security Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Cyber Security Academy course, upgrade to CoddyKit PRO.

What will I learn in “SIEM Architecture and Log Ingestion”?

Set up a Splunk or ELK stack pipeline to collect and index logs from multiple sources. You practise Cyber Security Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Cyber Security Academy?

No prior experience is required. Cyber Security Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “SIEM Architecture and Log Ingestion” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Cyber Security Academy lesson?

Yes. Every Cyber Security Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

Log Sources: OS, Network, and Application Logs
SIEM Architecture and Log Ingestion
Writing Detection Rules and Correlation
Alert Triage and SOC Workflow

← Back to Cyber Security Academy