SIEM Architecture: Log Ingestion, Parsing, and Correlation

A Security Information and Event Management (SIEM) platform aggregates log data from across an organization's infrastructure and analyzes it for signs of security incidents. SIEM combines two capabilities: Security Information Management (SIM) — storing and analyzing historical log data — and Security Event Management (SEM) — real-time monitoring and alerting. Together they give security teams visibility across the entire environment from a single interface.

A SIEM ingests logs from diverse sources: firewalls and IDS/IPS, operating systems (Windows Event Logs, Linux syslog), authentication systems (Active Directory, RADIUS, Okta), endpoints (EDR agents), cloud platforms (AWS CloudTrail, Azure Activity Log), applications (web servers, databases), and network devices (switches, routers, VPN gateways). The breadth of ingestion determines the SIEM's detection coverage.

Raw logs arrive in different formats: CEF (Common Event Format), LEEF (Log Event Extended Format), syslog, JSON, Windows XML, and proprietary formats. The SIEM's parser extracts structured fields (timestamp, source IP, destination IP, user, event type) from raw text. Normalization maps these fields to a common schema so that queries and rules can work across all log sources uniformly, regardless of origin format.

Event correlation is the process of combining related events from multiple sources to identify patterns that indicate a security incident. A single failed login is noise; 50 failed logins across 10 accounts in 5 minutes from the same IP is a brute-force attack. Correlation engines apply time-windowed rules to streams of events, grouping related events and generating alerts when suspicious patterns emerge from the noise.

SIEM platforms come in two main architectural flavors. On-premises SIEMs (Splunk Enterprise, IBM QRadar, ArcSight) provide full data control but require significant infrastructure and operational overhead. Cloud-native SIEMs (Microsoft Sentinel, Google Chronicle, Elastic SIEM) offer elastic scaling, reduced operational burden, and native integration with cloud services. Many organizations operate hybrid architectures, using cloud SIEM for cloud logs and on-prem for sensitive data that cannot leave the environment.

SIEMs organize ingested data into indexes or tables by log type or time period. Data pipelines pre-process logs before storage: filtering noise (excluding health check pings), enriching fields (adding geolocation to IP addresses), and routing high-volume logs to cheaper storage tiers. Retention policies determine how long logs are kept — compliance requirements often mandate 12 months online, with additional archival storage for 7 years.

SIEM platforms use specialized query languages for searching log data. Splunk SPL (Search Processing Language) uses a pipe-based syntax. Microsoft Sentinel uses KQL (Kusto Query Language). Elastic uses EQL (Event Query Language) and Lucene. These languages allow analysts to filter, aggregate, join, and visualize log data to investigate incidents and build detection rules. Proficiency in the SIEM's query language is a core analyst skill.

Modern SIEMs integrate with threat intelligence platforms (TIP) to automatically enrich events with context. When an event contains an IP address or domain name, the SIEM checks threat intel feeds (VirusTotal, AlienVault OTX, commercial feeds) and appends whether that indicator is known-malicious, its threat category, and confidence score. This enrichment dramatically speeds up triage — analysts see context without having to manually look up each indicator.

SIEM dashboards provide at-a-glance operational visibility for SOC teams. Common dashboard panels include: top alert sources by severity, authentication failure trends over time, geographic maps of incoming connections, user activity anomaly scores, and active incident counts. Dashboards serve different audiences — analysts need operational detail, while managers need KPI summaries like mean time to detect (MTTD) and alert volume trends.

Alert fatigue occurs when too many false positives overwhelm analysts, causing real threats to be missed. Managing false positives in a SIEM requires: tuning correlation rules by adding exclusions for known-good behavior, risk scoring to prioritize high-confidence alerts, suppression rules to mute repetitive benign patterns, and regular review of alert volume metrics. The goal is a manageable volume of high-fidelity alerts that analysts can investigate thoroughly.

Security Orchestration, Automation, and Response (SOAR) platforms integrate with SIEMs to automate the response to common alert types. When a SIEM fires an alert, SOAR can automatically: query the user's activity, check the device compliance status, look up the IP in threat intel, block the IP at the firewall, disable the user account, and create an incident ticket — all within seconds. SOAR enables SOC teams to handle higher alert volumes without proportionally increasing headcount.

Test your understanding of CompTIA Security+ (SY0-701) concepts from this lesson.

In this lesson you learned: SIEMs aggregate and normalize logs from diverse sources into a common schema, event correlation engines apply time-windowed rules to detect attack patterns across multiple events, and threat intelligence integration and SOAR automation accelerate analyst response and reduce alert fatigue. Next up we explore how to write effective SIEM detection rules and alerts.