Using Observability for Security

Welcome! In this lesson, we'll explore how observability — our ability to understand a system from its external outputs — is a powerful tool for enhancing security.

It's not just for performance! Logs, metrics, and traces provide crucial insights into system behavior, helping us detect and respond to security threats.

Logs are often the first line of defense. They record events, giving us a detailed history of what happened in a system. For security, we focus on specific types of log entries:

• Authentication: Successful and failed login attempts.
• Authorization: Changes to user permissions or access.
• Access: Attempts to access sensitive files or data.
• System Changes: Configuration updates or software installations.
• Network Events: Connection attempts, firewall blocks.

Example log entry:

By analyzing logs, we can identify patterns that often indicate malicious activity. Some common examples include:

• Brute-force attacks: Numerous failed login attempts from a single IP address or user account in a short period.
• Port scanning: Repeated connection attempts to various ports on a target system.
• Unauthorized access: Log entries showing access to resources by users without appropriate permissions.
• SQL injection attempts: Malformed database queries appearing in application logs.

Structured logging makes querying and filtering these patterns much easier!

Metrics provide aggregated data over time, which can reveal security anomalies by showing deviations from normal behavior. Look for:

• Failed Login Rate: A sudden spike could signal a brute-force attack.
• Network Traffic (Egress/Ingress): Unexpected increases might indicate data exfiltration or a Denial-of-Service (DoS) attack.
• API Error Rates: High error rates on specific endpoints, especially authorization errors (e.g., HTTP 401/403), could mean attack attempts.
• Resource Usage: Unusual spikes in CPU or memory could indicate malware, cryptominers, or unauthorized processes.

Distributed traces track a single request as it flows through multiple services. This end-to-end view is incredibly valuable for security:

• Malicious Request Path: See the entire journey of an unauthorized request, identifying all services it touched.
• Unexpected Service Calls: Detect if a service is calling another service it shouldn't, or performing an unusual operation.
• Data Exfiltration: Trace a request that might be attempting to extract sensitive data, seeing where the data originated and where it was sent.

Traces provide the crucial context of an operation.

Once you know what to look for, you can set up alerts to notify you of suspicious log events. This is often done using search queries on your centralized log management system.

Examples of log-based alerts:

• Alert if event.action: "login_failed" count exceeds 50 within 5 minutes from a single source.ip.
• Alert if user.role: "admin" performs an event.action: "delete_database" outside of normal business hours.
• Alert if any log contains a specific string indicating a known exploit (e.g., "union select password" for SQL injection).

Similarly, metric-based alerts can warn you when key performance indicators related to security cross certain thresholds. These alerts are great for detecting widespread or high-volume attacks.

Consider these examples:

• Alert if the http.server.requests.status_401_total metric (total 401 Unauthorized responses) exceeds 100 per minute across the application.
• Alert if network.bytes_sent_total for the entire system increases by 200% compared to its 7-day average.
• Alert if process.cpu_usage for an application server remains above 80% for more than 10 minutes during off-peak hours.

The true power of observability for security comes from correlating all three signals: logs, metrics, and traces. No single signal tells the whole story.

• A spike in failed login metrics (metric) can trigger an investigation into specific log entries to identify the attacking IPs and usernames.
• An unusual API call observed in a trace can be cross-referenced with logs for associated errors or unauthorized attempts.
• An unauthorized access log can be linked to a trace ID to see the full path of the malicious request through your services.

This combined view enables faster and more accurate incident response.

To maximize your security posture with observability, follow these best practices:

• Granular Logging: Log enough detail to be useful, but avoid logging sensitive data directly.
• Centralized Collection: Aggregate all logs, metrics, and traces into a single, queryable platform.
• Baseline Monitoring: Understand your system's 'normal' behavior to more easily spot anomalies.
• Regular Review: Periodically audit your security alerts and dashboards to ensure they are still relevant and effective.
• Access Control: Implement least privilege for access to observability tools and data themselves.

A user reports that their account was locked after multiple failed login attempts. Your security team suspects a brute-force attack. Which observability signals are most useful for detecting this specific type of attack and understanding its scope?

Great job! You've learned how observability plays a critical role in system security. By leveraging logs, metrics, and traces, you can:

• Identify suspicious patterns and anomalies.
• Set up proactive alerts for potential threats.
• Gain deep contextual understanding during security incidents.
• Correlate data across signals for faster root cause analysis.

Integrating observability into your security strategy helps build more resilient and secure systems. Keep exploring how these powerful tools can safeguard your applications!