System Observability: Logging, Metrics & Tracing (ELK + OpenTelemetry) · 课时

Logstash 过滤器与管道

探索高级 Logstash 配置，包括条件逻辑、多个管道和自定义过滤器，以实现复杂的数据转换。

Logstash 过滤器与管道 是 CoddyKit 上的免费 System Observability: Logging, Metrics & Tracing (ELK + OpenTelemetry) 课时。这是第 2 节课，共 4 节。你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。这是 System Observability: Logging, Metrics & Tracing (ELK + OpenTelemetry) 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 System Observability: Logging, Metrics & Tracing (ELK + OpenTelemetry) 课程共包含 4 节课。

本课时的部分内容尚未翻译，以英文显示。

Intro to Advanced Logstash

Welcome to an advanced look at Logstash! So far, you've learned how to get logs into Logstash and apply basic filters. But what happens when your data gets more complex?

In this lesson, we'll explore powerful techniques like conditional logic, managing multiple pipelines, and leveraging advanced filters for intricate data transformations. This will help you handle real-world logging challenges.

Conditional Logic: The 'if' Statement

Not all logs are created equal! You might have different log formats coming from various services, or you might want to process events differently based on their content.

Conditional logic allows Logstash to apply filters or outputs only when certain conditions are met. This is achieved using if statements, similar to programming languages.

Use if to check field values, tags, or other event properties.
Apply specific filters or actions only to matching events.

Conditional Logic in Action

Let's see a simple example. We'll use if to check if a field named type exists and has a specific value. If it does, we'll add a tag.

Try inputting {"message": "Hello", "type": "app_log"} and then {"message": "World"} to see the difference.

input {
  stdin {
    codec => json
  }
}
filter {
  if [type] == "app_log" {
    mutate {
      add_tag => ["processed_app_log"]
    }
  }
}
output {
  stdout {
    codec => rubydebug
  }
}

Beyond 'if': Else and Else If

Just like in programming, you can extend your conditional logic with else if and else blocks. This allows for more complex branching and ensures every event is handled appropriately.

Logstash executes these conditions sequentially. The first matching condition's block is executed, and then it moves on.

if [field] == "value": Executes if the condition is true.
else if [another_field] == "another_value": Executes if the first if was false, and this condition is true.
else: Executes if none of the preceding if or else if conditions were true.

Multiple Pipelines: Why Separate?

As your system grows, you might be collecting logs from many different sources (e.g., web servers, databases, security devices). Each source might require entirely different processing logic.

Multiple pipelines allow you to isolate and manage these distinct processing flows independently. Instead of one giant, complex Logstash configuration, you can have several smaller, focused ones.

Isolation: Errors in one pipeline won't affect others.
Resource Management: Assign specific resources to different pipelines.
Modularity: Easier to develop, test, and maintain configurations.

Configuring Multiple Pipelines

To use multiple pipelines, you define them in a file called pipelines.yml, usually located in your Logstash configuration directory (e.g., /etc/logstash/pipelines.yml).

Each entry specifies a unique ID, the path to its configuration file (.conf), and optional settings like number of worker threads.

Example pipelines.yml:

- pipeline.id: web_logs
  path.config: "/etc/logstash/conf.d/web-pipeline.conf"
- pipeline.id: db_logs
  path.config: "/etc/logstash/conf.d/db-pipeline.conf"

Deep Dive: The Ruby Filter

Sometimes, built-in Logstash filters aren't enough for very specific or complex data transformations. That's where the ruby filter comes in!

The ruby filter allows you to execute arbitrary Ruby code within your Logstash pipeline. This provides immense flexibility to manipulate events in ways not possible with standard filters.

Use for: Complex string manipulations, mathematical operations, custom data lookups, or logic that depends on multiple fields.
Caution: Can impact performance if not used carefully.

Ruby Filter Example

Let's use the ruby filter to create a new field that combines parts of existing fields and calculates a value.

Try inputting: {"user_id": "123", "item_count": 5, "price_per_item": 10.5}

input {
  stdin {
    codec => json
  }
}
filter {
  ruby {
    code => "
      event.set('total_cost', event.get('item_count').to_f * event.get('price_per_item').to_f)
      event.set('user_item_summary', 'User ' + event.get('user_id') + ' bought ' + event.get('item_count').to_s + ' items.')
    "
  }
}
output {
  stdout {
    codec => rubydebug
  }
}

Advanced Mutate Operations

The mutate filter is a workhorse for basic field manipulation, but it has some advanced operations that are incredibly useful:

split: Splits a string field into an array based on a delimiter.
join: Joins an array field into a string using a specified separator.
convert: Changes the data type of a field (e.g., string to integer, float to string).
rename: Changes the name of an existing field.

These operations help you shape your data precisely for storage and analysis.

Quiz: Logstash Logic

Which of the following are valid reasons to use multiple Logstash pipelines?

Recap: Advanced Logstash Config

Great job! You've leveled up your Logstash skills. We covered:

How conditional logic (if, else if, else) allows for dynamic event processing.
The benefits and configuration of multiple pipelines for modular and isolated data flows.
Leveraging the powerful ruby filter for highly custom data transformations.
Advanced operations within the mutate filter like split, join, and convert.

These techniques are crucial for building robust and adaptable Logstash configurations for complex, real-world data.

常见问题解答

「Logstash 过滤器与管道」课时是免费的吗？

是的 — 「Logstash 过滤器与管道」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 System Observability: Logging, Metrics & Tracing (ELK + OpenTelemetry) 课程的其余内容，请升级到 CoddyKit PRO。 System Observability: Logging, Metrics & Tracing (ELK + OpenTelemetry) 课程共包含 4 节课。

「Logstash 过滤器与管道」这节课中我会学到什么？

探索高级 Logstash 配置，包括条件逻辑、多个管道和自定义过滤器，以实现复杂的数据转换。你通过在浏览器中直接运行的动手代码来练习 System Observability: Logging, Metrics & Tracing (ELK + OpenTelemetry)，全天候 AI 导师会在你学习这节课的过程中回答你的问题。

学习 System Observability: Logging, Metrics & Tracing (ELK + OpenTelemetry) 需要有经验吗？

无需任何先前经验。CoddyKit 上的 System Observability: Logging, Metrics & Tracing (ELK + OpenTelemetry) 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。这是第 2 节课，共 4 节。

「Logstash 过滤器与管道」课时需要多长时间？

大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。

我能在这节 System Observability: Logging, Metrics & Tracing (ELK + OpenTelemetry) 课中编写并运行代码吗？

能。每节 System Observability: Logging, Metrics & Tracing (ELK + OpenTelemetry) 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。

此课程中的所有课时

← 返回 System Observability: Logging, Metrics & Tracing (ELK + OpenTelemetry)