Elasticsearch: Indexing and Search

Welcome to the first lesson on the ELK Stack! We'll start with Elasticsearch, the 'E' in ELK.

Elasticsearch is a powerful, open-source distributed search and analytics engine. It's designed to store, search, and analyze large volumes of data quickly.

• Distributed: Runs across multiple servers.
• Real-time: Data is available for search almost instantly.
• Scalable: Easily handles growing data needs.

Elasticsearch stores data as JSON documents. Think of a document as a single record, like a row in a database, but more flexible.

Each document is a collection of fields (key-value pairs) and can contain various data types, including text, numbers, dates, and even other JSON objects.

Here's a simple example of a document:

In Elasticsearch, documents are organized into indices. An index is like a database in a relational database system, or a collection in a NoSQL database.

You can have multiple indices, and each index can store documents that are somewhat related. For example, you might have one index for 'logs' and another for 'products'.

• An index is a logical namespace.
• It groups similar documents.
• You search within specific indices.

Indexing is the process of adding or updating documents in an Elasticsearch index. When you index a document, Elasticsearch stores it and makes it searchable.

Each document needs a unique ID within its index. If you don't provide one, Elasticsearch will generate it for you.

We use HTTP API calls, typically with PUT or POST requests, to interact with Elasticsearch.

Let's index a simple log document into an index called my_logs. We'll specify an ID of 1.

Try running this command (assuming Elasticsearch is running on localhost:9200):

Once a document is indexed, you can retrieve it using its unique ID. This is useful when you know exactly which document you want.

To retrieve a document, you send an HTTP GET request to the specific index and document ID endpoint.

This operation is very fast as Elasticsearch can directly fetch the document.

Let's retrieve the document we just indexed with ID 1 from the my_logs index.

Run this command to see the stored document:

The real power of Elasticsearch comes from its searching capabilities. Instead of knowing an ID, you often want to find documents based on their content.

You can search across all documents in an index (or multiple indices) using various query types. Elasticsearch uses a query language based on JSON.

• Find documents by keywords.
• Filter by date ranges or specific values.
• Combine multiple search criteria.

The simplest search query is the match_all query. It returns all documents in the specified index.

This is often used to verify that documents are indexed correctly or as a starting point for more complex queries.

You send an HTTP GET request to the _search endpoint of your index.

Let's search for all documents in our my_logs index. You'll see the document we indexed earlier.

Run this command:

You've learned about documents, indices, and how to index and retrieve data. Let's test your understanding of indexing.

Great job! In this lesson, you've learned the fundamentals of Elasticsearch:

• Elasticsearch is a distributed search and analytics engine.
• Data is stored as JSON documents.
• Documents are organized into indices.
• Indexing adds or updates documents using PUT/POST requests.
• Documents can be retrieved by ID using GET requests.
• Basic searching can be done with queries like match_all.

Next, we'll dive deeper into Logstash, the 'L' in ELK, to ingest and process data!