Indexing Documents into Elasticsearch

Welcome to indexing! In Elasticsearch, indexing is the process of storing data into an index to make it searchable.

Think of it like adding a new book to a library's catalog. You provide the book's details, and the library stores them in a way that makes the book easy to find later.

Before we dive in, let's quickly recap two core concepts:

• Document: A basic unit of information in Elasticsearch, similar to a row in a traditional database. It's usually a JSON object.
• Index: A collection of documents that have similar characteristics. It's like a database in a relational world.

When you index, you add a document to an index.

You interact with Elasticsearch using its REST API. To index a document, you'll typically use HTTP POST or PUT requests.

• POST /<index>/_doc: Used to index a document, often letting Elasticsearch generate an ID.
• PUT /<index>/_doc/<id>: Used to index a document with a specific, user-provided ID.

Let's see them in action!

The simplest way to index is to let Elasticsearch generate a unique ID for your document. You use the POST method to the _doc endpoint without specifying an ID.

Here's an example using curl to index a document into an index named products:

After the previous POST request, Elasticsearch would return a response including a unique _id for your document, like "_id": "AbCdEfGhIjKlMnOpQrSt".

When should you use auto-generated IDs?

• When you don't have a natural unique identifier for your data.
• For logs or temporary data where a unique ID isn't critical for external reference.
• When you want to guarantee a new document is always created.

Often, your data already has a unique identifier from another system (e.g., a database primary key). In such cases, you can provide your own ID using the PUT method.

The ID is specified directly in the URL path: /<index>/_doc/<your_id>.

Using custom IDs offers several advantages:

• Integration: Easily map Elasticsearch documents to records in an external database.
• Predictability: You know the document's ID beforehand.
• Updates: It makes updating specific documents straightforward, as you always refer to them by their known ID.

A key concept when using PUT with a custom ID is idempotency. This means that performing the same operation multiple times will produce the same result as performing it once.

• If a document with the specified ID already exists, PUT will update it.
• If it doesn't exist, PUT will create it.

This is different from POST, which always creates a *new* document with a new ID.

While indexing documents one-by-one is fine for small numbers, it can be inefficient for large datasets due to network overhead.

Elasticsearch provides a powerful _bulk API that allows you to perform multiple index, update, or delete operations in a single request. This dramatically improves indexing performance.

We'll explore the _bulk API in more detail in a future lesson!

Imagine you have a new set of sensor readings. Each reading is unique, and you don't have a predefined ID for them, but you want to store them in Elasticsearch to be searchable.

Great job! In this lesson, you learned the fundamentals of indexing documents into Elasticsearch:

• What indexing means and its role in making data searchable.
• The difference between documents and indices.
• How to use POST /<index>/_doc to index documents with auto-generated IDs.
• How to use PUT /<index>/_doc/<id> to index documents with custom IDs.
• The concept of idempotency when using PUT.
• A brief introduction to the efficiency of bulk indexing.

Next, we'll explore more operations on these documents!