Vector Databases: Pinecone, Weaviate & pgvector · 강의

재현율을 위한 HNSW 색인

속도와 정확도의 균형을 유지하면서 유사도 검색의 재현율을 높이는 pgvector용 HNSW 색인을 살펴봅니다.

재현율을 위한 HNSW 색인은(는) CoddyKit의 무료 Vector Databases: Pinecone, Weaviate & pgvector 강의입니다. 이것은 4개 중 2번째 강의입니다. 아래에서 전체 강의를 무료로 읽을 수 있으며, 내장 코드 에디터와 24/7 AI 튜터와 함께 브라우저에서 직접 실습할 수 있습니다. 이 강의는 Vector Databases: Pinecone, Weaviate & pgvector 학습 경로의 일부이며, 진행 상황이 웹과 CoddyKit 앱에 동기화됩니다. Vector Databases: Pinecone, Weaviate & pgvector 강의에는 총 4개의 강의가 포함되어 있습니다.

이 강의의 일부는 아직 번역되지 않았으며 영어로 표시됩니다.

Boost Recall with HNSW

Welcome to HNSW indexing! In the previous lesson, we explored IVFFlat for speed. Now, we'll dive into Hierarchical Navigable Small World (HNSW), an advanced indexing technique in pgvector.

HNSW is excellent when you need to find most of the relevant results, even if it means a slight trade-off in query speed compared to IVFFlat. This is known as high recall.

HNSW vs. IVFFlat: A Quick Look

Remember IVFFlat indexes? They partition data for faster, approximate searches, optimizing for speed. HNSW takes a different approach to prioritize recall.

IVFFlat: Faster queries, good enough recall.
HNSW: Higher recall (finds more true positives), potentially slower build and query times.

Choosing between them depends on your application's needs: speed or comprehensive results.

How HNSW Indexes Work

Imagine HNSW as a multi-layered graph. It connects similar vectors across different layers:

Top layers: Sparse graphs, quickly navigate large distances.
Bottom layers: Dense graphs, fine-tune search for nearest neighbors.

This structure allows for efficient approximate nearest neighbor (ANN) search, quickly narrowing down the search space to find highly similar vectors.

Creating an HNSW Index

To use HNSW, you first need the pgvector extension. Then, you can create an HNSW index on your vector column. Here's the basic syntax:

CREATE INDEX ON items USING HNSW (embedding vector_l2_ops);

The vector_l2_ops specifies using L2 (Euclidean) distance. Other operators like vector_cosine_ops for cosine similarity are also available.

HNSW Parameter: `m` (Max Connections)

The m parameter determines the maximum number of connections a node (vector) has in the HNSW graph on each layer. It's crucial for index quality:

Higher m: More connections, better recall, but increases index size and build time.
Lower m: Fewer connections, smaller index, faster build, but lower recall.

A common value for m is between 8 and 16, but it depends on your dataset and desired accuracy.

HNSW Parameter: `ef_construction`

The ef_construction parameter controls the size of the dynamic candidate list during graph construction. It impacts how thoroughly the index is built:

Higher ef_construction: More thorough search during build, better index quality (higher recall), but significantly slower build time.
Lower ef_construction: Faster build, but potentially lower recall.

It's generally recommended to set ef_construction to a value 2-4 times m, or even higher for very high recall needs.

Code: Create an HNSW Index

Let's create a table and then an HNSW index with specific parameters. This example uses m=16 and ef_construction=64.

CREATE EXTENSION IF NOT EXISTS vector;

DROP TABLE IF EXISTS docs;
CREATE TABLE docs (
    id serial PRIMARY KEY,
    embedding vector(3)
);

INSERT INTO docs (embedding) VALUES
    ('[1,2,3]'),
    ('[1.1,2.1,3.1]'),
    ('[10,11,12]'),
    ('[10.5,11.5,12.5]'),
    ('[100,101,102]');

CREATE INDEX ON docs USING HNSW (embedding vector_l2_ops) WITH (
    m = 16,
    ef_construction = 64
);

Querying with HNSW Indexes

Once your HNSW index is built, pgvector automatically uses it for similarity queries. The query syntax is the same as for other vector indexes:

SELECT id, embedding <-> '[1,2,3]' AS distance FROM docs ORDER BY distance LIMIT 3;

However, HNSW introduces another parameter at query time: ef_search.

HNSW Parameter: `ef_search`

The ef_search parameter controls the size of the dynamic candidate list during the actual search operation. You set this via a session variable:

Higher ef_search: More thorough search at query time, higher recall, but slower query execution.
Lower ef_search: Faster queries, but potentially lower recall.

You typically set ef_search equal to or higher than ef_construction for optimal results, or tune it based on real-world query performance.

HNSW Trade-offs & Considerations

While HNSW offers superior recall, it comes with trade-offs:

Memory Usage: HNSW indexes are generally larger and consume more memory than IVFFlat.
Build Time: Index creation can be significantly slower, especially with high m and ef_construction.
Query Latency: Queries might be slightly slower than IVFFlat, depending on ef_search.

Always test with your specific dataset to find the best balance of parameters for your application.

Check Your HNSW Knowledge

Which HNSW parameter primarily affects the recall and build time of the index by controlling the thoroughness of the graph construction?

Recap: HNSW for Recall

Great job! You've explored HNSW indexing in pgvector.

HNSW prioritizes recall, aiming to find most relevant results.
It works by building a multi-layered graph structure.
Key parameters are m (max connections) and ef_construction (build thoroughness).
ef_search tunes query-time recall and speed.
HNSW indexes can be larger and slower to build/query than IVFFlat, but offer higher recall.

Next, we'll learn how to tune queries for optimal performance!

자주 묻는 질문

“재현율을 위한 HNSW 색인” 강의는 무료인가요?

네 — “재현율을 위한 HNSW 색인” 전체 내용을 이 웹사이트에서 무료로 읽을 수 있습니다. 인터랙티브하게 실습하려면(내장 코드 에디터와 24/7 AI 튜터), CoddyKit PRO로 업그레이드하면 Vector Databases: Pinecone, Weaviate & pgvector 강의 전체를 잠금 해제할 수 있습니다. Vector Databases: Pinecone, Weaviate & pgvector 강의에는 총 4개의 강의가 포함되어 있습니다.

“재현율을 위한 HNSW 색인”에서 뭘 배우나요?

속도와 정확도의 균형을 유지하면서 유사도 검색의 재현율을 높이는 pgvector용 HNSW 색인을 살펴봅니다. 브라우저에서 직접 실행하는 실습 코드로 Vector Databases: Pinecone, Weaviate & pgvector을(를) 배우며, 24/7 AI 튜터가 강의를 진행하면서 질문에 답변해줍니다.

Vector Databases: Pinecone, Weaviate & pgvector을(를) 시작하는 데 경험이 필요한가요?

사전 경험은 필요하지 않습니다. CoddyKit의 Vector Databases: Pinecone, Weaviate & pgvector은(는) 초급자부터 고급 학습자까지를 위해 구성되어 있으므로, 여기서 시작하거나 처음부터 시작할 수 있으며 자신의 속도대로 진행할 수 있습니다. 이것은 4개 중 2번째 강의입니다.

“재현율을 위한 HNSW 색인” 강의는 얼마나 걸리나요?

대부분의 CoddyKit 강의는 약 5~10분이 소요됩니다. 각 강의는 간결하고 인터랙티브하여 꾸준한 진행이 가능하며, 웹과 앱에서 중단한 부분부터 바로 시작할 수 있습니다.

이 Vector Databases: Pinecone, Weaviate & pgvector 강의에서 코드를 작성하고 실행할 수 있나요?

네. 모든 Vector Databases: Pinecone, Weaviate & pgvector 강의에는 내장 코드 에디터가 포함되어 있으므로, 브라우저에서 바로 실제 코드를 작성하고 실행한 후 즉시 AI 피드백을 받을 수 있습니다 — 로컬 설정이 필요 없습니다.

이 강의의 모든 강의

← Vector Databases: Pinecone, Weaviate & pgvector(으)로 돌아가기