Vector Databases: Pinecone, Weaviate & pgvector · 课时

使用 HNSW 索引提升召回率

探索 pgvector 的 HNSW 索引，在相似度搜索中获得更高召回率，并平衡速度与准确性。

使用 HNSW 索引提升召回率 是 CoddyKit 上的免费 Vector Databases: Pinecone, Weaviate & pgvector 课时。这是第 2 节课，共 4 节。你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。这是 Vector Databases: Pinecone, Weaviate & pgvector 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 Vector Databases: Pinecone, Weaviate & pgvector 课程共包含 4 节课。

本课时的部分内容尚未翻译，以英文显示。

Boost Recall with HNSW

Welcome to HNSW indexing! In the previous lesson, we explored IVFFlat for speed. Now, we'll dive into Hierarchical Navigable Small World (HNSW), an advanced indexing technique in pgvector.

HNSW is excellent when you need to find most of the relevant results, even if it means a slight trade-off in query speed compared to IVFFlat. This is known as high recall.

HNSW vs. IVFFlat: A Quick Look

Remember IVFFlat indexes? They partition data for faster, approximate searches, optimizing for speed. HNSW takes a different approach to prioritize recall.

IVFFlat: Faster queries, good enough recall.
HNSW: Higher recall (finds more true positives), potentially slower build and query times.

Choosing between them depends on your application's needs: speed or comprehensive results.

How HNSW Indexes Work

Imagine HNSW as a multi-layered graph. It connects similar vectors across different layers:

Top layers: Sparse graphs, quickly navigate large distances.
Bottom layers: Dense graphs, fine-tune search for nearest neighbors.

This structure allows for efficient approximate nearest neighbor (ANN) search, quickly narrowing down the search space to find highly similar vectors.

Creating an HNSW Index

To use HNSW, you first need the pgvector extension. Then, you can create an HNSW index on your vector column. Here's the basic syntax:

CREATE INDEX ON items USING HNSW (embedding vector_l2_ops);

The vector_l2_ops specifies using L2 (Euclidean) distance. Other operators like vector_cosine_ops for cosine similarity are also available.

HNSW Parameter: `m` (Max Connections)

The m parameter determines the maximum number of connections a node (vector) has in the HNSW graph on each layer. It's crucial for index quality:

Higher m: More connections, better recall, but increases index size and build time.
Lower m: Fewer connections, smaller index, faster build, but lower recall.

A common value for m is between 8 and 16, but it depends on your dataset and desired accuracy.

HNSW Parameter: `ef_construction`

The ef_construction parameter controls the size of the dynamic candidate list during graph construction. It impacts how thoroughly the index is built:

Higher ef_construction: More thorough search during build, better index quality (higher recall), but significantly slower build time.
Lower ef_construction: Faster build, but potentially lower recall.

It's generally recommended to set ef_construction to a value 2-4 times m, or even higher for very high recall needs.

Code: Create an HNSW Index

Let's create a table and then an HNSW index with specific parameters. This example uses m=16 and ef_construction=64.

CREATE EXTENSION IF NOT EXISTS vector;

DROP TABLE IF EXISTS docs;
CREATE TABLE docs (
    id serial PRIMARY KEY,
    embedding vector(3)
);

INSERT INTO docs (embedding) VALUES
    ('[1,2,3]'),
    ('[1.1,2.1,3.1]'),
    ('[10,11,12]'),
    ('[10.5,11.5,12.5]'),
    ('[100,101,102]');

CREATE INDEX ON docs USING HNSW (embedding vector_l2_ops) WITH (
    m = 16,
    ef_construction = 64
);

Querying with HNSW Indexes

Once your HNSW index is built, pgvector automatically uses it for similarity queries. The query syntax is the same as for other vector indexes:

SELECT id, embedding <-> '[1,2,3]' AS distance FROM docs ORDER BY distance LIMIT 3;

However, HNSW introduces another parameter at query time: ef_search.

HNSW Parameter: `ef_search`

The ef_search parameter controls the size of the dynamic candidate list during the actual search operation. You set this via a session variable:

Higher ef_search: More thorough search at query time, higher recall, but slower query execution.
Lower ef_search: Faster queries, but potentially lower recall.

You typically set ef_search equal to or higher than ef_construction for optimal results, or tune it based on real-world query performance.

HNSW Trade-offs & Considerations

While HNSW offers superior recall, it comes with trade-offs:

Memory Usage: HNSW indexes are generally larger and consume more memory than IVFFlat.
Build Time: Index creation can be significantly slower, especially with high m and ef_construction.
Query Latency: Queries might be slightly slower than IVFFlat, depending on ef_search.

Always test with your specific dataset to find the best balance of parameters for your application.

Check Your HNSW Knowledge

Which HNSW parameter primarily affects the recall and build time of the index by controlling the thoroughness of the graph construction?

Recap: HNSW for Recall

Great job! You've explored HNSW indexing in pgvector.

HNSW prioritizes recall, aiming to find most relevant results.
It works by building a multi-layered graph structure.
Key parameters are m (max connections) and ef_construction (build thoroughness).
ef_search tunes query-time recall and speed.
HNSW indexes can be larger and slower to build/query than IVFFlat, but offer higher recall.

Next, we'll learn how to tune queries for optimal performance!

常见问题解答

「使用 HNSW 索引提升召回率」课时是免费的吗？

是的 — 「使用 HNSW 索引提升召回率」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 Vector Databases: Pinecone, Weaviate & pgvector 课程的其余内容，请升级到 CoddyKit PRO。 Vector Databases: Pinecone, Weaviate & pgvector 课程共包含 4 节课。

「使用 HNSW 索引提升召回率」这节课中我会学到什么？

探索 pgvector 的 HNSW 索引，在相似度搜索中获得更高召回率，并平衡速度与准确性。你通过在浏览器中直接运行的动手代码来练习 Vector Databases: Pinecone, Weaviate & pgvector，全天候 AI 导师会在你学习这节课的过程中回答你的问题。

学习 Vector Databases: Pinecone, Weaviate & pgvector 需要有经验吗？

无需任何先前经验。CoddyKit 上的 Vector Databases: Pinecone, Weaviate & pgvector 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。这是第 2 节课，共 4 节。

「使用 HNSW 索引提升召回率」课时需要多长时间？

大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。

我能在这节 Vector Databases: Pinecone, Weaviate & pgvector 课中编写并运行代码吗？

能。每节 Vector Databases: Pinecone, Weaviate & pgvector 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。

此课程中的所有课时

← 返回 Vector Databases: Pinecone, Weaviate & pgvector