การสร้างดัชนี การกรอง และการค้นหาแบบผสม

Comparing a query against millions of vectors one by one is too slow for production. Vector databases use indexes to find near neighbors quickly without scanning everything.

Most vector indexes are approximate (ANN): they trade a tiny bit of accuracy for huge speed gains. For RAG, near-perfect recall at fast speed is a great deal.

HNSW (Hierarchical Navigable Small World) is a popular graph-based index. It navigates layers of connections to reach neighbors fast, balancing speed and accuracy via tunable parameters.

IVF clusters vectors into cells; a query only searches the closest cells. The nprobe parameter trades recall for speed by controlling how many cells to check.

Pure similarity can return the wrong scope — old versions, other tenants. Storing metadata with each vector lets you filter results to the right subset.

Filters apply two ways:

• Pre-filter: restrict candidates before the ANN search (more correct)
• Post-filter: search, then drop non-matches (may return too few)

Prefer pre-filtering when the DB supports it.

Vectors capture meaning but can miss exact terms — product codes, names, acronyms. A query for SKU-9F may semantically match nothing useful.

Hybrid search fixes this.

Hybrid search runs both keyword (e.g. BM25) and vector search, then merges the results. You get semantic understanding plus exact-term precision.

Reciprocal Rank Fusion merges the two ranked lists by rewarding items ranked high in either, without needing comparable score scales.

Many databases let you weight keyword vs vector contributions (alpha). Term-heavy domains lean keyword; conceptual queries lean vector. Tune on your test set.

For healthy vector search at scale:

• Rebuild or update indexes as data grows
• Keep embeddings and index dimensions consistent
• Benchmark recall and latency together

You learned to scale and sharpen vector search:

• ANN indexes like HNSW and IVF make search fast
• Metadata filters scope results; prefer pre-filtering
• Hybrid search blends keyword and vector retrieval
• Fuse rankings with RRF and tune the balance
• Maintain indexes and benchmark recall vs latency

These techniques keep retrieval both fast and accurate in production.