0Pricing
AI Engineering Academy · Lesson

Choosing and Benchmarking Vector Stores

Compare Pinecone, pgvector, Chroma, Weaviate, and Qdrant across cost, latency, filtering capabilities, and operational complexity to pick the right tool for your use case.

Choosing and Benchmarking Vector Stores is a free AI Engineering Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Engineering Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

The Vector Store Landscape

The ecosystem of vector databases has exploded in the past few years. Options range from purpose-built cloud services like Pinecone to PostgreSQL extensions like pgvector, open-source servers like Chroma, Qdrant, and Weaviate, and in-memory libraries like FAISS. Choosing the right tool matters because switching later is costly once your data is indexed.

Key Dimensions to Evaluate

When comparing vector stores, evaluate five dimensions: query latency at your target scale, indexing throughput for batch ingestion, filtering capabilities for metadata-based pre-filtering, operational complexity (managed vs self-hosted), and cost per million vectors stored and queried. No single tool wins on every dimension.

Pinecone: Managed Simplicity

Pinecone is a fully managed cloud vector database that requires zero infrastructure management. It excels at high-concurrency workloads, offers native sparse-dense hybrid search, and provides consistent single-digit millisecond queries at any scale. The trade-off is cost: it is more expensive than self-hosted options and has vendor lock-in since all data lives in Pinecone's cloud.

import pinecone

pc = pinecone.Pinecone(api_key='YOUR_API_KEY')
index = pc.Index('my-index')

# Query with metadata filter
results = index.query(
    vector=[0.1, 0.2, 0.3],
    top_k=10,
    filter={'category': {'$eq': 'finance'}},
    include_metadata=True
)

pgvector: Embeddings in PostgreSQL

pgvector extends PostgreSQL with a vector data type and approximate nearest neighbor (ANN) indexes using HNSW or IVFFlat algorithms. It is ideal when you already run PostgreSQL because your embeddings live in the same database as your relational data, enabling powerful SQL joins between vector search and structured filters with no additional infrastructure.

-- Create table with embedding column
CREATE TABLE documents (
    id SERIAL PRIMARY KEY,
    content TEXT,
    category TEXT,
    embedding vector(1536)
);

-- Create HNSW index for fast ANN search
CREATE INDEX ON documents
USING hnsw (embedding vector_cosine_ops);

-- Query nearest neighbors with SQL filter
SELECT content, 1 - (embedding <=> '[0.1,0.2,...]')
FROM documents
WHERE category = 'finance'
ORDER BY embedding <=> '[0.1,0.2,...]'
LIMIT 10;

Chroma: Developer-Friendly Local First

Chroma is an open-source embedding database designed for rapid prototyping. It runs in-process for local development (no server needed) and supports a persistent server mode for production. Chroma is popular in LangChain tutorials because of its extremely simple API, but it has limitations at scale: no distributed mode and weaker filtering than Pinecone or Qdrant.

import chromadb

client = chromadb.PersistentClient(path='./chroma_db')
collection = client.get_or_create_collection('my_docs')

# Add documents
collection.add(
    documents=['text one', 'text two'],
    metadatas=[{'source': 'doc1'}, {'source': 'doc2'}],
    ids=['id1', 'id2']
)

# Query
results = collection.query(
    query_texts=['search query'],
    n_results=5
)

Qdrant: Filtering and Payload Search

Qdrant is an open-source vector database written in Rust that excels at complex metadata filtering. Unlike databases that apply filters after ANN search, Qdrant pre-filters candidate vectors by payload fields before scoring, dramatically improving precision when filters are selective. It supports on-disk HNSW indexes, making it suitable for datasets that do not fit in RAM.

from qdrant_client import QdrantClient
from qdrant_client.models import Filter, FieldCondition, MatchValue

client = QdrantClient(url='http://localhost:6333')

# Search with payload filter
results = client.search(
    collection_name='documents',
    query_vector=[0.1, 0.2, 0.3],
    query_filter=Filter(
        must=[
            FieldCondition(
                key='category',
                match=MatchValue(value='finance')
            )
        ]
    ),
    limit=10
)

Weaviate: GraphQL and Multi-Modal

Weaviate is an open-source vector database with a unique GraphQL API and built-in support for multi-modal objects (text, images, audio). It integrates directly with embedding model providers via modules, so you can ingest raw text and let Weaviate call the embedding API automatically. This convenience comes at the cost of a more complex setup compared to Chroma or Qdrant.

import weaviate

client = weaviate.Client('http://localhost:8080')

# Near-text search using built-in vectorizer
result = client.query.get(
    'Document', ['content', 'category']
).with_near_text(
    {'concepts': ['financial analysis']}
).with_where({
    'path': ['category'],
    'operator': 'Equal',
    'valueString': 'finance'
}).with_limit(10).do()

FAISS: In-Memory at Scale

FAISS (Facebook AI Similarity Search) is a C++ library with Python bindings that provides extremely fast in-memory vector search. It is not a database (no persistence or server), but it handles billion-scale similarity search on a single machine with GPU acceleration. FAISS is the right choice for read-heavy, offline batch search workloads where you control the full stack.

import faiss
import numpy as np

dimension = 1536
vectors = np.random.random((100000, dimension)).astype('float32')

# Normalize for cosine similarity
faiss.normalize_L2(vectors)

# Build HNSW index
index = faiss.IndexHNSWFlat(dimension, 32)  # M=32 neighbors
index.add(vectors)

# Search
query = np.random.random((1, dimension)).astype('float32')
faiss.normalize_L2(query)
D, I = index.search(query, k=10)  # top-10 results

Building a Benchmark Test

The only way to choose confidently is to benchmark on your own data. A good benchmark measures: (1) indexing time for your full dataset, (2) query latency at the p50, p95, and p99 percentiles under concurrent load, (3) recall@K comparing ANN results against brute-force exact results, and (4) memory and cost at your target scale. Run the same queries against each candidate store.

import time
import numpy as np

def benchmark_store(store, queries, k=10):
    latencies = []
    for q in queries:
        start = time.perf_counter()
        store.search(q, k)
        latencies.append(time.perf_counter() - start)
    latencies.sort()
    n = len(latencies)
    print(f'p50: {latencies[n//2]*1000:.1f}ms')
    print(f'p95: {latencies[int(n*0.95)]*1000:.1f}ms')
    print(f'p99: {latencies[int(n*0.99)]*1000:.1f}ms')

Measuring Recall vs Latency Trade-off

ANN indexes trade recall for speed. A higher HNSW ef_search parameter finds more accurate neighbors but takes longer. Measure recall@10 (fraction of true top-10 neighbors returned) at different parameter settings and plot recall vs latency. Most production systems target 95-99% recall. Falling below 90% recall means users get irrelevant chunks even if queries are fast.

def compute_recall(approx_ids, exact_ids):
    '''Compute recall@K for one query'''
    return len(set(approx_ids) & set(exact_ids)) / len(exact_ids)

def benchmark_recall(index, brute_force, queries, k=10):
    recalls = []
    for q in queries:
        approx = index.search(q, k)
        exact = brute_force.search(q, k)
        recalls.append(compute_recall(approx, exact))
    print(f'Mean recall@{k}: {sum(recalls)/len(recalls):.3f}')

Decision Framework for Choosing

Use this decision tree: If you need zero infrastructure management and budget is not a constraint, choose Pinecone. If you already run PostgreSQL and your dataset is under 10 million vectors, add pgvector. For open-source self-hosted with complex payload filtering, choose Qdrant. For rapid prototyping and local development, start with Chroma and migrate later. For billion-scale offline batch jobs, use FAISS.

Quick Check

Test your understanding of AI Engineering concepts from this lesson.

Lesson Recap

In this lesson you learned: the landscape of vector stores including Pinecone, pgvector, Chroma, Qdrant, Weaviate, and FAISS, the five evaluation dimensions of latency, throughput, filtering, complexity, and cost, and a practical decision framework for choosing the right store based on your infrastructure and scale requirements. Next up we explore the problem that RAG was invented to solve.

Frequently asked questions

Is the “Choosing and Benchmarking Vector Stores” lesson free?

Yes — the full text of “Choosing and Benchmarking Vector Stores” is free to read here on the web, and the AI Engineering Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Engineering Academy course, upgrade to CoddyKit PRO.

What will I learn in “Choosing and Benchmarking Vector Stores”?

Compare Pinecone, pgvector, Chroma, Weaviate, and Qdrant across cost, latency, filtering capabilities, and operational complexity to pick the right tool for your use case. You practise AI Engineering Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Engineering Academy?

No prior experience is required. AI Engineering Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Choosing and Benchmarking Vector Stores” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Engineering Academy lesson?

Yes. Every AI Engineering Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Why You Need a Vector Database
  2. Getting Started with Pinecone
  3. pgvector: Embeddings in PostgreSQL
  4. Choosing and Benchmarking Vector Stores
← Back to AI Engineering Academy