LangChain / RAG / Vector DBs · درس

البحث الهجين وإعادة الترتيب

اجمع بين البحث بالكلمات المفتاحية والبحث الدلالي (البحث الهجين)، واستخدم نماذج إعادة الترتيب لمنح الأولوية للمستندات الأكثر صلة.

البحث الهجين وإعادة الترتيب درس مجاني في LangChain / RAG / Vector DBs على CoddyKit. هذا هو الدرس 3 من أصل 4. يمكنك قراءة الدرس كاملاً أدناه مجاناً — ثم تمرن عليه مباشرة في المتصفح باستخدام محرر أكواد مدمج ومدرس ذكاء اصطناعي متاح 24/7. هذا الدرس جزء من مسار التعلم في LangChain / RAG / Vector DBs، وتقدمك يتزامن عبر الويب وتطبيق CoddyKit. تتضمن دورة LangChain / RAG / Vector DBs 4 دروس في المجموع.

بعض أجزاء هذا الدرس لم تُترجم بعد وتظهر باللغة الإنجليزية.

Beyond Basic Retrieval

When building advanced Retrieval Augmented Generation (RAG) systems, simply finding documents isn't enough. We need to find the most relevant documents efficiently.

Traditional keyword or semantic searches, while powerful, each have limitations. To overcome these, we can combine their strengths.

Keyword Search: Specificity

Keyword search (also known as sparse retrieval, e.g., using BM25 or TF-IDF) is excellent for finding exact matches and specific terms.

Strengths: Great for precise queries, proper nouns, and when you know the exact wording.
Weaknesses: Struggles with synonyms, different phrasing, or understanding conceptual meaning.

For example, searching 'Python list append' works well, but 'add element to Python array' might miss results.

Semantic Search: Understanding Meaning

Semantic search (dense retrieval, using embeddings) understands the meaning and context of your query and documents.

Strengths: Handles synonyms, rephrased questions, and conceptual searches effectively.
Weaknesses: Can struggle with very specific, rare terms or highly technical jargon if not well-represented in its embedding space.

It can understand 'add element to Python array' is similar to 'Python list append'.

Introducing Hybrid Search

Hybrid search combines the best of both worlds: the precision of keyword search and the contextual understanding of semantic search.

By running both types of retrieval and intelligently merging their results, hybrid search can significantly improve the relevance and completeness of retrieved documents for your RAG system.

Merging Results: Reciprocal Rank Fusion

A common method to combine results from multiple retrievers in hybrid search is Reciprocal Rank Fusion (RRF).

RRF assigns a score to each document based on its rank in each individual retriever's result list. Documents that appear high in multiple lists get a boosted score, leading to a more robust final ranking.

Try running this simplified example of RRF:

def reciprocal_rank_fusion(rank_lists, k=60):
    fused_scores = {}
    for rank_list in rank_lists:
        for rank, doc_id in enumerate(rank_list):
            if doc_id not in fused_scores:
                fused_scores[doc_id] = 0.0
            fused_scores[doc_id] += 1.0 / (k + rank + 1)
    
    sorted_docs = sorted(fused_scores.items(), key=lambda item: item[1], reverse=True)
    return [doc_id for doc_id, score in sorted_docs]

if __name__ == "__main__":
    # Simulate results from two retrievers
    keyword_results = ["docA", "docC", "docB", "docE"]
    semantic_results = ["docB", "docA", "docD", "docC"]

    fused_order = reciprocal_rank_fusion([keyword_results, semantic_results])
    print("Fused Order:", fused_order)

Hybrid Search with LangChain

LangChain provides an EnsembleRetriever to easily implement hybrid search. It takes multiple retrievers (e.g., a keyword retriever and a vector store retriever) and combines their results, often using RRF by default.

This allows you to leverage both precise keyword matches and semantic understanding in one powerful retrieval step.

# from langchain.retrievers import EnsembleRetriever
# from langchain_community.retrievers import BM25Retriever
# from langchain_community.vectorstores import FAISS
# from langchain_openai import OpenAIEmbeddings

# # Assume you have a BM25 retriever and a vector store retriever
# bm25_retriever = BM25Retriever.from_documents(docs)
# vectorstore = FAISS.from_documents(docs, OpenAIEmbeddings())
# vectorstore_retriever = vectorstore.as_retriever()

# ensemble_retriever = EnsembleRetriever(retrievers=[
#     bm25_retriever, 
#     vectorstore_retriever
# ], weights=[0.5, 0.5])

# # query = "What are the capital cities of Europe?"
# # docs = ensemble_retriever.invoke(query)

The Need for Re-ranking

Even after hybrid search, the initial set of retrieved documents might contain some noise or documents that are not perfectly ordered by relevance.

Re-ranking is a crucial next step. It takes the top-k documents from the initial retrieval and re-evaluates their relevance to the query using a more sophisticated model.

How Re-ranking Works

A re-ranking model, often a cross-encoder, takes both the user query and each retrieved document as input.

Unlike embedding models that create separate embeddings, a cross-encoder jointly processes the query and document to generate a single relevance score. This allows for a more nuanced understanding of their interaction.

Integrating Re-rankers

Various re-ranking models and services are available, such as Cohere's Re-rank API or open-source cross-encoders from libraries like sentence-transformers.

Integrating a re-ranker typically involves passing the initial retrieval results and the query to the re-ranker, which then returns the documents in a new, optimized order.

# from langchain.retrievers.document_compressors import CohereRerank
# from langchain.retrievers import ContextualCompressionRetriever

# # Assume you have an existing base_retriever (e.g., your EnsembleRetriever)
# # base_retriever = ensemble_retriever

# # Initialize the Cohere Rerank compressor
# # cohere_re_ranker = CohereRerank(top_n=5, cohere_api_key="YOUR_COHERE_API_KEY")

# # Create a compression retriever that uses the re-ranker
# # compression_retriever = ContextualCompressionRetriever(
# #     base_compressor=cohere_re_ranker,
# #     base_retriever=base_retriever
# # )

# # query = "What is the capital of France?"
# # compressed_docs = compression_retriever.invoke(query)

The Full Advanced Retrieval Pipeline

By combining hybrid search and re-ranking, you create a robust retrieval pipeline:

User Query
Hybrid Search: Combines keyword and semantic retrieval to get an initial set of relevant documents.
Re-ranking: A specialized model re-orders the top documents from hybrid search for maximum relevance.
Context for LLM: The highly relevant, re-ranked documents are passed to the LLM for generation.

This approach leads to more accurate and contextually rich answers from your RAG system.

Quick Check: Retrieval Steps

Which of the following statements accurately describe the roles of Hybrid Search and Re-ranking in a RAG system?

Recap: Advanced Retrieval

We've explored advanced retrieval techniques to boost RAG performance:

Hybrid Search: Combines keyword and semantic approaches for comprehensive initial retrieval.
Reciprocal Rank Fusion (RRF): A method to merge and score results from different retrievers.
Re-ranking: Uses specialized models (like cross-encoders) to refine the relevance order of retrieved documents.

These techniques ensure your RAG system provides the most accurate and contextually relevant information to the LLM.

الأسئلة الشائعة

هل درس «البحث الهجين وإعادة الترتيب» مجاني؟

نعم — نص درس «البحث الهجين وإعادة الترتيب» كامل متاح مجاناً هنا على الويب. لتمرينه بشكل تفاعلي (محرر أكواد مدمج ومدرس ذكاء اصطناعي متاح 24/7) وفتح باقي دورة LangChain / RAG / Vector DBs، انتقل إلى CoddyKit PRO. تتضمن دورة LangChain / RAG / Vector DBs 4 دروس في المجموع.

ماذا ستتعلم في «البحث الهجين وإعادة الترتيب»؟

اجمع بين البحث بالكلمات المفتاحية والبحث الدلالي (البحث الهجين)، واستخدم نماذج إعادة الترتيب لمنح الأولوية للمستندات الأكثر صلة. تتمرن على LangChain / RAG / Vector DBs مع أكواد عملية تشغلها مباشرة في المتصفح، ومدرس ذكاء اصطناعي متاح 24/7 يجيب على أسئلتك أثناء عملك.

هل أحتاج إلى خبرة سابقة لأبدأ LangChain / RAG / Vector DBs؟

لا تُشترط خبرة سابقة. LangChain / RAG / Vector DBs على CoddyKit منظم للمبتدئين حتى المتقدمين، لذا يمكنك البدء من هنا أو من البداية والتقدم بسرعتك الخاصة. هذا هو الدرس 3 من أصل 4.

كم من الوقت يستغرق درس «البحث الهجين وإعادة الترتيب»؟

معظم دروس CoddyKit تستغرق حوالي 5–10 دقائق. كل منها موجز وتفاعلي، لذا تحرز تقدماً مستمراً وتستأنف من حيث توقفت عبر الويب والتطبيق.

هل يمكنني كتابة وتشغيل أكواد في درس LangChain / RAG / Vector DBs هذا؟

نعم. كل درس في LangChain / RAG / Vector DBs يتضمن محرر أكواد مدمج، لذا تكتب وتشغل أكواداً حقيقية مباشرة في متصفحك وتحصل على تعليقات فورية من الذكاء الاصطناعي — بدون إعداد محلي.

جميع الدروس في هذه الدورة

← العودة إلى LangChain / RAG / Vector DBs