Memuat Beragam Format Dokumen
Jelajahi metode untuk memasukkan data dari berbagai sumber seperti PDF, halaman web, basis data, dan jenis berkas khusus.
Memuat Beragam Format Dokumen adalah pelajaran LLM Apps in Production (RAG + Vector DB + Caching) gratis di CoddyKit. Ini adalah pelajaran 1 dari 4. Kamu bisa membaca pelajaran lengkapnya di bawah secara gratis — lalu praktikkan langsung di browser dengan editor kode bawaan dan tutor AI 24/7. Ini adalah bagian dari jalur belajar LLM Apps in Production (RAG + Vector DB + Caching), dan progresmu tersinkronisasi di web dan aplikasi CoddyKit. Kursus LLM Apps in Production (RAG + Vector DB + Caching) mencakup 4 pelajaran total.
Bagian dari pelajaran ini belum diterjemahkan dan ditampilkan dalam bahasa Inggris.
Ingesting Diverse Document Types
Welcome! In RAG, your LLM needs information from various sources. This lesson explores how to load data from different document formats into your application.
The goal is to get raw text from places like web pages, PDFs, and databases, preparing it for the next steps in your RAG pipeline.
Loading Web Pages (HTML)
Web pages are a common source of information. To ingest them, you typically:
- Fetch the HTML: Use an HTTP client to download the page content from a URL.
- Parse the HTML: Extract the main text and discard navigation, ads, and other irrelevant elements.
Libraries like requests for fetching and BeautifulSoup for parsing are very popular in Python.
Web Page Loading Example
This Python snippet demonstrates fetching a simple web page and extracting its title. Imagine doing this for many pages to build your knowledge base!
import requests
from bs4 import BeautifulSoup
def main():
url = "http://quotes.toscrape.com/"
try:
response = requests.get(url)
response.raise_for_status() # Raise HTTPError for bad responses
soup = BeautifulSoup(response.text, 'html.parser')
title = soup.find('title').get_text()
print(f"Page Title: {title}")
except requests.exceptions.RequestException as e:
print(f"Error fetching URL: {e}")
if __name__ == "__main__":
main()Extracting Text from PDFs
PDFs (Portable Document Format) are widely used for reports and documents. Extracting text from PDFs can be tricky due to their complex structure, which combines text, images, and formatting.
Fortunately, programming libraries exist to help. They can read the PDF structure and pull out the textual content, often page by page.
PDF Text Extraction Example
This conceptual Python code shows how you might extract text from the first page of a PDF using a library like pypdf. For a real run, you'd need a sample.pdf file.
from pypdf import PdfReader
def main():
# In a real scenario, 'sample.pdf' would exist
# For this example, we'll simulate the output
pdf_file_path = "sample.pdf"
print(f"Attempting to read from: {pdf_file_path}")
print("\n--- Simulated PDF Content ---")
print("This is some text from the first page of a sample PDF document.")
print("It contains important information for our RAG system.")
print("-----------------------------")
# Actual code might look like this:
# reader = PdfReader(pdf_file_path)
# page = reader.pages[0]
# text = page.extract_text()
# print(text)
if __name__ == "__main__":
main()Loading Data from Databases
Databases, both SQL (like PostgreSQL, MySQL) and NoSQL (like MongoDB, Cassandra), store structured data that can be valuable for RAG.
To ingest from databases:
- Connect: Establish a connection using database drivers.
- Query: Write queries (e.g., SQL statements) to retrieve relevant data.
- Process: Extract text fields from the query results.
Database Loading Example
Here's a Python example using SQLite, an embedded SQL database. It creates a simple table, inserts data, and then retrieves it. This is a common pattern for database ingestion.
import sqlite3
def main():
# Connect to an in-memory SQLite database
conn = sqlite3.connect(':memory:')
cursor = conn.cursor()
# Create a simple table
cursor.execute('''
CREATE TABLE IF NOT EXISTS documents (
id INTEGER PRIMARY KEY,
title TEXT,
content TEXT
)
''')
# Insert some data
cursor.execute("INSERT INTO documents (title, content) VALUES (?, ?)",
("RAG Overview", "RAG enhances LLMs by retrieving relevant docs."))
cursor.execute("INSERT INTO documents (title, content) VALUES (?, ?)",
("Vector DBs", "Store embeddings for fast similarity search."))
conn.commit()
# Retrieve data
cursor.execute("SELECT title, content FROM documents")
rows = cursor.fetchall()
print("--- Retrieved Documents ---")
for row in rows:
print(f"Title: {row[0]}, Content: {row[1]}")
print("---------------------------")
conn.close()
if __name__ == "__main__":
main()Handling Plain Text & Custom Files
Beyond specific formats, you'll often deal with plain text files (.txt, .md) or custom formats (e.g., CSV, JSON). For these:
- Plain Text: Read directly, paying attention to encoding (UTF-8 is common).
- Custom Formats: Use libraries specific to the format (e.g.,
csv,jsonmodules in Python) to parse and extract text fields.
The key is transforming the data into a usable text string.
Unified Data Loading with Libraries
For complex RAG systems, you don't always need to write custom loaders for every format. Libraries like LlamaIndex and LangChain offer 'Document Loaders' that abstract away much of this complexity.
- They provide connectors for many data sources (web, PDF, databases, cloud storage).
- They often handle basic parsing and text extraction automatically.
These tools simplify the initial ingestion step, letting you focus on retrieval and generation.
Check Your Knowledge
Which of the following is typically a primary challenge when extracting text content from PDF documents for a RAG system?
Recap: Loading Diverse Data
Great job! You've learned the fundamentals of loading diverse document formats for your RAG system.
- We covered fetching and parsing web pages.
- Discussed extracting text from complex PDFs.
- Explored querying databases for structured content.
- Touched upon handling plain text and other custom files.
- Recognized the value of unified data loading libraries.
The next step is to prepare this raw text for effective retrieval!
Belajar LLM Apps in Production (RAG + Vector DB + Caching) dengan tutor AI — gratis
Tulis dan jalankan kode asli di browser kamu, dapatkan bantuan instan dari tutor AI 24/7, dan lanjutkan di mana kamu tinggalkan di web atau aplikasi.
- Kursus
- 12
- Pelajaran
- 48
Pertanyaan yang Sering Diajukan
Apakah pelajaran “Memuat Beragam Format Dokumen” gratis?
Ya — teks lengkap “Memuat Beragam Format Dokumen” gratis dibaca di sini di web. Untuk praktiknya secara interaktif (editor kode bawaan dan tutor AI 24/7) dan buka sisa kursus LLM Apps in Production (RAG + Vector DB + Caching), upgrade ke CoddyKit PRO. Kursus LLM Apps in Production (RAG + Vector DB + Caching) mencakup 4 pelajaran total.
Apa yang akan aku pelajari di “Memuat Beragam Format Dokumen”?
Jelajahi metode untuk memasukkan data dari berbagai sumber seperti PDF, halaman web, basis data, dan jenis berkas khusus. Kamu berlatih LLM Apps in Production (RAG + Vector DB + Caching) dengan kode praktik yang langsung kamu jalankan di browser, dan tutor AI 24/7 menjawab pertanyaanmu saat kamu mengerjakan pelajaran ini.
Apakah aku perlu pengalaman untuk memulai LLM Apps in Production (RAG + Vector DB + Caching)?
Tidak diperlukan pengalaman sebelumnya. LLM Apps in Production (RAG + Vector DB + Caching) di CoddyKit dirancang untuk pemula hingga pelajar tingkat lanjut, jadi kamu bisa memulai di sini atau dari awal dan belajar sesuai kecepatan kamu sendiri. Ini adalah pelajaran 1 dari 4.
Berapa lama pelajaran “Memuat Beragam Format Dokumen” memakan waktu?
Sebagian besar pelajaran CoddyKit memakan waktu sekitar 5–10 menit. Setiap pelajaran ringkas dan interaktif, jadi kamu membuat kemajuan stabil dan melanjutkan dari tempat kamu tinggalkan di web dan aplikasi.
Bisakah aku menulis dan menjalankan kode dalam pelajaran LLM Apps in Production (RAG + Vector DB + Caching) ini?
Ya. Setiap pelajaran LLM Apps in Production (RAG + Vector DB + Caching) menyertakan editor kode bawaan, jadi kamu menulis dan menjalankan kode nyata langsung di browser dan mendapatkan umpan balik AI instan — tidak diperlukan penyiapan lokal.
Semua pelajaran dalam kursus ini
- Memuat Beragam Format Dokumen
- Strategi Pemenggalan yang Sadar Konteks
- Pengelolaan dan Penyaringan Metadata
- Membersihkan dan Menghapus Duplikasi Data Sumber