AI Engineering Academy · Lesson

Document-Specific Strategies for Code and HTML

Apply specialized chunking for Python code using AST-based function splitters, for HTML using tag-aware parsers, and for Markdown using header hierarchy.

Document-Specific Strategies for Code and HTML is a free AI Engineering Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Engineering Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Generic Chunking Fails Specialized Docs

Text-based chunking was designed for prose, but real-world data includes source code, HTML pages, and Markdown documentation. Splitting code at a fixed character boundary can sever a function in the middle of its body, making the chunk useless for retrieval. Specialized documents need chunkers that understand their internal structure, not just their length.

AST-Based Python Code Chunking

The Abstract Syntax Tree (AST) of a Python file captures every function, class, and module as a structured node. By walking the AST you can extract each function or method as its own chunk, keeping the signature, docstring, and body together. LangChain's PythonCodeTextSplitter uses this approach internally.

import ast
import textwrap

def extract_functions(source_code: str) -> list[dict]:
    tree = ast.parse(source_code)
    chunks = []
    for node in ast.walk(tree):
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
            start = node.lineno - 1
            end = node.end_lineno
            lines = source_code.splitlines()[start:end]
            chunks.append({
                'name': node.name,
                'code': '\n'.join(lines),
                'start_line': node.lineno,
            })
    return chunks

Chunking by Class Boundaries

For object-oriented codebases, chunking at the class level is often better than at the function level. A class chunk retains the relationship between methods and the shared state they operate on. You can include the class docstring and all method bodies as a single chunk, then create separate finer-grained chunks for long methods only.

from langchain_text_splitters import Language, RecursiveCharacterTextSplitter

python_splitter = RecursiveCharacterTextSplitter.from_language(
    language=Language.PYTHON,
    chunk_size=1000,
    chunk_overlap=100,
)

with open('my_module.py', 'r') as f:
    source = f.read()

chunks = python_splitter.create_documents([source])
print(f'Created {len(chunks)} code chunks')

Adding Code Metadata to Chunks

Raw code chunks are only as useful as their metadata. When storing code chunks in a vector database, include the file path, function name, programming language, and line range. This metadata allows the retriever to filter by language or file, and the LLM to cite the exact source location in its answer.

from langchain_core.documents import Document

def chunk_python_file(filepath: str) -> list[Document]:
    with open(filepath) as f:
        source = f.read()

    functions = extract_functions(source)  # from previous example
    docs = []
    for fn in functions:
        docs.append(Document(
            page_content=fn['code'],
            metadata={
                'source': filepath,
                'function': fn['name'],
                'language': 'python',
                'start_line': fn['start_line'],
            }
        ))
    return docs

HTML: Structure Over Characters

HTML documents are hierarchically structured with headings, sections, paragraphs, and lists. Splitting HTML by character count often cuts across tags, producing malformed fragments. The right approach is to parse the HTML with a proper parser like BeautifulSoup and extract semantically meaningful elements such as <article>, <section>, and <p> tags.

from bs4 import BeautifulSoup

def chunk_html_by_section(html: str) -> list[dict]:
    soup = BeautifulSoup(html, 'html.parser')
    chunks = []
    for tag in soup.find_all(['h1', 'h2', 'h3', 'p', 'li']):
        text = tag.get_text(separator=' ', strip=True)
        if len(text) > 40:  # skip trivial fragments
            chunks.append({
                'tag': tag.name,
                'text': text,
            })
    return chunks

Header-Hierarchical HTML Chunking

A more sophisticated HTML strategy groups content under its nearest heading. Every paragraph and list that follows an <h2> header belongs to that section. By grouping text with its parent heading, you preserve the topic context that a standalone paragraph would otherwise lose. LangChain's HTMLHeaderTextSplitter implements this automatically.

from langchain_text_splitters import HTMLHeaderTextSplitter

headers_to_split_on = [
    ('h1', 'Header 1'),
    ('h2', 'Header 2'),
    ('h3', 'Header 3'),
]

splitter = HTMLHeaderTextSplitter(headers_to_split_on=headers_to_split_on)

with open('page.html') as f:
    html = f.read()

sections = splitter.split_text(html)
for sec in sections[:3]:
    print(sec.metadata)
    print(sec.page_content[:200])
    print('---')

Markdown: Respecting Heading Hierarchy

Markdown documentation is organized with #, ##, and ### headings. The MarkdownHeaderTextSplitter splits at heading boundaries and stores the heading hierarchy in metadata. This means every chunk knows its full heading path, which greatly improves the relevance of retrieved context when users ask about specific sections of documentation.

from langchain_text_splitters import MarkdownHeaderTextSplitter

headers_to_split_on = [
    ('#', 'H1'),
    ('##', 'H2'),
    ('###', 'H3'),
]

md_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split_on)

with open('README.md') as f:
    markdown = f.read()

docs = md_splitter.split_text(markdown)
for doc in docs[:2]:
    print('Metadata:', doc.metadata)
    print('Content:', doc.page_content[:300])
    print()

Secondary Splitting After Header Split

After splitting by heading, individual sections may still be too long for your embedding model's token limit. The recommended pattern is a two-step split: first split by heading hierarchy to preserve semantic context, then apply a character-based splitter to any section that exceeds your chunk size limit. This ensures no chunk is too large while preserving heading metadata.

from langchain_text_splitters import MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter

header_splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=[('#', 'H1'), ('##', 'H2')]
)
char_splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,
)

with open('docs.md') as f:
    md = f.read()

header_chunks = header_splitter.split_text(md)
final_chunks = char_splitter.split_documents(header_chunks)
print(f'{len(final_chunks)} final chunks produced')

Chunking PDFs with Table Awareness

PDFs extracted with tools like PyMuPDF or pdfplumber often lose table structure, producing garbled rows of text. To handle this, use layout-aware PDF parsers that detect table bounding boxes and convert them to Markdown or CSV format before chunking. Treat each table as a single chunk with structured metadata identifying it as a table rather than prose.

import pdfplumber

def extract_pdf_chunks(pdf_path: str) -> list[dict]:
    chunks = []
    with pdfplumber.open(pdf_path) as pdf:
        for page_num, page in enumerate(pdf.pages):
            # Extract tables separately
            for table in page.extract_tables():
                rows = ['|'.join(str(c) for c in row) for row in table]
                chunks.append({
                    'type': 'table',
                    'content': '\n'.join(rows),
                    'page': page_num + 1,
                })
            # Extract prose text
            text = page.extract_text()
            if text:
                chunks.append({'type': 'text', 'content': text, 'page': page_num + 1})
    return chunks

Language Detection for Mixed Corpora

Enterprise knowledge bases often mix different file types: Python scripts, API documentation in HTML, architecture notes in Markdown, and data exports in CSV. A robust chunking pipeline should detect the file type from the extension or MIME type and route each document to the appropriate specialist chunker. This avoids applying code splitting logic to prose or vice versa.

from pathlib import Path

def route_document(filepath: str) -> list[dict]:
    ext = Path(filepath).suffix.lower()
    if ext == '.py':
        return chunk_python_file(filepath)
    elif ext in ('.html', '.htm'):
        with open(filepath) as f:
            return chunk_html_by_section(f.read())
    elif ext == '.md':
        # use MarkdownHeaderTextSplitter
        return chunk_markdown(filepath)
    elif ext == '.pdf':
        return extract_pdf_chunks(filepath)
    else:
        # fallback: plain text recursive splitter
        return chunk_plain_text(filepath)

Preserving Context with Surrounding Lines

When chunking code by function, it is often valuable to include a few lines of surrounding context, such as import statements at the top of the file or the class definition that contains a method. This context helps the LLM understand what libraries are available and what the function's role is within the broader class, improving the quality of generated answers.

def chunk_with_imports(source_code: str, fn_node, lines: list[str]) -> str:
    # Gather top-of-file imports (first block before first non-import)
    import_lines = []
    for line in lines:
        stripped = line.strip()
        if stripped.startswith('import ') or stripped.startswith('from '):
            import_lines.append(line)
        elif stripped and not stripped.startswith('#'):
            break

    fn_body = '\n'.join(lines[fn_node.lineno - 1:fn_node.end_lineno])
    return '\n'.join(import_lines) + '\n\n' + fn_body

Quick Check

Test your understanding of document-specific chunking strategies from this lesson.

Lesson Recap

In this lesson you learned: AST-based chunking preserves Python function and class boundaries, HTMLHeaderTextSplitter and MarkdownHeaderTextSplitter respect heading hierarchy to keep context with its section, and a two-step approach (heading split followed by character split) handles oversized sections without losing structural metadata. Next up we explore hybrid search combining dense and sparse retrieval.

Frequently asked questions

Is the “Document-Specific Strategies for Code and HTML” lesson free?

Yes — the full text of “Document-Specific Strategies for Code and HTML” is free to read here on the web, and the AI Engineering Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Engineering Academy course, upgrade to CoddyKit PRO.

What will I learn in “Document-Specific Strategies for Code and HTML”?

Apply specialized chunking for Python code using AST-based function splitters, for HTML using tag-aware parsers, and for Markdown using header hierarchy. You practise AI Engineering Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Engineering Academy?

No prior experience is required. AI Engineering Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Document-Specific Strategies for Code and HTML” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Engineering Academy lesson?

Yes. Every AI Engineering Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

Why Naive Chunking Hurts Retrieval
Semantic Chunking with Embedding Similarity
Parent-Child and Small-to-Big Retrieval
Document-Specific Strategies for Code and HTML

← Back to AI Engineering Academy