Directory Traversal and File Discovery
os.walk(), glob patterns, and filtering files by type or date.
Directory Traversal and File Discovery is a free AI Agents lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Agents learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Why Agents Need Directory Traversal
Agents often need to find files matching certain criteria — all Python files in a codebase, all CSV files in a data directory, or all recent logs. Python provides three main tools: os.walk(), Path.iterdir(), and Path.rglob(). Each has different strengths depending on how deep you need to search.
import os
from pathlib import Path
# Count all files in a directory tree
root = Path('/tmp/agent_workspace')
file_count = sum(1 for f in root.rglob('*') if f.is_file())
print(f'Found {file_count} files under {root}')os.walk() — Recursive Directory Traversal
os.walk(root) yields a tuple of (dirpath, dirnames, filenames) for every directory in the tree. It's the classic Python approach for recursive traversal and gives you fine-grained control over which subdirectories to descend into.
import os
import tempfile
# --- demo setup: build a small project tree so the walk has something to show ---
root = tempfile.mkdtemp(prefix='project_')
os.makedirs(os.path.join(root, 'src'), exist_ok=True)
os.makedirs(os.path.join(root, '.git'), exist_ok=True)
with open(os.path.join(root, 'README.md'), 'w') as f:
f.write('demo')
with open(os.path.join(root, 'src', 'main.py'), 'w') as f:
f.write('print(1)')
for dirpath, dirnames, filenames in os.walk(root):
# Skip hidden directories (like .git)
dirnames[:] = [d for d in dirnames if not d.startswith('.')]
print(f'In directory: {dirpath}')
print(f' Subdirs: {dirnames}')
print(f' Files: {filenames}')
for filename in filenames:
full_path = os.path.join(dirpath, filename)
print(f' File: {full_path}')
Path.iterdir() — Single Level Listing
Path.iterdir() lists the immediate contents of a directory — it does not recurse into subdirectories. Use it when you only need the direct children of a directory and want to avoid descending deeper.
from pathlib import Path
data_dir = Path('data')
data_dir.mkdir(exist_ok=True)
(data_dir / 'a.csv').write_text('x')
(data_dir / 'sub').mkdir(exist_ok=True)
for item in data_dir.iterdir():
if item.is_file():
print(f'FILE: {item.name} ({item.stat().st_size} bytes)')
elif item.is_dir():
print(f'DIR: {item.name}/')
csv_files = [f for f in data_dir.iterdir()
if f.is_file() and f.suffix == '.csv']
print(f'Found {len(csv_files)} CSV files')Path.rglob() — Recursive Glob
Path.rglob(pattern) recursively finds all files matching a glob pattern. It's the most concise way to find all files of a given type anywhere in a directory tree. The ** in glob means "any number of directories".
from pathlib import Path
project = Path('/tmp/my_project')
# Find all Python files recursively
python_files = list(project.rglob('*.py'))
print(f'Found {len(python_files)} Python files:')
for f in python_files:
print(f' {f.relative_to(project)}')
# Find all JSON files
json_files = list(project.rglob('*.json'))
# Find files matching a name pattern
test_files = list(project.rglob('test_*.py'))
print(f'Found {len(test_files)} test files')glob.glob() — Pattern Matching
glob.glob(pattern, recursive=True) is the traditional glob interface. With recursive=True, the ** wildcard matches any subdirectory path. It returns strings, while pathlib.rglob returns Path objects.
import glob
# Find all CSV files recursively
csv_files = glob.glob('data/**/*.csv', recursive=True)
print(f'Found {len(csv_files)} CSV files')
for f in csv_files:
print(f' {f}')
# Find all log files in a specific directory (non-recursive)
logs = glob.glob('/var/log/*.log')
# Find files matching multiple criteria
import fnmatch
all_files = glob.glob('/tmp/**/*', recursive=True)
config_files = [
f for f in all_files
if fnmatch.fnmatch(f, '*.yaml') or fnmatch.fnmatch(f, '*.yml')
]
print(f'Found {len(config_files)} YAML config files')Filtering by File Extension
When traversing directories, filter files by their extension using Path.suffix (which includes the dot, e.g., '.py') or by checking against a set of allowed extensions. This is faster and more precise than glob patterns for multi-extension filtering.
from pathlib import Path
directory = Path('/tmp/mixed_files')
ALLOWED_EXTENSIONS = {'.py', '.js', '.ts', '.json', '.yaml'}
# Filter by allowed extensions
code_files = [
f for f in directory.rglob('*')
if f.is_file() and f.suffix.lower() in ALLOWED_EXTENSIONS
]
print(f'Found {len(code_files)} code/config files')
# Group files by extension
from collections import defaultdict
by_ext = defaultdict(list)
for f in directory.rglob('*'):
if f.is_file():
by_ext[f.suffix].append(f)
for ext, files in sorted(by_ext.items()):
print(f'{ext}: {len(files)} files')Filtering by File Size
Agents sometimes need to find files above or below a size threshold — skip tiny empty files or avoid processing huge files that would overwhelm memory. Use Path.stat().st_size to get file size in bytes.
from pathlib import Path
data_dir = Path('/tmp/data')
MIN_SIZE = 1024 # 1 KB — skip empty/tiny files
MAX_SIZE = 50_000_000 # 50 MB — skip very large files
processable = []
skipped_small = []
skipped_large = []
for f in data_dir.rglob('*.json'):
if not f.is_file():
continue
size = f.stat().st_size
if size < MIN_SIZE:
skipped_small.append(f.name)
elif size > MAX_SIZE:
skipped_large.append(f.name)
else:
processable.append(f)
print(f'Processable: {len(processable)}')
print(f'Too small: {len(skipped_small)}')
print(f'Too large: {len(skipped_large)}')Filtering by Modification Time
Agents in incremental pipelines only need to process files that changed since the last run. Use stat().st_mtime (modification time as a Unix timestamp) to find recently modified files.
from pathlib import Path
import time
import datetime
data_dir = Path('/tmp/data')
# Files modified in the last 24 hours
cutoff = time.time() - (24 * 60 * 60)
recent_files = [
f for f in data_dir.rglob('*.log')
if f.is_file() and f.stat().st_mtime > cutoff
]
print(f'Modified in last 24h: {len(recent_files)} files')
# Sort by modification time (newest first)
recent_files.sort(key=lambda f: f.stat().st_mtime, reverse=True)
for f in recent_files[:5]: # top 5 most recent
mtime = datetime.datetime.fromtimestamp(f.stat().st_mtime)
print(f' {f.name}: {mtime.strftime("%Y-%m-%d %H:%M:%S")}')Skipping Hidden Files and System Directories
When traversing code repositories or user directories, skip hidden files (starting with .) and system directories like .git, __pycache__, node_modules, and .venv. These contain thousands of files irrelevant to most agents.
from pathlib import Path
SKIP_DIRS = {'.git', '__pycache__', 'node_modules', '.venv',
'.env', 'dist', 'build', '.mypy_cache'}
def iter_source_files(root, extensions):
root = Path(root)
for path in root.rglob('*'):
# Skip if any parent is in SKIP_DIRS
if any(part in SKIP_DIRS for part in path.parts):
continue
# Skip hidden files
if path.name.startswith('.'):
continue
if path.is_file() and path.suffix in extensions:
yield path
python_files = list(iter_source_files('/tmp/project', {'.py'}))
print(f'Found {len(python_files)} Python source files')Building a File Inventory
A file inventory is a structured summary of a directory tree — useful for agents that need to plan work before processing. Collect name, path, size, extension, and modification time into a list of dicts that can be serialized to JSON.
from pathlib import Path
import datetime
import json
def build_inventory(root_dir, pattern='**/*'):
root = Path(root_dir)
inventory = []
for f in root.glob(pattern):
if not f.is_file():
continue
stat = f.stat()
inventory.append({
'name': f.name,
'path': str(f.relative_to(root)),
'extension': f.suffix,
'size_bytes': stat.st_size,
'modified': datetime.datetime.fromtimestamp(
stat.st_mtime
).isoformat()
})
return sorted(inventory, key=lambda x: x['path'])
inventory = build_inventory('/tmp/data')
with open('file_inventory.json', 'w') as out:
json.dump(inventory, out, indent=2)
print(f'Inventoried {len(inventory)} files')Watching a Directory for Changes
Some agents need to react to new files appearing in a directory. While a full file-watcher library like watchdog is best for production, a simple approach is to scan the directory periodically and compare against a known set of files.
from pathlib import Path
import time
def watch_for_new_files(watch_dir, extension='.json', interval=5):
watch_dir = Path(watch_dir)
known_files = set(watch_dir.glob(f'*{extension}'))
print(f'Watching {watch_dir} for new {extension} files...')
while True:
time.sleep(interval)
current_files = set(watch_dir.glob(f'*{extension}'))
new_files = current_files - known_files
for f in new_files:
print(f'New file detected: {f.name}')
process_new_file(f) # handle the new file
known_files = current_files
def process_new_file(file_path):
print(f'Processing: {file_path}')
# --- demo (one polling cycle, without the infinite while True loop) ---
import tempfile
demo_dir = Path(tempfile.mkdtemp())
known_files = set(demo_dir.glob('*.json'))
print(f'Watching {demo_dir} for new .json files...')
(demo_dir / 'result_a.json').write_text('{}', encoding='utf-8')
(demo_dir / 'result_b.json').write_text('{}', encoding='utf-8')
current_files = set(demo_dir.glob('*.json'))
new_files = current_files - known_files
for f in sorted(new_files, key=lambda p: p.name):
print(f'New file detected: {f.name}')
process_new_file(f)
Quick Check: rglob vs iterdir
Test your understanding of directory traversal methods.
Directory Traversal Recap
You can now find any file your agent needs:
- os.walk() — full control over recursive traversal, modify
dirnamesin-place to prune subdirectories - Path.iterdir() — single-level listing of immediate children
- Path.rglob('*.ext') — cleanest recursive search by pattern
- glob.glob('**/*.ext', recursive=True) — traditional alternative returning strings
- Filter by
.suffixfor extension,.stat().st_sizefor size,.stat().st_mtimefor modification time - Always skip hidden files and system directories like
.gitandnode_modules
Frequently asked questions
Is the “Directory Traversal and File Discovery” lesson free?
Yes — the full text of “Directory Traversal and File Discovery” is free to read here on the web, and the AI Agents course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Agents course, upgrade to CoddyKit PRO.
What will I learn in “Directory Traversal and File Discovery”?
os.walk(), glob patterns, and filtering files by type or date. You practise AI Agents with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start AI Agents?
No prior experience is required. AI Agents on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Directory Traversal and File Discovery” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this AI Agents lesson?
Yes. Every AI Agents lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Reading and Writing Files in Agent Context
- Directory Traversal and File Discovery
- File Format Handling: CSV, JSON, and TXT
- Safe File Operations with Error Handling