AI Agents · Lesson

Running Local Models with Ollama and llama.cpp

Ollama makes it 'docker run' for LLMs; llama.cpp goes deeper with quantization and direct C++ control.

Running Local Models with Ollama and llama.cpp is a free AI Agents lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Agents learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Two Easy Paths

For self-hosting open models on a workstation or server:

Ollama — simplest, Docker-like UX for LLMs
llama.cpp — closer to the metal, more control, smaller footprint

Ollama Quick Start

# Install (macOS):
brew install ollama

# Pull a model:
ollama pull llama3.1:8b

# Run interactively:
ollama run llama3.1:8b 'Hello'

# Serve via HTTP (OpenAI-compatible):
ollama serve

Calling Ollama Via SDK

from openai import OpenAI
client = OpenAI(base_url='http://localhost:11434/v1', api_key='ollama')

response = client.chat.completions.create(
    model='llama3.1:8b',
    messages=[{'role': 'user', 'content': 'Hello'}]
)
print(response.choices[0].message.content)

Ollama Tool Calling

Ollama supports tool calling for compatible models (Llama 3.1+, Mistral, Qwen 2.5):

tools = [{'type': 'function', 'function': {...}}]
response = client.chat.completions.create(
    model='llama3.1:8b',
    messages=messages,
    tools=tools
)

llama.cpp Basics

llama.cpp is a C++ implementation that runs ANY GGUF-format model on CPU, GPU, or Metal (Apple Silicon):

# Build (or install pre-built):
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp && make

# Run a quantized model:
./llama-cli -m models/llama-3.1-8b-instruct.Q4_K_M.gguf -p 'Hello'

Server Mode

./llama-server -m models/llama-3.1-8b-instruct.Q4_K_M.gguf --port 8080

# Now OpenAI-compatible endpoint at http://localhost:8080/v1

Quantization

4-bit quantization (Q4_K_M) lets you run a "16GB" model in ~5GB RAM with marginal quality loss. Try Q4 first; go higher (Q5, Q6, Q8) for quality-critical tasks.

Hardware Requirements

Model	RAM Q4	VRAM Q4
7-8B	~6 GB	~6 GB
13B	~10 GB	~10 GB
70B	~40 GB	~40 GB

Apple Silicon's unified memory makes large local models surprisingly usable.

vLLM for Throughput

For production serving with high concurrency, use vLLM (PagedAttention, continuous batching):

# pip install vllm
from vllm import LLM, SamplingParams
llm = LLM(model='meta-llama/Llama-3.1-8B-Instruct')

params = SamplingParams(temperature=0.2, max_tokens=512)
result = llm.generate(['Hello world'], params)

Text Generation Inference (TGI)

HuggingFace TGI — another production server. Strong tool-use support.

Comparing Servers

Ollama — best UX, simple CLI/HTTP, good for dev
llama.cpp — lightest, runs anywhere
vLLM — highest throughput, GPU-only
TGI — HuggingFace integration, monitoring

When to Self-Host

Strict privacy
Very high volume (cost crossover ~10M tokens/day)
Fine-tuned models
Edge / offline scenarios

When NOT to Self-Host

Small-volume side projects (hosted is cheaper)
Frontier reasoning quality required
Multimodal tasks

OpenAI-Compatible Endpoint

Why is the OpenAI-compatible endpoint convention nice for local models?

Recap

Ollama for dev simplicity, llama.cpp for portability, vLLM for production throughput. All speak OpenAI-shaped APIs so you swap with no code change.

Frequently asked questions

Is the “Running Local Models with Ollama and llama.cpp” lesson free?

Yes — the full text of “Running Local Models with Ollama and llama.cpp” is free to read here on the web, and the AI Agents course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Agents course, upgrade to CoddyKit PRO.

What will I learn in “Running Local Models with Ollama and llama.cpp”?

Ollama makes it 'docker run' for LLMs; llama.cpp goes deeper with quantization and direct C++ control. You practise AI Agents with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Agents?

No prior experience is required. AI Agents on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Running Local Models with Ollama and llama.cpp” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Agents lesson?

Yes. Every AI Agents lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

Llama, Mistral and Qwen Overview
Running Local Models with Ollama and llama.cpp
Function-Calling Open Models (Hermes, Functionary)
Trade-offs: Latency, Cost, Capability

← Back to AI Agents