Quantisation and Speculative Decoding
For self-hosted models: int8/int4 quantization for memory, speculative decoding for throughput.
Quantisation and Speculative Decoding is a free AI Agents lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Agents learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Self-Hosted Model Optimisations
For self-hosted open models, two big speed/cost wins:
- Quantisation — smaller weights, less RAM/VRAM, faster inference
- Speculative decoding — generate multiple tokens per step
Quantisation Basics
Models are normally stored as FP16 (16 bits per weight). Quantisation reduces to fewer bits:
- FP16 — baseline
- INT8 — 2x smaller, ~negligible quality loss
- INT4 / Q4_K_M — 4x smaller, small quality loss
- INT2 / 1.58-bit — 8x+ smaller, real quality loss
GGUF and Quantisation Levels
GGUF is the format used by llama.cpp. Common levels:
- Q4_K_M — best balance (quality vs size)
- Q5_K_M — slightly bigger, slightly better
- Q8_0 — near-FP16 quality, 2x compression
Quantise With llama.cpp
./llama-quantize models/llama-3-8b-f16.gguf models/llama-3-8b-q4_k_m.gguf Q4_K_M
# Or download pre-quantised from TheBloke / unsloth on HuggingFaceHardware Impact
An 8B FP16 model needs ~16GB VRAM. Same model in Q4 fits in ~5GB VRAM — runs on a much cheaper GPU.
Speculative Decoding
The trick: use a small "draft" model to propose tokens, then verify with the big "target" model in one forward pass. Often 2-3x speedup.
from transformers import AutoModelForCausalLM
target = AutoModelForCausalLM.from_pretrained('meta-llama/Llama-3.1-70B')
draft = AutoModelForCausalLM.from_pretrained('meta-llama/Llama-3.1-8B')
outputs = target.generate(
input_ids,
assistant_model=draft, # speculative decoding
max_new_tokens=200
)Why It Works
The draft proposes K tokens. The target processes them all in PARALLEL (one forward pass). Each accepted token is "free". Rejection rolls back; usually you accept most.
Other Decoding Speedups
- Lookahead decoding — multiple drafts per step
- Medusa — multiple decoding heads trained jointly
- EAGLE — fast tree-based speculation
Tools That Implement These
- vLLM — has both quantisation (AWQ, GPTQ, FP8) and speculative decoding
- llama.cpp — quantisation, speculative via --draft-model
- TensorRT-LLM — NVIDIA's production server
- SGLang — fast batch-friendly server with continuous batching
Continuous Batching
vLLM's flagship feature: process multiple requests with different lengths in the same forward pass. Massive throughput win for production servers.
Choosing Quantisation Level
Test each candidate level on YOUR eval. Q4_K_M is the default starting point; go higher if quality drops below threshold.
Per-Token Latency vs Throughput
Different optimisations help different metrics:
- Single-request latency: speculative decoding
- Multi-user throughput: continuous batching
- Memory cost: quantisation
Quantisation Effect
What is the primary effect of int4 quantisation?
Recap
Quantise to Q4 for memory + speed. Use speculative decoding for latency. Continuous batching for throughput. vLLM and llama.cpp implement all three.
Frequently asked questions
Is the “Quantisation and Speculative Decoding” lesson free?
Yes — the full text of “Quantisation and Speculative Decoding” is free to read here on the web, and the AI Agents course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Agents course, upgrade to CoddyKit PRO.
What will I learn in “Quantisation and Speculative Decoding”?
For self-hosted models: int8/int4 quantization for memory, speculative decoding for throughput. You practise AI Agents with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start AI Agents?
No prior experience is required. AI Agents on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Quantisation and Speculative Decoding” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this AI Agents lesson?
Yes. Every AI Agents lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Token Budgets Per Step
- Model Routing (Cheap -> Expensive)
- Caching Prompts and Results (Anthropic, Vertex)
- Quantisation and Speculative Decoding