Prompt Engineering for Production LLM Apps
System prompt design, few-shot examples, output formatting, retry logic, cost optimization.
Prompt Engineering for Production LLM Apps is a free Learn AI with Python lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Learn AI with Python learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Prompting in Production
In real apps, prompts are code: they need structure, testing, and reliability. Good prompt engineering reduces hallucinations, enforces output format, and controls cost. This lesson covers the techniques that matter in production.
System Prompt Design
The system prompt sets the model's role, tone, constraints, and rules. Be specific: state what the assistant is, what it must and must not do, and the format of its answers. A clear system prompt is your strongest steering tool.
system = (
"You are a support agent for an e-commerce store. "
"Answer only questions about orders and shipping. "
"If asked anything else, politely decline. "
"Keep replies under 3 sentences."
)Few-Shot Examples
Few-shot prompting means showing the model example input/output pairs so it learns the pattern. You place these as alternating user/assistant messages before the real request.
messages = [
{"role": "system", "content": system},
{"role": "user", "content": "Classify: I love this!"},
{"role": "assistant", "content": "positive"},
{"role": "user", "content": "Classify: This broke instantly."},
{"role": "assistant", "content": "negative"},
{"role": "user", "content": "Classify: It is okay I guess."}
]Why Few-Shot Works
Examples disambiguate your intent better than instructions alone. They lock in the exact output style (one word, JSON, a label) and dramatically improve consistency for classification, extraction, and formatting tasks.
Forcing JSON Output
For machine-readable replies, request JSON mode with response_format. With {"type": "json_object"} the model is constrained to emit valid JSON, which you can then parse safely.
response = client.chat.completions.create(
model="gpt-4o",
messages=messages,
response_format={"type": "json_object"}
)
import json
data = json.loads(response.choices[0].message.content)Counting Tokens with tiktoken
Cost and context limits are measured in tokens. The tiktoken library counts tokens locally so you can budget before sending. Install with pip install tiktoken.
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
tokens = enc.encode("Hello, how are you?")
print(len(tokens))Why Token Counting Matters
Each model has a context window (a max token count) and you pay per token. Counting ahead lets you trim history, reject oversized inputs, and predict cost before the call ever leaves your server.
def cost_estimate(text, price_per_1k=0.005):
enc = tiktoken.encoding_for_model("gpt-4o")
n = len(enc.encode(text))
return n, n / 1000 * price_per_1kHandling Rate Limits
Production traffic hits rate limits (HTTP 429) and transient errors. Instead of crashing, retry with exponential backoff. The tenacity library makes this a one-line decorator.
pip install tenacity
from tenacity import retry, wait_exponential, stop_after_attemptRetry with tenacity
Decorate your API call with @retry. Use wait_exponential to grow the delay between attempts and stop_after_attempt to cap total tries. This survives temporary outages gracefully.
@retry(wait=wait_exponential(min=1, max=30), stop=stop_after_attempt(5))
def ask(prompt):
return client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": prompt}]
)Defining a JSON Schema for Output
When you force JSON mode, always describe the exact shape you want inside the prompt so keys are predictable. Spelling out the schema in plain text plus JSON mode gives you parseable, validatable results every time.
system = (
"Return ONLY JSON with this shape: "
"{\"sentiment\": \"positive|negative|neutral\", "
"\"confidence\": number between 0 and 1}"
)Putting It Together
A production prompt pipeline: design a strict system prompt, add few-shot examples, force JSON output when needed, count tokens to manage cost and context, and wrap calls in retry logic. Together these make LLM features reliable.
Quick Check
Test your production prompting knowledge.
Recap: Production Prompting
You learned to craft strict system prompts, steer behavior with few-shot examples, and guarantee structured replies via response_format JSON mode. You counted tokens with tiktoken to manage context and cost, and added resilience with tenacity retries for rate limits.
Frequently asked questions
Is the “Prompt Engineering for Production LLM Apps” lesson free?
Yes — the full text of “Prompt Engineering for Production LLM Apps” is free to read here on the web, and the Learn AI with Python course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Learn AI with Python course, upgrade to CoddyKit PRO.
What will I learn in “Prompt Engineering for Production LLM Apps”?
System prompt design, few-shot examples, output formatting, retry logic, cost optimization. You practise Learn AI with Python with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Learn AI with Python?
No prior experience is required. Learn AI with Python on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Prompt Engineering for Production LLM Apps” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Learn AI with Python lesson?
Yes. Every Learn AI with Python lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- OpenAI API: chat.completions and Streaming
- Anthropic Claude API in Python
- Function Calling and Tool Use with LLMs
- Prompt Engineering for Production LLM Apps