Calculating and Predicting API Costs

API costs for LLM applications can be surprisingly large at scale. A single query that seems cheap at $0.002 becomes $200 when run 100,000 times. Without cost prediction and monitoring, AI features can generate unexpected cloud bills that dwarf your entire infrastructure spend.

The good news is that LLM costs are entirely predictable before you send a request: you know the model, you can count the input tokens with tiktoken, and you can estimate output tokens based on your max_tokens setting or historical averages. Building cost prediction into your application from day one prevents billing surprises.

OpenAI charges separately for input tokens and output tokens, with output typically costing 3-4x more. Prices vary by model. As a guide for 2025 (always check the current pricing page as it changes):

• gpt-4o-mini: ~$0.15/million input, ~$0.60/million output
• gpt-4o: ~$2.50/million input, ~$10.00/million output
• text-embedding-3-small: ~$0.02/million tokens

The cost difference between models is enormous: gpt-4o is approximately 17x more expensive than gpt-4o-mini per input token. Model selection is your biggest lever for cost control — always start with the cheapest model that meets your quality requirements.

Build a cost estimator that you call before sending any request. It counts input tokens with tiktoken, estimates output tokens from your max_tokens parameter, looks up the per-model price, and returns the estimated cost in dollars. Call this in development and log the results so you build intuition for what different query types cost.

After each API call, the response object contains the actual token counts used. Extract these to log real costs and compare them to your estimates. Over time, the gap between estimated and actual output tokens tells you how accurately you are predicting usage, and the logs give you a cost breakdown by feature or user segment.

Once you know the average cost per request and the expected request volume, projecting monthly costs is straightforward. Build a cost model spreadsheet or a simple Python script that lets you test different assumptions: what if daily active users doubles? What if we add a feature that makes 3 API calls per user action instead of 1?

The single biggest lever for cost reduction is using the cheapest model that meets your quality requirement. For many tasks, gpt-4o-mini performs comparably to gpt-4o but at ~17x lower cost. Before defaulting to the most powerful model, benchmark the cheaper model on your specific task and switch to the more expensive one only if quality falls below your threshold.

A tiered routing strategy is even more effective: classify incoming requests by complexity and route simple queries to cheap models and complex ones to expensive models. Even routing 70% of traffic to the cheap model while 30% goes to the expensive one saves approximately 70% of your AI costs.

Every token in your prompt costs money. A verbose system prompt that could be condensed without losing meaning directly increases your API bill for every request. Benchmark the token count of your prompts and look for opportunities to tighten wording. Similarly, long few-shot examples can often be replaced by shorter equivalents without sacrificing accuracy.

For RAG systems, the retrieved context is often the largest part of the prompt. Returning 10 large chunks when 3 well-chosen smaller ones would suffice wastes tokens on every query. Tune your retrieval to minimize redundant context while maximizing relevance.

OpenAI offers a Batch API that processes requests asynchronously at 50% off the standard price. If your use case is not latency-sensitive — document processing, nightly analysis jobs, bulk content generation — the Batch API can halve your costs with minimal code changes.

Batch requests are submitted as JSONL files, processed within 24 hours, and results retrieved from the API. This is ideal for preprocessing pipelines that run on a schedule and do not need real-time responses.

Always configure spending limits to prevent runaway costs. In the OpenAI dashboard, you can set monthly spending caps that cut off API access when the limit is reached. Set a hard limit at your maximum acceptable spend and a soft limit at 80% of that value to receive an email warning before hitting the hard cap.

In your application code, implement a per-user or per-feature budget tracked in your database. Check the budget before each API call and return an error if it is exhausted. This prevents a single runaway user or a bug in a batch job from consuming your entire monthly quota in hours.

The cheapest API call is the one you never make. Implement caching at the application layer to serve identical requests from cache rather than calling the API again. Even a simple Redis cache keyed on the SHA256 hash of the prompt can eliminate a significant fraction of redundant calls in a production application.

For embeddings, caching is especially impactful: the same text should only be embedded once. Store embeddings in your vector database with the original text as a key, and check for an existing embedding before calling the embeddings API. In a RAG pipeline, document embeddings are computed once at index time and reused for every query that retrieves them.

In production, you need visibility into your AI costs broken down by feature, user, and model. Build a cost dashboard by logging every API call's token counts and associated metadata (user ID, feature name, model) to a time-series database. Then aggregate to answer questions like: which feature is driving the most spend? Are power users generating disproportionate costs? Is cost per query trending up after a prompt change?

This visibility is essential for making data-driven optimization decisions rather than guessing where to cut costs. Most companies discover that 20% of their features account for 80% of their AI spend, and optimizing those features has outsized impact.

Test your understanding of AI Engineering concepts from this lesson.

In this lesson you learned: API costs can be predicted before sending by counting input tokens with tiktoken and estimating output tokens, model selection is the biggest cost lever — gpt-4o-mini can be 17x cheaper than gpt-4o for appropriate tasks, and caching, batching, and spending limits prevent runaway costs in production. Next up we explore strategies for managing long conversations that exceed the context window.