Author(s): Garvit Agarwal

Originally published on Towards AI.

Optimizing LLM Token Costs in Production: A Practical Engineering Playbook [Part 3]

In Part 2, we focused on optimizing how requests are constructed before they reach the language model. We explored how techniques like Model Routing, Prompt Caching, and Conversation Summarization reduce unnecessary token usage without affecting the user experience.
Those optimizations alone can significantly reduce production costs. But here’s something that surprised me when I started studying production AI systems.

Many applications continue to spend thousands — or even millions — of unnecessary tokens after the request has already been optimized.

How?

Because they still:

• Retrieve far more context than the model actually needs.
• Process requests one at a time instead of efficiently batching them.
• Generate responses that are much longer than users require.

None of these problems originate from the language model itself. They’re engineering decisions.

And just like the techniques we discussed in Part 2, they can often be improved without changing models or sacrificing response quality.
Let’s look at three more production optimization techniques that help AI systems become faster, cheaper, and more scalable.

Lever 4- Adaptive Retrieval

Retrieval-Augmented Generation (RAG) has become one of the most common architectures for production AI applications. Instead of relying solely on the model’s training data, a RAG pipeline retrieves relevant information from an external knowledge base before generating a response.

The idea is simple:

Give the model the right context so it can produce a more accurate answer.

The challenge is deciding how much context to retrieve.

The Hidden Cost of Fixed Retrieval

Imagine you’re building an internal company chatbot.
A user asks: “What are your office hours?”
Your vector database retrieves 10 documents because the retrieval pipeline is configured with:

documents = vectorstore.similarity_search(query, k=10)

Those documents might include: Employee handbook, HR policy, Security guidelines, Travel policy and so on. The answer only needs one sentence. Yet thousands of tokens are sent to the language model.

Now imagine this happens for 50,000 requests every day.
Most of those retrieved tokens contribute nothing to the final answer — but you’re still paying for them.

One Size Doesn’t Fit Every Query

Not every question deserves the same amount of context.
Compare these two requests.

Query 1: “What are your office hours?”
A couple of relevant documents are enough.

Now consider, Query 2: “Compare our healthcare reimbursement policy with last year’s finance guidelines.”
This question requires multiple documents from different sources.

Both queries are important. But they shouldn’t retrieve the same amount of information.

Making Retrieval Adaptive
Instead of treating every query equally, production systems first estimate its complexity. Simple factual questions retrieve a small amount of context. Broader analytical questions retrieve more.

A simplified implementation looks like this:

if query_type == "simple":
 top_k = 2
elif query_type == "medium":
 top_k = 5
else:
 top_k = 10

documents = vectorstore.similarity_search(query, k=top_k)p

The logic isn’t complicated. But over millions of requests, this small engineering decision can eliminate a huge amount of unnecessary token processing.

Reranking: Quality Matters More Than Quantity

Retrieving more documents doesn’t necessarily improve answer quality. Production systems often perform a second filtering step called reranking.

Instead of passing every retrieved document to the model:
Retriever — > Top 10 Documents — > Reranker — > Top 3 Relevant Documents — >LLM

The reranker scores each document according to its relevance and forwards only the best matches.

This has two advantages:
First, the model processes fewer tokens.
Second, it receives higher-quality context.

In many cases, fewer documents actually produce better responses because the model isn’t distracted by irrelevant information.

Production Insight

Many teams spend weeks experimenting with better embedding models.
Sometimes the biggest improvement comes from something much simpler:
Stop sending documents the model doesn’t need.
A smaller, cleaner context often improves both accuracy and cost efficiency.

Lever 5- Batch Inference

So far, we’ve optimized what reaches the language model. But optimization isn’t just about reducing tokens. It’s also about how efficiently requests are processed.
This becomes particularly important when your AI application isn’t serving a single user — it might be processing thousands of documents, emails, product descriptions, or customer reviews every hour. At this scale, sending one request at a time can become surprisingly expensive.

The Hidden Cost of Sequential Processing

Imagine you’re building a document search system. Before users can search your documents, each one needs to be converted into an embedding and stored in a vector database.
Suppose you have 1,000 PDF documents waiting to be indexed.

A straightforward implementation might look like this:

for document in documents:
 embedding = embedding_model.embed(document)
 vector_db.insert(embedding)

It works. But behind the scenes, you’re making 1,000 separate API calls.

Each request carries its own:

• Network latency
• Authentication overhead
• Request initialization
• Response processing

The model spends almost as much time handling requests as it does generating embeddings.

A Better Approach

Instead of sending documents individually, production systems process them in batches.

batch_size = 100

for i in range(0, len(documents), batch_size):
 batch = documents[i:i + batch_size]

 embeddings = embedding_model.embed(batch)

 vector_db.insert(embeddings)

Now, instead of making 1,000 API calls, you’re making only 10.

The number of tokens remains almost the same, but the infrastructure becomes far more efficient.

Why Batching Improves Performance

Think of ordering coffee for your team.
Would you rather:

• Walk to the café twenty times and order one coffee each trip?
  or
• Collect everyone’s order and make a single visit?

Both approaches produce the same result. One simply wastes much less time.

Batch inference works in exactly the same way.

Instead of repeatedly setting up new requests, the system processes multiple inputs together, reducing overhead and improving throughput.

Where Batch Inference Works Best

Batching is most effective for workloads that don’t require an immediate response. Some common examples include:

• Generating embeddings for large document collections
• Indexing knowledge bases
• Classifying customer feedback
• Offline summarization
• Processing support tickets
• Content moderation

These are background jobs where processing speed matters more than instant user interaction.

For real-time chatbots, however, batching is often less suitable because users expect responses within a few seconds.

Production Insight

One question we often ask while designing AI pipelines is:
“Does this request need to be processed immediately?”
If the answer is no, it’s usually a good candidate for batching.

This simple decision can improve throughput dramatically without changing the underlying AI model. Sometimes, optimizing an AI system isn’t about making the model faster. It’s about making the pipeline smarter.

Lever 6 – Output Budgeting

By now, we’ve optimized how requests are built, how context is retrieved, and how efficiently requests are processed.

But there’s one final piece of the puzzle that many teams overlook:

The model’s response itself.

Every token generated by the LLM contributes to your bill. If your application consistently produces responses that are longer than necessary, you’re paying for information your users may never read.

Bigger Responses Aren’t Always Better

Imagine you’re building an AI assistant for internal documentation.
A user asks: “What is vector search?”
The application only needs a brief explanation.

However, because the prompt doesn’t specify a desired response length, the model generates an 800-word article covering its history, algorithms, indexing strategies, and implementation details. Technically, the answer is excellent.

But did the user really need all of that?
Probably not.

This is a common issue in production AI systems. Models naturally tend to provide detailed responses unless they’re given clear boundaries.

Set a Response Budget

One of the simplest ways to control output cost is by limiting the maximum number of generated tokens.

For example:

response = client.responses.create(
 model="gpt-4.1-mini",
 input=prompt,
 max_output_tokens=200
)

This doesn’t force the model to generate exactly 200 tokens. Instead, it prevents responses from growing unnecessarily long.Think of it as defining a budget.

Just as engineering teams monitor CPU, memory, and storage usage, it’s equally important to monitor how many output tokens your application generates.

Better Prompts Lead to Better Responses

Output budgeting isn’t only about token limits. It’s also about giving the model clearer instructions.

Consider these two prompts.

Prompt 1: Explain Retrieval-Augmented Generation.

Now compare it with:

Prompt 2: Explain Retrieval-Augmented Generation in five concise bullet points suitable for software engineers.

The second prompt communicates both the content and the expected format. As a result, the response is usually shorter, easier to understand, and consumes fewer output tokens.

Small improvements in prompt design often lead to noticeable savings when your application serves thousands of users.

Production Insight

When reviewing production AI systems, one question is surprisingly effective:

“Will the user benefit from a longer answer, or simply a clearer one?”

Most users aren’t looking for the longest response. They’re looking for the right response. By controlling output length and encouraging concise formatting, you not only reduce costs but also improve readability and user experience.

Sometimes, better AI isn’t about saying more. It’s about saying exactly what’s needed.

Bringing It All Together

Across Part 2 and Part 3, we’ve explored six practical optimization techniques that production AI teams use every day.

• Model Routing ensures each request is handled by the most appropriate model.
• Prompt Caching avoids paying repeatedly for static prompt components.
• Conversation Summarization keeps long-running chats efficient.
• Adaptive Retrieval retrieves only the context that adds value.
• Batch Inference improves throughput by processing requests more efficiently.
• Output Budgeting prevents unnecessary response generation.

Notice something interesting.

None of these techniques required training a new model. None required changing model providers. And none sacrificed response quality.

Instead, they focused on improving the engineering surrounding the model. That’s an important mindset shift.

The biggest improvements in production AI often come from system design, not model selection.

Wrapping Up

Throughout this series, we’ve explored where LLM tokens come from and the engineering techniques that keep them under control. From model routing and prompt caching to adaptive retrieval, batch inference, and output budgeting, each optimization targets a different source of inefficiency.

Individually, these techniques may seem small. Together, they fundamentally change how production AI systems are designed.

The biggest lesson isn’t that language models are expensive.

It’s that unnecessary computation is expensive.

Every unnecessary document, repeated prompt, oversized response, or redundant request quietly adds to your operational costs. Remove those inefficiencies, and you don’t just reduce your AI bill — you build systems that are faster, more scalable, and easier to maintain.

The goal isn’t to process more tokens. It’s to make every token count.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI