Author(s): Arpit Baranwal

Originally published on Towards AI.

Beyond Bigger Context Windows: 10 Context Engineering Patterns Every LLM Engineer Should Know

Large context windows are impressive, but they haven’t eliminated the need for context engineering. In production AI systems, deciding what not to send to the model is often more important than increasing the number of tokens it can process.

Introduction

When frontier models began offering very large context windows, many people predicted that Retrieval-Augmented Generation would become less important. If a model can read an entire codebase, documentation library, or book in one prompt, why bother with retrieval?
In practice, the opposite happened. Teams building production AI systems still spend significant engineering effort deciding which information reaches the model. Larger context windows increase capacity, but they do not automatically improve reasoning, reduce latency, or lower cost.
The problem has shifted
from : How do we fit information into the prompt?
to: What information actually deserves the model’s attention?
This discipline is increasingly called context engineering.

Why Bigger Context Windows Are Not Enough

Consider a technical assistant helping engineers troubleshoot industrial equipment. The available information might include manuals, maintenance logs, sensor histories, support tickets, internal engineering documents, and regulatory standards. Even if an LLM could technically process all of it, that does not mean it should. In practice, indiscriminate ingestion introduces several real operational penalties:

• Higher inference cost
• Increased latency
• More irrelevant information competing for attention
• Greater risk of conflicting or stale information
• Reduced reasoning efficiency
• Every additional token consumes computation

Every additional token consumes computation. More importantly, every irrelevant token competes for the model’s attention. Good context engineering is therefore not about maximizing context. It is about maximizing signal while minimizing noise.

Context Engineering as an Optimization Problem

Useful Context = Relevant Information — Noise — Redundancy + Structure

The objective is not to provide the model with everything you know. The objective is to provide exactly what it needs to solve the current task. This design philosophy underpins modern RAG systems, long-running assistants, and agentic AI architectures.

Pattern 1: Semantic Chunking

Many RAG systems split documents into fixed-size chunks, such as 500 or 1,000 tokens. Although simple, this can separate related ideas or merge unrelated topics.
A better strategy is semantic chunking: split by headings, sections, paragraphs, functions, API definitions or code blocks.
Each chunk should represent a complete concept rather than an arbitrary number of tokens. This improves retrieval quality because embeddings represent coherent ideas instead of fragmented text.

The following is simplified pseudocode showing the chunking:

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
 chunk_size=500,
 chunk_overlap=100,
 separators=["\n## ", "\n### ", "\n\n", "."]
)
chunks = splitter.split_text(document)

Pattern 2: Hierarchical Retrieval

Instead of searching every paragraph individually, production systems often retrieve information in multiple stages.
A system may first identify relevant documents, then relevant sections, and finally the paragraphs most useful for the current question.
This reduces search complexity while improving retrieval precision, especially in large enterprise knowledge bases.

The following is simplified pseudocode showing the retrieval flow:

docs = retrieve_documents(query, top_k=20)

sections = retrieve_sections(
 query,
 documents=docs,
 top_k=5
)

paragraphs = retrieve_paragraphs(
 query,
 sections=sections,
 top_k=8
)
answer = llm(paragraphs)

Pattern 3: Context Compression

Retrieval often returns more information than necessary. An intermediate compression step extracts only what matters for the current query. Compression can include key sentence extraction, bullet-point summaries, table generation, equation extraction or code snippet isolation. The model receives the essential evidence without unnecessary detail.

from langchain_openai import ChatOpenAI
from langchain.retrievers.document_compressors import LLMChainExtractor

# Initialize the LLM

llm = ChatOpenAI(model="gpt-4.1")
# Create the compressor
compressor = LLMChainExtractor.from_llm(llm)

# Compress retrieved documents based on the user query
compressed_docs = compressor.compress_documents(
 documents=retrieved_docs,
 query=user_query
)

# Generate the final response
response = llm.invoke(
 f""" Context: {compressed_docs} Question: {user_query} """ )

Pattern 4: Sliding Conversation Memory

Long-running assistants cannot retain every previous message indefinitely. A practical approach is to keep the last few turns unchanged while summarizing older discussion into goals, constraints, decisions and unresolved questions. This maintains continuity while keeping prompts compact.

from langchain_openai import ChatOpenAI
from langchain.memory import ConversationSummaryBufferMemory

llm = ChatOpenAI(model="gpt-4.1")

memory = ConversationSummaryBufferMemory(
 llm=llm,
 max_token_limit=1000
)

# Save each interaction
memory.save_context(
 {"input": user_query},
 {"output": assistant_response}
)

# Returns:
# • Summary of older messages
# • Most recent conversation turns
context = memory.load_memory_variables({})["history"]

response = llm.invoke(
 f""" Context: {context} User: {user_query} """
)

Pattern 5: Query Expansion and Metadata Filtering

Users rarely phrase questions using exactly the same terminology found in documentation. For example, battery degradation may need to expand into capacity fade, cycle aging, calendar aging, state of health and lithium-ion aging.
Metadata filtering then narrows the search by project, product version, department, author, date, language or security level.

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-4.1")

# Step 1: Expand the user query
expanded_queries = llm.invoke(
 f"""
 Generate 3 alternative search queries for this user question.
 Keep them short and technical. Question: {user_query}"""
).content.split("\n")

# Step 2: Search only inside the relevant metadata scope
all_results = []

for query in expanded_queries:
 results = vector_db.similarity_search(
 query=query,
 k=5,
 filter={
 "department": "Battery",
 "document_type": "technical_report",
 "year": 2025
 }
 )
 all_results.extend(results)

# Step 3: Pass filtered, expanded results to the LLM
context = "\n\n".join(doc.page_content for doc in all_results)

response = llm.invoke(f""" Context: {context} Question: {user_query} """)

Pattern 6: Retrieval Re-ranking

Vector search is fast, but embedding similarity is not always the same as task relevance. Modern systems often retrieve a larger candidate set using embeddings, then apply a cross-encoder or LLM-based re-ranker.
This often improves answer quality more than simply increasing the number of retrieved chunks.

The following is simplified pseudocode showing the retrieval re-ranking:

docs = vector_db.search(query, top_k=20)

ranked = reranker.rank(query, docs)

context = ranked[:5]

Pattern 7: Structured Agent Memory

Agentic systems often fail because they repeatedly rediscover information or carry too much irrelevant history.
Instead of storing raw conversation logs, maintain compact structured state: goal, constraints, completed work, intermediate results and next action.
This is smaller, easier to inspect, and better aligned with multi-step reasoning.

from dataclasses import dataclass, field
from typing import List


@dataclass
class AgentMemory:
 goal: str
 constraints: List[str] = field(default_factory=list)
 completed_tasks: List[str] = field(default_factory=list)
 intermediate_results: List[str] = field(default_factory=list)
 next_action: str = ""


memory = AgentMemory(
 goal="Write a Medium article on context engineering",
 constraints=[
 "Keep examples practical",
 "Use simple architecture diagrams",
 "Avoid overly academic language"
 ]
)

# Agent completes a step
memory.completed_tasks.append("Drafted introduction")
memory.intermediate_results.append(
 "Main thesis: bigger context windows do not remove the need for context engineering."
)
memory.next_action = "Write the section on structured agent memory"

# Build compact context for the next LLM call
prompt = f""" Agent State: Goal: {memory.goal} 
 Constraints: {memory.constraints} 
 Completed Tasks: {memory.completed_tasks}
 Intermediate Results: {memory.intermediate_results}
 Next Action: {memory.next_action} """

response = llm.invoke(prompt)

Pattern 8: Dynamic Prompt Assembly

A production system should not rely on one massive static prompt.
Instead, prompts are assembled from modules: system instructions, user query, retrieved documents, tool outputs, user preferences, examples and agent memory.
We include each module only when it applies, which keeps the prompt lean and the system easier to maintain.

def build_prompt(
 user_query,
 retrieved_docs=None,
 tool_outputs=None,
 user_preferences=None,
 examples=None,
 agent_memory=None
):
 prompt_parts = []

 # Always included
 prompt_parts.append(""" You are a helpful AI assistant.
 Answer clearly and use only reliable information. """)

 # Optional modules
 if user_preferences:
 prompt_parts.append(f""" User Preferences: {user_preferences} """)

 if agent_memory:
 prompt_parts.append(f""" Agent Memory: {agent_memory} """)

 if retrieved_docs:
 prompt_parts.append(f""" Retrieved Documents: {retrieved_docs} """)

 if tool_outputs:
 prompt_parts.append(f""" Tool Outputs: {tool_outputs} """)

 if examples:
 prompt_parts.append(f""" Examples: {examples} """)

 # Current task
 prompt_parts.append(f""" User Query: {user_query} """)

 return "\n".join(prompt_parts)


prompt = build_prompt(
 user_query="Explain context compression",
 retrieved_docs=retrieved_context,
 user_preferences="Prefer short examples with Python code",
 agent_memory="Previous section discussed structured memory"
)

response = llm.invoke(prompt)

Pattern 9: Tool-First Context
Sometimes the best context is no context at all.
Instead of injecting thousands of tokens from a database, spreadsheet, or log file, let the model call a tool: SQL, calculator, search, knowledge graph, API, or code execution environment.
Only the result of the tool call needs to enter the prompt. This reduces token use and usually improves factual accuracy.

from langchain_openai import ChatOpenAI
from langchain_community.utilities import SQLDatabase
from langchain.chains import create_sql_query_chain

# Connect to the database
db = SQLDatabase.from_uri("sqlite:///company.db")

llm = ChatOpenAI(model="gpt-4.1")

# Create a tool that converts natural language to SQL
sql_chain = create_sql_query_chain(llm, db)

# User question
user_query = "What were the top 5 selling products in Q1 2025?"

# Execute the SQL query
sql_query = sql_chain.invoke({"question": user_query})
result = db.run(sql_query)

# Only the query result is sent to the LLM
response = llm.invoke(f""" Question: {user_query} SQL Result: {result}
Answer the question based only on the SQL result. """)

Pattern 10: Context Isolation in Multi-Agent Systems

A common mistake in multi-agent systems is giving every agent access to every piece of information. A better design separates context by responsibility: planner, researcher, coder, reviewer and final response generator.
This reduces distraction, lowers cost and creates clearer responsibility boundaries.

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-4.1")

# Planner Agent
plan = llm.invoke(""" Task: Write a blog on context engineering.
 Create an outline only. """).content

# Research Agent
research = llm.invoke(f""" Outline: {plan}
 Find supporting references and key technical concepts. """).content

# Writer Agent
draft = llm.invoke(f""" Outline: {plan} Research Notes: {research}
Write the first draft. """).content

# Reviewer Agent
review = llm.invoke(f""" Review the following draft for:
 - Technical accuracy
 - Grammar
 - Readability
Draft: {draft} """).content

# Final Response Generator
final_article = llm.invoke(f""" Original Draft: {draft} 
Reviewer Feedback: {review} Produce the final polished article. """)

Choosing the Right Strategy

No strategy is free each comes with a cost worth knowing before you commit to it.

Practical Implementation Checklist

• Define a context budget before building the prompt.
• Retrieve more candidates than you send to the model.
• Re-rank retrieved chunks before final prompt construction.
• Compress long evidence into query-specific summaries.
• Remove duplicated or overlapping chunks.
• Keep recent turns in full; summarize the rest.
• Store agent state as structured fields.
• Use tools instead of injecting raw databases or large files.
• Isolate context between agents in multi-agent workflows.
• Log which context was used so failures can be debugged.

Final Thoughts

As language models continue to expand their context windows, the temptation will always be to send more information. The best AI systems usually do the opposite. They retrieve less, compress more, organize information intelligently, and provide the model with only what is necessary to solve the current problem.

None of this comes free. Every pattern in this piece trades one cost , latency, complexity or the risk of dropping something relevant and for another benefit: precision, efficiency, or scale. The real skill isn’t memorizing these patterns; it’s knowing which cost your system can least afford, and designing around that constraint.

Context engineering is therefore not just a workaround for context limits. It is a core discipline for building efficient, reliable, and production-ready LLM applications. The future of LLM applications will not be shaped only by models with the largest context windows. It will be shaped by systems that know exactly how to use the context they already have.

Further Reading

• A Survey of Context Engineering for Large Language Models
  https://arxiv.org/abs/2507.13334
• Anthropic: Effective Context Engineering for AI Agents
  https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
• LangChain: Context Engineering for Agents
  https://www.langchain.com/blog/context-engineering-for-agents
• Lost in the Middle : How Language Models Use Long Contexts
  https://arxiv.org/abs/2307.03172

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI