Author(s): Priyanka Mali

Originally published on Towards AI.

How context windows actually work — and how I refactored my chatbot to handle them properly.

365 Days of AI · Day 16 · ~7 min read

In Day 7, I built a chatbot that remembers everything.

Tell it your name in message 1 — it still knows it in message 20. The trick was simple: every time you send a message, the app packs the entire conversation history into the prompt and sends it to the model. The model reads it all and replies as if it was paying attention the whole time.

If you haven’t read Day 7, the one line summary is: AI has no memory. The app fakes it by showing the model a full transcript every single time.

But here’s the problem I never addressed.

What happens at message 200? Or message 500?

The transcript keeps growing. Every message adds more tokens. And eventually — you hit the context window limit. The model starts forgetting the oldest messages. The “memory” breaks.

Today I fixed that. Three ways.

A Quick Recap — What Is a Context Window?

Every time you send a message to an AI model, it doesn’t just see your latest message. It sees the entire conversation — your system prompt, every message you’ve sent, every reply it gave, and your new message. All of it, every single time.

The context window is simply how much it can see at once. Think of it as the model’s attention span for a single request.

Everything has to fit inside it:

🔹 System Prompt

🔹 Chat History

🔹 Current User Message

🔹 Retrieved Chunks (if you’re using RAG)

Different models have different limits:

🔹 GPT-3.5 → 16K tokens

🔹 GPT-4o & GPT-4.5 → 128K tokens

🔹 Claude Sonnet 4.6 → 200K tokens

Sounds huge. But a long conversation adds up faster than you think. 1 token ≈ 4 characters. A chatty 50-message conversation? Easily 3,000–5,000 tokens gone.

What happens when it gets too full?

The oldest information starts getting dropped — silently. The model doesn’t tell you it forgot something. It simply no longer has access to that context. That’s why long chats sometimes feel like the AI suddenly forgot what you discussed earlier.

And this is different from hallucination — an important distinction:

🔹 Context window issue = the model lost information because it no longer fits

🔹 Hallucination = the model makes up information even when the context is available

In production, this is usually handled using strategies like:

🔹 Sliding Window

🔹 Conversation Summarisation

🔹 RAG (Retrieval-Augmented Generation)

🔹 A hybrid approach combining all of the above

Today I implemented all three in my chatbot. Let me show you how.

Fix 1 — The Simple Way (Message Count Limit)

The simplest fix is to just keep the last N messages:

const MAX_MESSAGES = 10
function trimToLastN(history) {
 return history.slice(-MAX_MESSAGES)
}

One line. Keep the last 10 messages, drop everything older.

The upside — dead simple. Easy to understand. Works.

The downside — it’s blunt. Message 1 could be 5 words. Message 2 could be 500 words. You have no idea how many tokens you’re actually sending. You’re just counting messages, not actual size.

Good enough for a quick fix. But we can do better.

Fix 2 — The Better Way (Token Counter)

Instead of counting messages, count tokens. Keep as many recent messages as fit within a token budget.

I added two functions to chatbot.js:

const MAX_TOKENS = 2048
const CHARS_PER_TOKEN = 4
function estimateTokens(text) {
 return Math.ceil(text.length / CHARS_PER_TOKEN)
}
function trimToContextWindow(history) {
 const systemTokens = estimateTokens(SYSTEM_PROMPT)
 let availableTokens = MAX_TOKENS - systemTokens
 const trimmed = []
 // Walk from newest to oldest, keep what fits
 for (let i = history.length - 1; i >= 0; i--) {
 const msgTokens = estimateTokens(history[i].content)
 if (availableTokens - msgTokens < 0) break
 trimmed.unshift(history[i])
 availableTokens -= msgTokens
 }
 const dropped = history.length - trimmed.length
 if (dropped > 0) {
 console.log(`context window: trimmed ${dropped} old message(s)`)
 }
 return trimmed
}

Walk the history backwards from the newest message. Keep adding messages as long as there’s token budget left. The moment you run out — stop.

Much smarter than counting messages.

Fix 3 — The Smartest Way (Summarisation)

Trimming still has one problem — you’re dropping messages forever.

What if message 3 was “my name is Priyanka” and that gets trimmed? The model forgets your name. Not great for a chatbot that’s supposed to remember you.

The smartest fix: instead of dropping old messages, summarise them.

When the history crosses 10 messages, the app now asks Ollama to summarise the older half of the conversation into 3–5 sentences. That summary gets placed at the top of the context. The recent messages follow it.

const SUMMARY_THRESHOLD = 10
async function summariseHistory(oldMessages) {
 const conversation = oldMessages
 .map(m => `${m.role === 'user' ? 'User' : 'Assistant'}: ${m.content}`)
 .join('\n')
 const prompt = `Summarise the following conversation in 3-5 sentences. 
Focus on key facts the user shared (name, preferences, important context). 
Be concise. Do not add commentary.
Conversation:
${conversation}
Summary:`
 const response = await fetch('http://localhost:11434/api/generate', {
 method: 'POST',
 headers: { 'Content-Type': 'application/json' },
 body: JSON.stringify({ model: 'llama3.1', prompt, stream: false })
 })
 const result = await response.json()
 return result.response.trim()
}

The Full Strategy — Summarise First, Trim as Fallback

The final buildContextHistory() function ties it all together:

async function buildContextHistory(history) {
 // Short enough — send everything as is
 if (history.length <= SUMMARY_THRESHOLD) return history
 const splitPoint = Math.floor(history.length / 2)
 const summary = await summariseHistory(history.slice(0, splitPoint))
 if (summary) {
 // Prepend summary, keep recent messages
 return [
 { role: 'assistant', content: `[Summary: ${summary}]` },
 ...history.slice(splitPoint)
 ]
 }
 // If summarisation fails - fall back to token trimming
 return trimToContextWindow(history)
}

Three levels:

1. History is short → send everything
2. History is long → summarise old messages, keep recent ones
3. Summarisation fails → trim by tokens

Nothing ever breaks. The model always gets the best possible context it can.

One Design Decision I’m Proud Of

The trimming and summarisation only happens before sending to the model. The full history is always saved to disk.

// Model gets smart trimmed context
const contextHistory = await buildContextHistory(data.history)
// But disk gets everything - nothing lost
data.history.push({ role: 'assistant', content: assistantReply })
saveHistory(data)

Your conversation file is a record. You might need it later for debugging, analytics, or future features. So the model gets a trimmed window — the file stays complete.

What the Logs Look Like Now

Before:

message received: hey what's my name?
history loaded — 14 messages so far
sending to Ollama...

After:

message received: hey what's my name?
history loaded — 14 messages so far
context window: history has 14 messages, summarising old ones...
summariser: summary generated — The user's name is Priyanka...
context window: using summary + 7 recent messages
estimated prompt size: ~1203 tokens
sending to Ollama...

That one extra block of logs tells me exactly what’s happening under the hood every single time.

What I Actually Learned

This felt like a small code change. It was actually a big mental shift.

Context window isn’t just a spec sheet number you read in documentation and forget. It’s an active constraint you design around. The apps that feel like they “just work” over long conversations? Someone built exactly this logic behind the scenes. It doesn’t happen by default.

And the models that charge more per token? Part of what you’re paying for is a bigger attention span. That 1M token Gemini model isn’t just impressive — it’s solving a real engineering problem.

Full updated code on GitHub: chatbot-app

I’m Priyanka — Senior Software Engineer, AI learner, and someone who just made her chatbot smarter without touching the model. I write about AI every day on this series. Follow along for Day 17.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI