Author(s): Deepanshu Gupta

Originally published on Towards AI.

Fine-tuning vs RAG vs MeMo: Where should LLM Knowledge Live?

LLM knowledge does not fail all at once. It goes stale quietly.

A policy changes or a product documentation is updated. A customer contract is amended, or a regulation is revised. The model still answers confidently, but now its confidence is attached to yesterday’s truth.

That is the real problem with “updating” large language models. The issue is not just whether the model knows a fact. It is where that fact lives, how quickly it can be changed, whether it can be cited, who is allowed to see it, and whether it can be deleted later.

This is why the old debate about “fine-tuning vs RAG” is becoming too small.

Fine-tuning changes weights.RAG changes context. MeMo ( Memory as a Model ) proposes something different — a separate trainable memory layer.

The next generation of LLM systems may not be defined by one model that knows everything, but by how well engineers separate behaviour, evidence, memory, and governance.

That separation matters because enterprise knowledge has different lifecycles. Some knowledge is stable enough to shape behaviour. Some changes daily and must remain auditable. Some is permissioned and should never enter model weights. Once those requirements diverge, updating knowledge becomes an architecture decision, not just a training decision.

The practical question becomes:

Where should changing knowledge live?

In model weights, retrieved context, long context, application memory, a separate memory model, or a governed knowledge layer outside the LLM?

For the last few years, the default enterprise answer has usually been retrieval-augmented generation. RAG keeps knowledge outside the model and injects evidence at inference time, which makes updates, citations and permissions easier to govern.

But a newer pattern, MeMo — Memory as a Model introduces a third option. Instead of changing the reasoning model or retrieving source chunks, MeMo keeps the executive LLM frozen and trains a separate memory model from a target corpus. At inference time, the executive model queries that memory model through a structured protocol.

That gives us a useful mental model:

Fine-tuning changes weights.

RAG changes context.

MeMo changes the memory layer

These are not interchangeable techniques. They solve different problems, introduce different failure modes, and belong to different layers of the production stack.

Fine-tuning is strongest when the goal is behavioural adaptation.

RAG is strongest when the goal is fresh, auditable, source-grounded access to knowledge.

MeMo is interesting because it treats memory as a trainable system component rather than a passive store.

Updating an LLM is not one problem

“Updating an LLM” can mean several different things.

Sometimes we want the model to behave differently. It may need to follow a schema, classify records, extract entities, write in a specific tone, generate code in a house style, call tools using a strict format, or follow a workflow more reliably.

Other times, we want the system to access new facts. It may need the latest policies, a new API reference, an updated contract or a recently published research article.

These are not the same problem.

Changing behaviour, accessing evidence, storing reusable knowledge, enforcing permissions, and reasoning across a corpus are different engineering problems. Treating them as one “LLM update” problem is where many production systems become fragile.

This is where many LLM systems become fragile. Teams reach for one mechanism and expect it to solve every update problem.

A better architecture starts by separating the problems:

The deeper engineering point is that modern LLM applications are moving away from the idea of one model that knows everything. The production stack is becoming modular, and fine-tuning, RAG and MeMo belong to different parts of that stack.

Fine-tuning: changing the model

Fine-tuning updates the model itself, or an attached adapter, so that it behaves differently after training.

This includes supervised fine-tuning, instruction tuning, domain-adaptive continued pretraining, task-adaptive training, parameter-efficient fine-tuning, and preference optimisation. In current practice, LoRA and QLoRA are especially important because they make adaptation cheaper than full fine-tuning.

LoRA freezes the base model and learns low-rank update matrices. QLoRA further reduces memory requirements by training adapters on a quantised base model. These techniques make it practical to adapt large models for specific domains, formats, or workflows without updating every parameter.

Fine-tuning is useful when the target is behaviour.

This is why fine-tuning works well for support bots, extraction pipelines, coding assistants, domain-specific classifiers, and structured-output systems. If the model already has the relevant information but does not reliably express it in the right format, fine-tuning can help.

But fine-tuning is usually the wrong place for rapidly changing factual knowledge. If a policy changes weekly, every update creates a model or adapter artefact that has to be trained, evaluated, promoted, monitored, rolled back if needed, and governed.

A compact fine-tuning workflow should look like this:

RAG: changing the context

Retrieval-Augmented Generation became the default enterprise pattern because it leaves the base model unchanged.

Instead of storing updated knowledge inside the model weights, RAG keeps knowledge in external systems. At inference time, the user query is used to retrieve relevant passages. Those passages are inserted into the prompt, and the model generates an answer grounded in that retrieved evidence.

A minimal RAG pipeline looks like this:

That simple pipeline hides most of the production work. Real RAG usually needs document parsing, layout-aware chunking, metadata, hybrid retrieval, reranking, access-control filtering, citation attribution, logging, and evaluation.

The hard part is not putting embeddings in a vector database. It is making retrieval reliable under messy, changing and permissioned, real-world conditions.

Dense retrieval is good at semantic similarity, but it can miss exact terms, IDs, policy numbers, acronyms, or legal phrases. Sparse retrieval such as BM25 handles lexical matching better but may miss paraphrases. Many production systems therefore combine sparse and dense retrieval, then use a reranker to improve precision.

Metadata filtering is equally important. In enterprise systems, the retriever should not only think about “which passages are relevant?” It must also think about “Is this document current?” “Is the source authoritative?” “ Does this answer require a newer version of the document?”

Access control must happen before the retrieved text reaches the model. Filtering after generation is too late because the model may already have seen restricted content.

This is one of RAG’s biggest production advantages. Documents can be updated without retraining the LLM. Old documents can be removed. Permissions can be enforced at retrieval time. Sources can be cited, and answers can be traced back to source passages.

But RAG does not mean the model has learned the knowledge. It only means the knowledge was made available at inference time. The failure modes are therefore retrieval-shaped: missed evidence, poor chunk boundaries, stale documents ranking too high, noisy top-k context, weak citation alignment, and brittle multi-hop synthesis.

This is why RAG evaluation must be layered.

Many apparent “LLM failures” in enterprise systems are actually retrieval failures. The model may be blamed for hallucinating when the retriever never gave the right evidence.

RAG is powerful, but it is an inference-time evidence pipeline, not a complete knowledge solution.

Long-context models reduce some retrieval pressure, but they do not remove the knowledge architecture problem. They still do not solve freshness, source authority, access control, versioning, deletion, or citation accuracy. Long context is best seen as more room to reason over selected evidence, not a replacement for retrieval.

MeMo: changing the memory layer

MeMo, or Memory as a Model, proposes a different architecture.

The core idea is to keep the main reasoning model frozen and train a separate memory model from a target corpus. At inference time, an executive LLM queries this memory model through a structured multi-turn protocol.

This creates a third option between standard fine-tuning and standard RAG. Fine-tuning updates the reasoning model. RAG retrieves source chunks. MeMo trains a separate model that internalises corpus knowledge.

A simplified MeMo architecture looks like this:

The important detail is how the memory is built.

The MeMo pipeline uses a generator model to convert the target corpus into synthetic reflection data, then trains a memory model on that representation. This matters because some questions are not answered by one retrieved chunk; they depend on relationships distributed across many documents. MeMo tries to encode that synthesis into the memory layer during training.

At inference time, the executive model does not simply ask the memory model one question. It uses a structured protocol:

• Grounding subquestions
• Entity Identification
• Answer seeking and synthesis

In other words, the executive model treats the memory model like an external knowledge oracle. The memory model is not a vector index over source chunks. It is a trained model that answers from internalised corpus knowledge.

That makes MeMo different from RAG.

RAG retrieves raw evidence; MeMo generates memory responses. RAG keeps knowledge external and inspectable while MeMo compresses knowledge into a memory model’s parameters.

This is why MeMo is architecturally interesting. It suggests that memory can become an active model component rather than a passive datastore.

The reported results are also notable. The MeMo paper evaluates on tasks such as BrowseComp-Plus, NarrativeQA, and MuSiQue, which involve browsing, long-form narrative understanding, and multi-hop reasoning. The paper reports that MeMo is competitive with, and in some settings stronger than, retrieval baselines such as BM25, dense embedding retrieval, HippoRAG2, and Cartridges. It also reports better robustness under distractor-heavy corpora, where retrieval systems can degrade as irrelevant documents increase.

But this advantage comes with a trade-off. MeMo does not remove cost. It moves cost. RAG pays for indexing, retrieval, reranking and context tokens around query time. MeMo moves more cost upstream into reflection generation and memory-model training, then adds inference cost through executive-memory turns. The trade-off is architectural, not free.

A rough comparison looks like:

The last row is important. MeMo keeps the executive LLM frozen, which helps avoid catastrophic forgetting in the reasoning model. It may also work well with closed-source LLMs because the executive model does not need to be fine-tuned.

A useful rule :

Treat MeMo as a promising memory layer, not as a drop-in RAG replacement.

Why MeMo is not just another RAG technique

The boundary is blurry, but the distinction is useful. Both RAG and MeMo augment a frozen reasoning model, but they store knowledge in different forms. RAG keeps knowledge in documents and indexes. MeMo stores knowledge in a trained memory model. RAG retrieves passages. MeMo answers memory queries. That difference matters most for provenance.

The practical difference is where knowledge lives.

The most important difference is provenance.

RAG can show the source document, whereas MeMo can answer from internalised knowledge, but unless it is paired with retrieval, citation generation, or provenance tracking, it may be harder to prove where an answer came from. That matters in legal, healthcare, finance, HR, procurement, scientific research, and other governed environments.

In these settings, being right is not enough. The system also needs to show which source it used, whether the user was allowed to access that source, whether the source was current, and whether the answer can be reproduced later.

This is why “MeMo kills RAG” is the wrong interpretation.

A better interpretation is:

MeMo may become a memory layer that complements RAG, especially when cross-document synthesis matters.

The memory layer may suggest. The retrieval layer may verify. The executive model may compose. The governance layer may enforce.

The broader shift: knowledge is becoming modular

MeMo is part of a wider movement toward modular knowledge systems.

Naive RAG works well when the answer lives in a small number of source passages. It struggles more when the question requires corpus-level sensemaking, such as identifying recurring risks across many incident reports.

GraphRAG, hierarchical summaries, long context and memory models are all attempts to handle that broader class of questions.

Model editing is another related direction. Methods such as ROME and MEMIT attempt to directly modify factual associations inside transformer weights. The goal is attractive: update specific facts without full training. But model editing raises difficult questions around side effects, generalisation, specificity, rollback, and whether edited knowledge behaves consistently across prompts.

The pattern across these approaches is clear:

Knowledge does not need to live in one place

Some knowledge belongs in weights. Some belongs in documents. Some in indexes. Some belong in tools and memory models. And some should never be placed inside a model at all.

This is the architectural shift that matters. The future is not one larger model that stores everything. It is a layered system that decides where each kind of knowledge should live.

For most production teams today, RAG should still be the default for fresh, auditable enterprise knowledge. It has the clearest operational path for document updates, source attribution, access control, and deletion. Its failure modes are not trivial, but they are relatively well understood, and the tooling ecosystem is mature.

Fine-tuning should be used when behaviour needs to change: output format, classification accuracy, extraction reliability, tone, tool-use discipline, or workflow consistency.

MeMo-like memory models are worth tracking, prototyping, and potentially combining with retrieval. They are especially interesting for cross-document synthesis, closed-source LLM compatibility, and reusable learned memory. But they should not yet replace source-grounded retrieval in regulated or high-governance environments.

A practical rule is enough:

• Use fine-tuning for behaviour.
• Use RAG for evidence.
• Use GraphRAG or hierarchical summaries for corpus-level structure.
• Use memory models for learned corpus memory.
• Use hybrid systems when the application needs all of the above.

This avoids common anti-patterns. Do not fine-tune solely to add a weekly policy update. Do not assume RAG citations are automatically correct. Do not treat long context as a substitute for retrieval quality. Do not deploy memory without provenance. Do not assume MeMo solves hallucinations. Do not evaluate only average answer quality while ignoring freshness, deletion, permissions, latency, and auditability.

The real work is not choosing the most fashionable acronym. The real work is designing a knowledge architecture that can survive change. Fine-tuning, RAG, and MeMo are not enemies; they are different answers to different update problems.

That is why MeMo matters. Not because it proves RAG is obsolete, but because it points toward a broader shift:

Memory may become an active architectural component rather than a passive store.

The next generation of reliable AI systems may not be defined by one model that knows everything. It may be defined by knowing exactly where each kind of knowledge should live.

#ArtificialIntelligence #GenerativeAI #LLM #RAG #MachineLearning

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI