Author(s): Anthony Usoro

Originally published on Towards AI.

From choosing the right LLM to shipping a production RAG pipeline — the practical path, not the demo-day version.

Building an AI demo takes an afternoon. Deploying AI that survives real users, real data, and real failure modes takes considerably more discipline. Most of the AI projects that stall inside a business don’t fail because the underlying model was weak — they fail because the gap between “it worked in my notebook” and “it works in production” was never properly planned for.

This guide walks through that gap step by step: how to decide what kind of AI system you actually need, how to build it, and how to deploy it in a way that holds up once real customers or employees depend on it.

Step 1: Define the problem before you touch a model

The single most common mistake is starting with “we should use AI” instead of “we have this specific problem.” Before writing any code, get clear on:

• What decision or task are you automating or augmenting? Answering customer questions, searching internal documents, summarizing reports, classifying support tickets — these are different problems with different architectures.
• Does the answer need to be grounded in your own data? If yes, you likely need retrieval (RAG), not just a raw LLM call.
• What does failure look like, and how costly is it? A wrong answer in a marketing brainstorm tool is harmless. A wrong answer in a compliance assistant is not. This should shape how much verification and citation your system needs.
• What’s your actual data volume and update frequency? A 20-document FAQ and a 50,000-document, daily-updating knowledge base call for very different pipelines.

This step feels like it slows you down. In practice, it’s the step that prevents you from building the wrong system quickly.

Step 2: Choose the right LLM for the job

There’s no single “best” model — there’s a best model for your constraints. The real decision comes down to four tradeoffs:

• Accuracy vs. cost. Larger, more capable models cost more per request. For high-volume, low-stakes tasks (like drafting product descriptions), a smaller or cheaper model is often the right call. For compliance-sensitive answers, the accuracy gain from a stronger model is usually worth the cost.
• Latency. Customer-facing chat needs to feel responsive. Batch processing (like summarizing overnight reports) can tolerate slower, cheaper inference.
• Hosted API vs. self-hosted open-source model. Hosted APIs (like OpenAI’s) are faster to integrate and require no infrastructure management, but your data leaves your environment for inference. Self-hosted open-source models (Llama, Mistral, and others) keep everything in-house, which matters for regulated industries, but require you to manage GPU infrastructure and model updates yourself.
• Data residency and compliance requirements. In regulated sectors, this can override every other consideration. If your compliance requirements mean sensitive documents can’t leave your infrastructure, that alone may decide whether you self-host.

Many production systems don’t pick just one model — they route different tasks to different models based on these tradeoffs.

Step 3: Build the retrieval layer, if your AI needs to know your data

If your use case requires answers grounded in your own documents, records, or knowledge base, you need a Retrieval-Augmented Generation (RAG) pipeline underneath your model. At a high level:

1. Ingest your documents — PDFs, spreadsheets, database records, support tickets — handling messy real-world formats, not just clean text.
2. Chunk them into passages small enough to search efficiently, carefully enough that you don’t cut a clause or table in half.
3. Embed each chunk into a vector representation and store it in a vector database. PostgreSQL with the pgvector extension is a strong default choice here: it adds vector search directly into a database most engineering teams already run, rather than standing up a separate specialized system.
4. Retrieve the most relevant chunks for a given question, ideally combining semantic (vector) search with keyword search so you don’t miss exact terms like reference numbers.
5. Generate the answer by handing the retrieved chunks to the LLM, and — this part is often skipped, and shouldn’t be — have it cite exactly which chunk supported which part of the answer.

That last point is what separates a trustworthy internal tool from a chatbot that occasionally sounds right.

Step 4: Build the application layer

This is where the AI model becomes an actual product. A typical production setup includes:

• A backend API — FastAPI is a common, well-supported choice for this in Python, handling request routing, authentication, and orchestration between your retrieval layer and the LLM.
• Authentication and access control — JWT-based authentication with role-based access control (RBAC) if different users or departments should see different data. This matters even for internal tools: not every employee should be able to query every document.
• Rate limiting and cost controls — LLM API calls cost money per request. Without limits, a bug or a bad actor can generate a very expensive bill quickly.
• Logging and audit trails — especially important if your system operates in a regulated environment, where you may need to show what was asked, what was retrieved, and what was answered.

Step 5: Design the conversational layer carefully

If you’re building a chatbot rather than a single-query tool, a few additional design decisions matter:

• Conversation memory — how much prior context does the model see in a multi-turn conversation? Too little, and it forgets what the user just said. Too much, and costs and latency increase.
• Guardrails — explicit instructions and validation to keep the model from answering questions outside its intended scope, especially for customer-facing deployments.
• Fallback behavior — what happens when there’s no good answer in the retrieved documents? A well-designed system says “I don’t have information on that” rather than fabricating a plausible-sounding response.

Step 6: Test like it’s already in production

Before launch, evaluate the system the way real usage will stress it:

• Build a test set of real questions, including edge cases and questions the system should refuse to answer.
• Check retrieval quality separately from generation quality. If the wrong document gets retrieved, no amount of prompting will produce a correct answer — this is one of the most common, hardest-to-diagnose failure points.
• Monitor for hallucination, particularly on questions where the answer isn’t actually in your documents.
• Load-test for concurrency and latency under realistic traffic, not just single-user testing.

Step 7: Deploy for production, not for a demo

A demo running on your laptop and a production system are different engineering problems. Production deployment typically means:

• Containerization with Docker, so your environment is reproducible across development, staging, and production.
• Orchestration with Kubernetes if you need to scale horizontally or manage multiple services reliably.
• Monitoring and alerting — track latency, error rates, and API costs in real time, not after a customer complains.
• A rollback plan — LLM behavior can shift when you change a prompt, a model version, or a retrieval parameter. Being able to revert quickly matters.
• Infrastructure choice aligned with your compliance needs — cloud-hosted, self-hosted, or hybrid, decided in Step 2 and carried through consistently.

Startups vs. enterprises: what actually differs

The steps above apply either way, but the emphasis shifts:

Startups typically move faster and tolerate more risk. Speed to a working version matters more than exhaustive guardrails, and a hosted LLM API with a lightweight RAG setup is often the right starting point. The main trap to avoid is skipping Step 1 — building an impressive demo that doesn’t map to a real, validated problem.

Enterprises, especially in regulated industries, need to weight Steps 2, 4, and 7 much more heavily from day one: data residency, access control, audit trails, and infrastructure ownership aren’t optional add-ons, they’re often procurement requirements before a deal can even close. The main trap here is the opposite of the startup one — spending months on infrastructure before validating that the AI system actually solves the intended problem.

The bottom line

Deploying AI in a business isn’t a single technical decision — it’s a sequence of them, each with real tradeoffs: which model, how much retrieval, how much infrastructure, how much guardrail. Skipping any one step tends to surface later as a production incident, a compliance gap, or a system nobody trusts enough to actually use.

The businesses that get real, durable value from AI aren’t the ones with the fanciest demo — they’re the ones that treated deployment as seriously as the model choice itself.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI