Author(s): Divakar Ungatla

Originally published on Towards AI.

AI Engineering Fundamentals
AI Evaluation · Part 5

← Part 4

In the previous article, we explored Human Evaluation and how human reviewers can assess AI application quality using structured evaluation criteria.

Human evaluation solves an important problem – it allows us to measure qualities that are difficult to express as deterministic rules, such as helpfulness, relevance, completeness and groundedness.

However human evaluation has a practical limitation — it does not scale.

As AI application grows, the number of interactions increases rapidly. Reviewing every response manually becomes expensive, time-consuming, and difficult to maintain.

This raises an important question:

Can we automate the evaluation process while maintaining the quality of human judgment ?

This is where LLM-as-a-Judge comes in.

Instead of relying entirely on human reviewers, we can use another large language model to evaluate AI-generated responses against predefined evaluation criteria.

At a high level, LLM-as-a-Judge introduces another LLM into the evaluation pipeline. The application LLM generates the response, while the judge LLM evaluates that response against predefined criteria.

In this article, we will explore how LLM-as-a-Judge works, build an evaluation pipeline for an AI application, and see how evaluation platforms like LangSmith help run and track these evaluations at scale.

How Does LLM-as-a-Judge Work?

The key idea behind LLM-as-a-Judge is simple — instead of asking a language model only to generate responses, we introduce another language model whose responsibility is to evaluate those responses.

In an AI application, there are now two distinct roles:

• Application LLM — generates the response for the user.
• Judge LLM — evaluates the generated response against predefined evaluation criteria.

The application LLM tries to solve the user’s request. The judge LLM does not solve the request again; it reviews whether the response produced by the application meets the expected quality standards.

Let’s understand this using Wayfinder, our flight search AI application.

A user asks:

Find the cheapest flight from Bangalore to Tokyo tomorrow.

The Wayfinder agent processes the request, searches available flight data, and generates a response:

The cheapest flight is Air India AI302 at ₹410.

Now the judge LLM evaluates this response.

The judge receives:

User Query:
Find the cheapest flight from Bangalore to Tokyo tomorrow.

Application Response:
The cheapest flight is Air India AI302 at ₹410.

Retrieved Context:
Available flight information returned by the search tool.

Evaluation Criteria:
- Helpfulness
- Relevance
- Completeness
- Groundedness
- Instruction Following

The judge is not checking whether the response matches a hardcoded expected answer. Instead, it evaluates whether the response is correct, useful, and supported by the available context.

The output is an evaluation result:

Helpfulness: 5/5
Groundedness: 5/5
Instruction Following: 5/5

Explanation:
The response correctly identifies the cheapest flight
and the information is supported by the retrieved flight data.

Designing Evaluation Criteria

An LLM judge is only as effective as the criteria it uses to evaluate responses.

Just like human reviewers need a clear rubric to evaluate AI outputs consistently, an LLM judge also needs predefined evaluation criteria that describe what a good response looks like.

For Wayfinder, we use the same evaluation dimensions introduced in human evaluation:

• Helpfulness — Does the response help the user complete their task?
• Relevance — Does the response address the user’s actual request?
• Completeness — Does the response include all important information needed by the user?
• Groundedness — Is the response supported by the available context?
• Instruction Following — Does the response satisfy the user’s requirements and constraints?

These criteria form the evaluation rubric that defines how the judge LLM should assess response quality.

For example, instead of asking:

Is this response good?

the judge receives specific criteria:

Evaluate this response for:
- Helpfulness
- Groundedness
- Instruction Following

This makes evaluations more consistent and repeatable across different responses.

In the next section, we will see how these criteria are translated into an LLM-as-a-Judge implementation and how the evaluator produces structured scores and explanations.

Building the LLM-as-a-Judge Evaluator

Now that we have defined our evaluation criteria, the next step is to build an evaluator that can apply this rubric to AI-generated responses.

An LLM-as-a-Judge pipeline starts with an evaluation dataset.

The complete code is here — Wayfinder.

Creating an Evaluation Dataset

An evaluation dataset contains representative user scenarios that we want our AI application to handle.

Each evaluation sample contains:

• Input — the user query sent to the application
• Expected behavior — describes the quality expectations that guide evaluation
• Metadata — information used to organize and analyze results

For Wayfinder, our dataset contains scenarios such as:

{
 "inputs": {
 "query": "Find the cheapest flight from Bangalore to Tokyo tomorrow."
 },
 "outputs": {
 "expected_behavior": 
 "Recommend the retrieved flight with the lowest price without inventing information."
 },
 "metadata": {
 "category": "cheapest"
 }
}

The important thing is that the dataset does not contain a fixed expected answer. Instead, it describes the expected behavior of the application.

During evaluation, each dataset sample is executed against the AI application.

Designing the Judge Prompt

The quality of an LLM-as-a-Judge system depends heavily on how the judge prompt is designed.

A good judge prompt should clearly define:

1. Role
   Tell the model that it is an evaluator, not a response generator.
2. Evaluation Criteria
   Define what dimensions should be scored.
3. Evaluation Context
   Provide the information needed to judge the response.
4. Output Format
   Require structured scores and explanations.

For Wayfinder, the judge prompt follows this pattern:

You are an expert evaluator assessing the quality of an AI assistant response.

You will be given:
- The user's original query.
- The expected behavior: a description of what a good assistant response should do for this query.
- The assistant's actual response.
- The flight data retrieved by the search tool before the response was generated.

Your task is to evaluate the assistant's response against each of the following criteria.

Evaluation criteria and rubric:
{criteria_block}

Instructions:
- "Expected behavior" describes what the application was expected to do for this sample.
 It is not a model answer. Use it to understand the intent of the query.
- Evaluate the response against each criterion independently.
- When evaluating Groundedness, cross-reference the response against the retrieved flight data.
- Do not invent facts that are not present in the supplied context.
- Score every criterion from 1 to 5 using the rubric above.
- Provide a concise explanation for every criterion score.
- Provide an overall score from 1 to 5 that reflects your holistic judgment.
 Do not calculate the overall score as an average of the criteria scores.
- Provide a concise overall explanation.
"""

User Query:
{query}

Application Response:
{response}

Retrieved Context:
{context}

Provide a score from 1-5 for each criterion
and explain your reasoning.

The judge is not asked to generate a better response. Its only responsibility is to evaluate the response it receives.

Generating Structured Evaluation Results

Free-form evaluation responses from the judge are difficult to analyze automatically.

Instead, the evaluator asks the LLM Judge to return structured output. In our implementation, we enforce a predefined response schema so every evaluation produces a consistent format.

The schema contains:

• A score for each evaluation criterion
• A concise explanation for each score
• An overall score and explanation

Example:

{
 "criteria_scores": [
 {
 "criterion": "Helpfulness",
 "score": 5,
 "explanation": "The response directly answers the user's request."
 },
 {
 "criterion": "Groundedness",
 "score": 5,
 "explanation": "The response is supported by the provided context."
 },
 {
 "criterion": "Instruction Following",
 "score": 5,
 "explanation": "The response satisfies the user's requirements."
 }
 ],
 "overall_score": 5
}

Implementing the Judge

With the evaluation dataset, criteria, judge prompt, and structured output format in place, we can now implement the LLM-as-a-Judge evaluator.

The evaluator acts as a reusable component that takes an AI application’s output and evaluates it using the predefined rubric.

At a high level, the judge performs four steps:

1. Build the evaluation context
2. Send the evaluation request to the judge LLM
3. Parse the structured response
4. Return the evaluation result

The evaluator receives:

User Query
Application Response
Expected Behavior
Retrieved Context
Evaluation Criteria

and produces:

Criterion Scores
+
Explanations
+
Overall Assessment

The application code exposes a simple interface:

result = judge.evaluate(
 query=query,
 expected_behavior=expected_behavior,
 response=response,
 retrieved_flights=retrieved_context
)

Internally, the evaluator builds the judge prompt:

messages = [
 {
 "role": "system",
 "content": judge_instructions
 },
 {
 "role": "user",
 "content": evaluation_input
 }
]

The system prompt defines the judge’s role, evaluation criteria and scoring guidelines. The user message provides the specific evaluation sample, including the query, response and supporting context.

The LLM response is then parsed into the structured evaluation model:

JudgeResult(
 criteria_scores=[
 CriterionScore(
 criterion="Groundedness",
 score=5,
 explanation="The response is supported by the retrieved context."
 )
 ],
 overall_score=5,
 overall_explanation="The response satisfies the user's request."
)

Keeping the judge as a separate component makes it reusable. The same evaluator can be applied to different versions of an AI application using the same evaluation dataset and criteria.

This allows teams to measure whether changes actually improve the system:

• Did a new prompt improve response quality?
• Did a model upgrade produce better responses?
• Did a retrieval change improve groundedness?

Instead of manually reviewing responses after every change, teams can run repeatable evaluations and compare results across versions.

Running the LLM-as-a-Judge Evaluation

The complete implementation used throughout this article is available in the companion GitHub repository. To follow along with exactly the same code shown in this article, check out the v0.4.0 release. This ensures the code, commands, and screenshots remain consistent over time.

To run the examples locally, clone the repository and checkout the release containing the LLM-as-a-Judge implementation.

git clone https://github.com/DivakarUngatla/wayfinder.git
cd wayfinder
git checkout v0.4.0-llm-judge

Install the project dependencies:

uv sync

Before running the evaluation, configure the required environment variables.

Create a .env file:

cp .env.example .env

Add your OpenAI API key:

OPENAI_API_KEY=<your-api-key>

For LangSmith experiments later in this article, also configure:

LANGSMITH_API_KEY=<your-api-key>
LANGSMITH_TRACING=true

Running the Evaluation

The LLM-as-a-Judge evaluator uses the evaluation dataset created earlier. The complete dataset containing evaluation scenarios, expected behavior and metadata is available in the companion repository:

examples/llm_judge_evaluation/llm_judge_evaluation_dataset.jsonl

The evaluator loads each sample from this dataset, runs the application, and passes the generated response to the LLM judge for evaluation.

uv run examples/llm_judge_evaluation/local_evaluation.py

For each evaluation sample, the judge evaluates the generated response against the predefined criteria and returns structured scores along with explanations.

For example, consider the arrival-time constraint scenario:

User Query:
Which flight from Bangalore to Tokyo arrives before 8 PM?

The judge evaluates the response across criteria that is provided to it.

After evaluating all samples in the dataset, the pipeline generates an overall evaluation summary.

While this run completed successfully, evaluation becomes especially valuable when it detects regressions after application changes.

Detecting Regressions with LLM-as-a-Judge

After establishing the baseline evaluation, we can introduce a controlled change to simulate a regression.

In a real AI system, changes to prompts, models or retrieval logic can unintentionally affect response quality.

For this example, let us temporarily modify the response-generation prompt. Change the below line in the

If the user asks for cheapest, select the flight with the lowest price among flights satisfying all constraints.

to

If the user asks for cheapest, provide the two cheapest available options.

The evaluation dataset remains unchanged. Only the application behavior changes.

Running the evaluation again produces something as shown below

The LLM judge identifies that the response is technically correct but does not fully satisfy the user’s intent.

The response identifies the cheapest flight correctly, but it provides an additional flight even though the user requested only the cheapest option.

The evaluator detects this through criteria such as:

• Helpfulness
• Relevance
• Instruction Following

This demonstrates an important advantage of LLM-as-a-Judge: it can evaluate whether a response satisfies the user’s intent, not just whether the output contains valid data.

Scaling Evaluations with LangSmith

Once we start running evaluations repeatedly across different application versions, we need a way to track experiments and compare results. Evaluation platforms like LangSmith provide the required infrastructure.

The same evaluation dataset used for local evaluation can be uploaded to LangSmith.

Uploading the Evaluation Dataset

The evaluation dataset contains:

• Inputs — user queries sent to the application
• Outputs — expected behavior used by the judge
• Metadata — information used to organize evaluation samples

The dataset used in this article is available in the companion repository.

wayfinder_llm_judge_evaluation_v1.jsonl

Create a new dataset and upload the JSONL file.

LangSmith automatically detects the fields from the dataset.

For Wayfinder:

• Input field → inputs.query
• Output field → outputs.expected_behavior
• Metadata fields → metadata.id, metadata.category

Running the Evaluation

The LangSmith evaluation runner uses the same LLM-as-a-Judge evaluator that we built earlier.

Run:

uv run examples/llm_as_a_judge/langsmith_evaluation.py

The evaluator executes each dataset example against Wayfinder and sends the generated response to the judge.

Once the evaluation completes, click on the langsmith link displayed to view the results

For each example, we can inspect:

• Input query
• Application response
• Judge scores
• Judge explanations
• Evaluation metadata

The important point is that the evaluator itself does not change. The same LLM-as-a-Judge component can run locally or through LangSmith.

Conclusion

In this article, we explored LLM-as-a-Judge, an evaluation approach where an LLM acts as an evaluator to assess AI-generated responses against predefined criteria.

We built an end-to-end LLM-as-a-Judge evaluation pipeline for an AI-powered flight search application, covering evaluation criteria, judge prompts, structured evaluation results, and running evaluations locally and through LangSmith.

Unlike traditional rule-based checks, LLM-as-a-Judge can evaluate whether AI responses satisfy user intent while considering the context available to the application. It provides both scores and explanations, helping teams understand and improve AI application quality.

What’s Next?

LLM-as-a-Judge enables powerful offline evaluation using curated scenarios. However, offline evaluation only covers predefined cases. Production AI systems also need evaluation approaches that learn from real user interactions and live application behavior.

In the next article, we’ll explore Online Evaluation, where we’ll learn how to evaluate AI applications using real-world usage data and feedback signals.

AI Evaluation Series

Follow along as we build a complete AI evaluation toolkit — from deterministic rule-based checks to human evaluation, LLM-as-a-Judge, evaluation datasets, and production-scale evaluation workflows.

• ✅ Part 1: Software Tests vs AI Evals
• ✅ Part 2: Understanding AI Evaluation
• ✅ Part 3: Rule-Based Evaluation
• ✅ Part 4: Human Evaluation
• ✅ Part 5: LLM-as-a-Judge
• ▶️ ⏳ Part 6: Online Evaluation (Coming Soon)
• ⏳ Part 7: Comparing Evaluation Experiments (Coming Soon)

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI