Author(s): Gaurav Bhardwaj

Originally published on Towards AI.

LLM-as-a-Judge: The Complete Guide to Automated Evaluation at Scale with Azure

Introduction: Why We Need Automated Judges

Every day, AI systems generate billions of outputs — chatbot responses, code suggestions, translations, summaries, and creative content. But how do we know if those outputs are actually good?

Traditionally, the answer was human evaluation: hire experts, have them rate outputs on quality, accuracy, and helpfulness. This works beautifully — until you need to evaluate 10,000 outputs per hour. Or 100,000. Or a million per minute.

LLM-as-a-Judge is a paradigm shift: instead of humans grading AI outputs, we use another LLM to do the grading. Think of it as hiring a very fast, very consistent (but imperfect) evaluator that never sleeps and can read 16,000 responses per second.

What Exactly Is LLM-as-a-Judge?

At its simplest, LLM-as-a-Judge means:

Using a Large Language Model to evaluate, score, or compare outputs based on defined criteria.

Here’s the formal way to think about it:

Evaluation = LLM(Input + Context)

Where:

• Input = the artifact you want evaluated (a chatbot response, a code snippet, a summary).
• Context = your evaluation criteria, rubric, examples, and instructions.
• Evaluation = the judgment (a score, a choice, a label, or a detailed critique).

A Simple Example

Imagine you have a customer support chatbot. A user asks: “How do I reset my password?”

Your chatbot responds: “Click on ‘Forgot Password’ on the login page, enter your email, and follow the link sent to your inbox.”

An LLM judge would evaluate this response like so:

Prompt to the Judge:
"Rate the following customer support response on a scale of 1-5 for 
helpfulness, accuracy, and clarity.

User question: How do I reset my password?
Response: Click on 'Forgot Password' on the login page, enter your email...

Provide your rating and a brief explanation."

Judge Output:
Helpfulness: 5/5 — Directly answers the question with actionable steps.
Accuracy: 4/5 — Correct but doesn't mention the 2FA verification step.
Clarity: 5/5 — Simple, sequential instructions easy to follow.
Overall: 4.7/5

That’s LLM-as-a-Judge in action. Now, let’s multiply that by a million times per minute.

The Four Ways LLMs Can Judge

1. Scoring (Point-based)

The judge assigns a numerical score on a defined scale.

Best for: Continuous quality monitoring, leaderboard rankings, regression testing.

2. Binary (Yes/No)

The judge makes a simple pass/fail decision.

Best for: High-volume filtering, safety gates, automated pipelines.

3. Pairwise Comparison

The judge picks a winner between two outputs.

"Which response better answers the user's question?
Response A: [...]
Response B: [...]
Winner: Response A / Response B / Tie"

Best for: A/B testing models, choosing the best candidate from multiple generations, RLHF training data.

4. Multi-Choice Classification

The judge selects from predefined categories.

"Classify this response: 
[A] Fully correct 
[B] Partially correct 
[C] Off-topic 
[D] Harmful/inappropriate"

Best for: Content categorization, error taxonomy, routing decisions.

Why Not Just Use Humans?

The honest truth: LLM judges are not perfect replacements for humans. They have biases. But for 90%+ of evaluation workloads at scale, they are good enough — and orders of magnitude cheaper and faster.

The winning strategy is humans for calibration + LLMs for scale: humans create rubrics and validate a sample, then LLMs apply those rubrics to millions of evaluations.

The Reliability Challenge: When Can You Trust the Judge?

This is the central question. An unreliable judge is worse than no judge — it gives false confidence.

Known Biases in LLM Judges

Strategies to Build Reliable Judges

1. Detailed rubrics — Don’t just say “rate quality.” Define exactly what 1, 3, and 5 mean with concrete examples.
2. Chain-of-thought — Ask the judge to explain its reasoning before giving a final score.
3. Multi-judge ensemble — Use 3 different models, take the majority vote.
4. Calibration sets — Test your judge against 200+ human-labeled examples. Require >85% agreement.
5. Temperature = 0 — For consistency.

Building an LLM Judge System on Azure

Handling 1 million requests per minute (approx. 16,600 requests per second) requires an enterprise-grade architecture. Synchronous APIs will collapse under this weight. You need buffers, robust partitioning, and failover routing.

The Azure Stack Architecture

• Front Door / Ingestion: Azure API Management (APIM)
• Shock Absorber / Buffer: Azure Event Hubs (64+ partitions)
• Compute / Workers: Azure Kubernetes Service (AKS) scaled via KEDA
• The LLM Brain: Azure OpenAI Service (Provisioned Throughput Units)
• State & Audit: Azure Cache for Redis & Azure Cosmos DB

Layer 1: Ingestion & Smart Routing

At ~16,700 requests per second, Azure API Management (APIM) sits at the front to authenticate callers, rate-limit, and validate request schemas.

Instead of waiting for an LLM response synchronously, APIM immediately drops the payload into Azure Event Hubs. Event Hubs acts as a durable, partitioned stream (Kafka-compatible), ensuring that traffic bursts do not overwhelm your backend workers.

Layer 2: Processing (AKS & Tiered Routing)

Not all evaluations are equal. A simple safety check doesn’t need GPT-4o. Azure Kubernetes Service (AKS) runs the evaluation workers, pulling messages from Event Hubs. KEDA (Kubernetes Event-Driven Autoscaling) watches the Event Hub lag and dynamically spins pods up or down.

These workers utilize tiered routing to save costs:

• Tier 1: Binary (yes/no) → GPT-4o-mini ($0.001/eval)
• Tier 2: Scoring (1–10) → GPT-4o ($0.01/eval)
• Tier 3: Complex (Ensemble) → Multiple Models ($0.05/eval)

Layer 3: The LLM Brain (Azure OpenAI at Scale)

You cannot hit a single pay-as-you-go Azure OpenAI endpoint at this scale; you will hit rate limits instantly. You must use Provisioned Throughput Units (PTUs), which provide reserved, dedicated compute capacity.

To handle scale and resilience, use APIM Backend Load-Balanced Pools. You configure a primary backend pool (e.g., East US PTU) and a secondary pool (e.g., West Europe PTU). By attaching a Circuit Breaker rule in APIM (e.g., tripping after three 5xx errors in 15 seconds), APIM automatically halts traffic to a failing region and seamlessly fails over to the secondary priority group.

Layer 4: Caching

In real workloads, 15–30% of evaluation requests are duplicates.

• Level 1 (Exact Match): Azure Cache for Redis hashes the input + rubric.
• Level 2 (Semantic): Azure AI Search finds vector similarity > 0.98.
• Impact: A 25% cache hit rate at 1M req/min saves hundreds of thousands of dollars monthly.

Layer 5: Cosmos DB & The Partitioning Bottleneck

To store 16,600+ results per second with sub-10ms latency, your Azure Cosmos DB partition key strategy must be flawless to avoid hot partitions.

• Bad Partition Key: /type (All evals hit one physical partition, causing throttling).
• Good Partition Key (SaaS): /tenantId combined with hierarchical partitioning or a composite key (/tenantId_date).
• Good Partition Key (Firehose): A Random Suffix Strategy (e.g., /evalId-hash). Distributing writes across a synthetic partition key ensures Request Unit (RU) consumption is spread perfectly across all physical partitions.

Making It Reliable: The Ensemble Pattern

For high-stakes evaluations (content safety, legal compliance, medical accuracy), a single judge isn’t enough. You build a 3-Judge Panel.

// Simplified C# example of ensemble evaluation
public async Task<EvalResult> EvaluateWithEnsemble(EvalRequest request)
{
 // Run 3 judges in parallel
 var tasks = new[]
 {
 _judgeA.Evaluate(request), // GPT-4o
 _judgeB.Evaluate(request), // Fine-tuned judge model
 _judgeC.Evaluate(request), // GPT-4o (different prompt variant)
 };
 
 var results = await Task.WhenAll(tasks);
 
 // Majority vote for discrete judgments
 var finalScore = results
 .Select(r => r.Score)
 .GroupBy(s => s)
 .OrderByDescending(g => g.Count())
 .First().Key;
 
 // Flag for human review if judges disagree significantly
 var disagreement = results.Max(r => r.Score) - results.Min(r => r.Score);
 var needsHumanReview = disagreement > 3; // On a 1-10 scale
 
 return new EvalResult
 {
 Score = finalScore,
 NeedsHumanReview = needsHumanReview
 };
}

Getting Started: Your First LLM Judge in C#

You don’t need the full multi-region architecture to start. Here’s the minimal viable judge using the modern Azure OpenAI SDK.

// C# with Azure.AI.OpenAI SDK
var client = new AzureOpenAIClient(
 new Uri("https://your-instance.openai.azure.com/"),
 new DefaultAzureCredential());

var chatClient = client.GetChatClient("gpt-4o");

var result = await chatClient.CompleteChatAsync(new[]
{
 new SystemChatMessage(rubricPrompt),
 new UserChatMessage($"Question: {question}\nResponse: {response}")
}, new ChatCompletionOptions 
{ 
 Temperature = 0,
 ResponseFormat = ChatResponseFormat.CreateJsonSchemaFormat(
 "evaluation_schema",
 jsonSchemaDocument,
 "Format for the LLM evaluation response")
});

var evaluation = JsonSerializer.Deserialize<EvalResult>(result.Content[0].Text);
await _cosmosContainer.CreateItemAsync(evaluation);

Critical Note: Using Structured Outputs (ChatResponseFormat.CreateJsonSchemaFormat) is absolutely mandatory for LLM judges. If you do not enforce a strict JSON schema, your parsing logic will inevitably break when the model decides to add "Here is your evaluation:" before the JSON payload.

Cost Reality Check

Let’s be candid about what 1M evaluations/minute costs:

The biggest cost driver is Azure OpenAI tokens. Optimize by:

1. Using the smallest model that works — GPT-4o-mini handles 60%+ of simple evals.
2. Caching aggressively — Avoid re-evaluating exact inputs.
3. Batching — Evaluate 5 items in one prompt instead of 5 separate calls.
4. Fine-tuning — A fine-tuned smaller model can match GPT-4o quality at a fraction of the PTU footprint.

Key Takeaways

1. LLM-as-a-Judge is a force multiplier, not a human replacement. Humans calibrate the system; LLMs execute it at scale.
2. Reliability requires engineering. Don’t just call an API. Mitigate linguistic and positional biases, use robust rubrics, and implement ensembles.
3. Design for failure. At scale, use Event Hubs for buffering and APIM Circuit Breakers for regional failover.
4. Protect your data tier. Cosmos DB will bottleneck if you don’t use high-cardinality or randomized synthetic partition keys.

Reference Documentation & Citations

To dive deeper into the architectural components and academic foundations referenced in this article, consult the following official documentation:

• Academic Foundation: A Survey on LLM-as-a-Judge (arXiv:2411.15594) — Comprehensive framework on the reliability, biases, and definitions of automated evaluation.
• Routing & Failover: Backends in API Management (Microsoft Learn) — Official documentation on configuring load-balanced pools and circuit breaker rules to protect backend services.
• Scale & Storage: Partitioning and horizontal scaling in Azure Cosmos DB (Microsoft Learn) — Guidelines on selecting logical partition keys and managing 20GB limits.
• Compute & Cost: Provisioned throughput billing and cost management (Microsoft Learn) — Details on PTU deployments, hourly billing, and capacity reservations for Azure OpenAI.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI