Author(s): Anna Jey

Originally published on Towards AI.

LLM Provider Quirks: Build Compatibility Tests Before You Trust Model Abstraction

Every multi-model AI app eventually reaches the same uncomfortable moment: the abstraction layer works in the demo, then breaks on the first serious provider swap.

The failure is rarely dramatic at first. A tool call arrives in a slightly different shape. A schema that passed with one model becomes too strict for another. A streaming parser assumes the wrong event order. A retry turns a harmless timeout into a duplicate write. The dashboard still says the provider returned a successful response, but the product quietly does the wrong thing.

This is why LLM provider quirks deserve their own test suite. You need compatibility tests that prove your application contract survives the weird parts of each provider.

That matters more now because the market is moving toward model diversity. Microsoft documentation says Microsoft 365 Copilot can use models from Microsoft, Anthropic, and OpenAI depending on the feature and need. Recent reporting also points to Microsoft expanding its in-house MAI model usage to improve cost efficiency and reduce dependence on external frontier labs. At the same time, OpenAI, Anthropic, and Google continue to ship richer tool use, structured output, and agent APIs with different assumptions baked into each surface.

The practical lesson for developers is simple: model abstraction is useful, but only after you know what you are abstracting.

The Provider-Swap Trap

The easy version of model abstraction looks like this:

const answer = await llm.generate({
 model: "some-model",
 prompt: userPrompt
});

That wrapper is fine for plain text. It becomes fragile when your app depends on tool calls, structured outputs, streaming, safety refusals, caching, files, images, function schemas, or provider-specific reasoning controls.

Developers often discover this after they have already promised portability. The team starts with one provider. Finance asks for a cheaper model on high-volume tasks. Security asks for a model that keeps data in a specific environment. Product wants a long-context model for large documents. Leadership sees Microsoft-style model choice and asks, “Can we do that too?”

The answer is yes, but the hidden work is not the adapter. The hidden work is the contract.

A good LLM abstraction does not hide provider differences. It makes the differences explicit, testable, and safe to route around.

OpenAI’s function calling guide describes tool calling as a multi-step loop where the model requests a tool, your application executes it, and you send the result back for a final response. Anthropic’s Claude docs describe a similar client-tool loop, but with Claude-specific content blocks, stop reasons, and tool-result handling. Google’s Gemini function calling docs frame the model as returning function arguments for external actions, knowledge augmentation, and capability extension. These are close enough to sound interchangeable and different enough to break code that assumes they are identical.

What Counts as an LLM Provider Quirk?

A provider quirk is not a bug. It is a behavior your app must understand before it can depend on a model safely.

Some quirks are documented API differences. Some are model behavior patterns. Some only show up under load, long context, unusual schemas, partial streams, or safety-sensitive prompts. You do not need to memorize every provider detail. You need to identify the categories that can break your product.

Tool Call Shape

One provider may return tool calls as separate response items. Another may return content blocks. A wrapper may normalize the happy path but leak provider-specific fields during parallel tool calls or partial failures. Your app should not assume that “tool call” means one universal object.

Schema Strictness

Structured output sounds simple: give the model a JSON Schema and receive valid JSON. In practice, provider support differs by API, model, schema keyword, strictness mode, SDK helper, and whether the schema is used for a final answer or a tool call.

OpenAI’s structured output docs distinguish schema adherence from older JSON mode and recommend Structured Outputs when available. Anthropic’s structured output docs use an output_config.format pattern with schema limitations. Google's Gemini docs describe JSON Schema support for structured outputs, with separate API modes. The words look similar. The integration details are not the same.

Streaming Events

Streaming is where many abstractions start to leak. You may receive text deltas, reasoning metadata, tool-call deltas, content block starts, content block stops, usage events, safety events, or provider-specific keepalive messages. If your parser expects one event order, a provider swap can turn a smooth UI into missing tokens or broken tool arguments.

Error Semantics

An invalid schema, blocked prompt, rate limit, overloaded model, timeout, content filter, or provider-side incident should not all become new Error("LLM failed"). Your retry logic needs to know whether to retry, fall back, ask the user for different input, page an engineer, or stop because the action might be unsafe to repeat.

Safety and Refusal Format

Refusals are product behavior, not just model behavior. If one provider returns a structured refusal field and another returns refusal text, your app still needs one user-facing policy. Otherwise the same unsafe request may produce inconsistent UI states, logs, and analytics.

Cost and Latency Controls

Some models expose reasoning effort, caching, priority inference, batch options, or task-specific modes. Others expose different levers or none at all. If your abstraction only accepts temperature and max_tokens, it may erase the exact controls that made a provider worth using.

Provider compatibility should be tested as a release workflow, not guessed during a migration.

The Compatibility Contract You Actually Need

Do not start by wrapping every SDK behind the smallest common denominator. That approach turns into a pile of exceptions.

Start by writing the contract your product needs. The contract should describe what your app expects from any model, regardless of provider. Then each provider adapter must prove it can satisfy that contract for the use cases you plan to route to it.

A practical contract includes:

• Input contract: messages, system instructions, files, images, tool definitions, schema definitions, safety context, and idempotency keys.
• Output contract: final answer, tool requests, refusal state, citations, structured payload, usage data, latency, and debug metadata.
• Error contract: retryable errors, user-fixable errors, policy refusals, provider incidents, invalid schema errors, and duplicate-action protection.
• Streaming contract: event types your UI accepts, event ordering rules, partial tool-call handling, completion signals, and parser recovery behavior.
• Quality contract: task-specific pass criteria, not generic “model answered well” scoring.

The most important part is the error contract. Many teams test output quality but forget operational behavior. That is backwards. A slightly weaker answer is annoying. A retried payment, deleted file, or invisible refusal is a production incident.

Build a Provider Compatibility Test Suite

Your compatibility suite should be small enough to run in CI and realistic enough to catch the failures that matter. It is not a replacement for deep model evals. It is the preflight check that says, “This provider can safely enter this route.”

1. Golden Prompts for Each Product Route

Pick the workflows your product actually runs: extract invoice fields, draft a pull request summary, classify support tickets, call a CRM tool, answer from documents, generate a UI component, or route an agent step.

For each workflow, save a few golden prompts with expected behavior. Do not only include clean examples. Include messy inputs, missing fields, ambiguous user requests, long context, policy-sensitive requests, and cases where the model should ask a question instead of acting.

2. Schema Torture Tests

Create schemas that represent your real payloads, then add edge cases. Test required fields, nested objects, arrays, enums, nullable values, descriptions, unknown properties, and large schemas. If you use Zod, Pydantic, or generated JSON Schema, test the generated schema exactly as your app sends it.

Your goal is not to prove that every model supports every schema. Your goal is to classify where each provider is safe.

3. Tool-Call Replay

Tool calling is a loop, so test the loop. The first model response is not enough. Verify that your adapter can:

• Detect the tool call.
• Validate arguments before execution.
• Return tool results in the provider’s required format.
• Continue until the model returns a final answer or a safe stop condition.
• Prevent duplicate execution when a retry happens after a timeout.

Use fake tools in CI. For example, a fake calendar tool can record whether a meeting would have been created without touching a real calendar. That lets you test dangerous paths without danger.

4. Streaming Parser Tests

Record streaming event samples from each provider and replay them through your parser. Include normal text, long output, tool calls, interrupted streams, provider errors, and user cancellation. If the UI depends on streaming, your compatibility suite should fail when an adapter drops tokens, emits duplicate chunks, or marks a response complete too early.

5. Failure Injection

Mock the ugly cases: rate limit, invalid request, bad schema, content policy refusal, empty response, network timeout, slow stream, malformed tool arguments, and provider outage. Then assert the product behavior. The answer should not be “log and move on.” It should be route-specific.

A Simple TypeScript Pattern

The following sketch is not a full framework. It shows the shape of a contract-first adapter. Each provider can keep its native SDK internally, but the rest of your product sees one tested result shape.

type LlmRequest = {
 route: "ticket_triage" | "invoice_extract" | "agent_step";
 messages: Array<{ role: "system" | "user" | "assistant"; content: string }>;
 tools?: ToolDefinition[];
 responseSchema?: unknown;
 idempotencyKey: string;
};

type LlmResult =
 | { type: "final"; text: string; usage: Usage; providerMeta: unknown }
 | { type: "tool_call"; calls: ToolCall[]; usage: Usage; providerMeta: unknown }
 | { type: "refusal"; reason: string; usage: Usage; providerMeta: unknown };type LlmFailure =
 | { type: "retryable"; reason: string; retryAfterMs?: number }
 | { type: "bad_request"; reason: string }
 | { type: "policy"; reason: string }
 | { type: "unsafe_retry"; reason: string };interface ProviderAdapter {
 name: string;
 supports(route: LlmRequest["route"]): boolean;
 generate(request: LlmRequest): Promise<LlmResult>;
}

Notice what is missing: no provider SDK types leak into the product route. That does not mean you throw away provider-specific features. It means you isolate them behind capabilities that tests can verify.

A provider adapter can expose metadata for observability. It can support special controls such as reasoning effort, prompt caching, or priority inference. But those controls should be mapped through route-level policy, not scattered across feature code.

Route by Capability, Not by Brand

The most common routing mistake is treating providers like interchangeable vendors with different prices. A better mental model is capability routing.

For a simple classification task, you may care about cost, latency, and stable JSON. For a coding agent, you may care about long context, tool reliability, repository instruction following, and patch quality. For a customer support workflow, refusal behavior, citation quality, and privacy boundaries may matter more than raw reasoning strength.

Build a route profile for each workflow:

• What is the user trying to accomplish?
• What must never happen?
• Does the route need tools, structured output, citations, files, images, or streaming?
• What latency is acceptable?
• What cost is acceptable per successful outcome?
• Which provider quirks are acceptable for this route?

Then promote models into routes only after they pass compatibility and quality gates. This turns “Should we use OpenAI, Claude, Gemini, or a smaller model?” into a better question: “Which tested model can satisfy this route’s contract at the best cost, latency, and risk level?”

How to Analyze Top-Ranking Content Without Copying It

If you search for LLM abstraction layers, you will find three common article types. The first says to use a unified API or gateway. The second compares model prices and context windows. The third explains how to migrate from one provider to another.

Those are useful, but they often skip the practical failure layer. They do not show the exact compatibility tests a team should run before trusting the abstraction. They underplay streaming and error semantics. They rarely treat refusals, retries, duplicate tool execution, or schema keyword differences as first-class product risks.

That is the gap this article fills: not “use an abstraction layer,” but “prove your abstraction layer handles provider quirks before it touches users.”

A Practical Rollout Plan

You do not need to rebuild your AI stack in one sprint. Start with one route that already hurts. Good candidates are high-volume extraction, support triage, document summarization, coding-agent review, or any workflow where cost, latency, or provider reliability has become visible.

First, capture twenty to fifty real examples with user data removed. Add expected outcomes and failure labels. Second, write the route contract. Third, implement one adapter for your current provider and make it pass. Fourth, add a second provider and run the same suite. Fifth, compare not only answer quality but parser behavior, retry safety, streaming stability, and cost per successful outcome.

Only then should you turn on traffic splitting. Start with shadow mode if possible. Send the same request to the candidate model without showing its answer to the user. Log compatibility failures and quality scores. When the new provider performs well, move a small percentage of low-risk traffic. Keep rollback as a configuration change, not a deploy.

The safest provider switch is boring: measured, reversible, and backed by route-level contracts.

What to Log

Provider compatibility is impossible to improve if your logs only store the final answer. For each request, capture the provider, model, route, adapter version, prompt template version, schema version, tool catalog version, latency, token usage, cache status, retry count, refusal state, tool-call count, validation errors, and final outcome.

Do not log sensitive user content unless your policy allows it. You can often store hashes, redacted samples, metadata, and failure classes instead. The goal is to answer operational questions quickly:

• Which provider fails schema validation most often on this route?
• Which adapter emits the most retryable errors?
• Are refusals rising after a model change?
• Did latency improve at the cost of more manual review?
• Which tool arguments are most often malformed?

This is also where your abstraction pays off. Once every adapter reports the same normalized failure classes, your product team can compare behavior without reading raw provider responses all day.

Where Gateways Fit

AI gateways and unified APIs can be valuable. They centralize keys, routing, rate limits, observability, and sometimes provider normalization. For many teams, using a gateway is better than wiring five SDKs directly into product code.

But a gateway does not remove your responsibility for product-level compatibility. It can tell you that a provider returned a response. It may even normalize response shape. It cannot know whether your invoice extraction route treats a missing tax ID as a user-fixable issue, a model failure, or an acceptable blank field. That is your contract.

The best pattern is layered:

• Use provider SDKs or a gateway for transport, authentication, and basic normalization.
• Use adapters for provider-specific behavior and capabilities.
• Use route contracts for product expectations.
• Use compatibility tests to decide where each model is allowed to run.

The Decision Rule

Before adding a new model to production, ask three questions.

Can it satisfy the route contract? If not, do not ship it on that route, no matter how impressive the benchmark looks.

Can the adapter explain failures in normalized terms? If not, support and operations will suffer during incidents.

Can you roll back without a deploy? If not, you are not ready for live traffic.

The teams that win with multi-model AI will not be the teams with the prettiest wrapper. They will be the teams with the clearest contracts, the most boring rollback plan, and the discipline to test provider quirks before users find them.

FAQ

What are LLM provider quirks?

LLM provider quirks are behavior differences between model APIs that can affect production apps. Common examples include tool-call shape, JSON Schema support, streaming event format, refusal handling, retry semantics, context limits, file support, and provider-specific model controls.

Do I still need compatibility tests if I use an AI gateway?

Yes. A gateway can simplify routing, keys, observability, and some normalization, but it cannot define your product’s expected behavior. You still need route-level tests for schemas, tool calls, errors, refusals, latency, and safe rollback.

Is an LLM abstraction layer a bad idea?

No. An abstraction layer is useful when it is honest about provider differences. The risky version hides everything behind a tiny generic interface. The safer version exposes capabilities, normalizes failures, and requires every provider adapter to pass compatibility tests before use.

How many test cases do I need before switching LLM providers?

Start with twenty to fifty high-quality examples for one product route. Include normal cases, messy cases, missing data, long context, safety-sensitive prompts, malformed tool arguments, timeouts, and retry scenarios. Add more cases as real failures appear.

What is the difference between model evals and compatibility tests?

Model evals measure whether a model performs a task well. Compatibility tests measure whether a provider can safely fit your application’s contract. You need both. A model can score well in an eval and still break your app through streaming, schema, retry, or refusal behavior.

Which LLM provider is best for tool calling?

There is no universal answer. OpenAI, Anthropic, Google Gemini, and other providers all support tool-like workflows, but their APIs and model behavior differ. The best provider is the one that passes your route’s tool-call, schema, error, latency, and quality tests at an acceptable cost.

Should I build my own multi-model router?

Build only the part that is specific to your product. You can use SDKs or gateways for transport and provider access, then build your own route contracts, compatibility tests, and rollout rules. That keeps your product behavior under your control without reinventing every infrastructure piece.

Sources and Further Reading

• OpenAI API: Function calling
• OpenAI API: Structured Outputs
• Anthropic Claude docs: Tool use
• Anthropic Claude docs: Structured outputs
• Google AI for Developers: Function calling with the Gemini API
• Microsoft Learn: Microsoft 365 Copilot application card

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI