LLM APIs are not commodity infrastructure yet. Both OpenAI and Anthropic experience periodic overload events — 503 responses that last anywhere from seconds to minutes. If your product has LLM calls in the critical path of a user request, a single provider outage means a broken product.
The solution is a failover middleware that transparently retries against a secondary provider when the primary returns a specific error class. This guide builds a production-ready implementation in TypeScript/Node.js.
The Architecture
Client Request
│
▼
┌─────────────────────┐
│ LLM Middleware │
│ (your code) │
├─────────────────────┤
│ 1. Try OpenAI │──── 200 OK ──────────► Return to client
│ 2. On 503/429 → │
│ Try Anthropic │──── 200 OK ──────────► Return to client
│ 3. On both fail → │
│ Return error │──── 503 ─────────────► Return to client
└─────────────────────┘
The middleware is transparent to the caller — they send a standardized request, and the middleware handles provider selection, translation, and retry logic.
Step 1: Define a Provider-Agnostic Interface
Start by defining the interface that both providers must satisfy. This is your internal contract — callers never touch provider SDKs directly.
export interface LLMRequest {
systemPrompt?: string;
userMessage: string;
maxTokens?: number;
temperature?: number;
jsonMode?: boolean;
}
export interface LLMResponse {
content: string;
model: string;
provider: "openai" | "anthropic";
inputTokens: number;
outputTokens: number;
latencyMs: number;
}
export interface LLMProvider {
name: "openai" | "anthropic";
call(req: LLMRequest): Promise<LLMResponse>;
}
Step 2: Implement the OpenAI Provider
import OpenAI from "openai";
export class OpenAIProvider implements LLMProvider {
readonly name = "openai" as const;
private client: OpenAI;
constructor(apiKey: string) {
this.client = new OpenAI({ apiKey });
}
async call(req: LLMRequest): Promise<LLMResponse> {
const start = Date.now();
const messages: OpenAI.ChatCompletionMessageParam[] = [];
if (req.systemPrompt) {
messages.push({ role: "system", content: req.systemPrompt });
}
messages.push({ role: "user", content: req.userMessage });
const completion = await this.client.chat.completions.create({
model: "gpt-4o",
messages,
max_tokens: req.maxTokens ?? 1024,
temperature: req.temperature ?? 0.7,
response_format: req.jsonMode ? { type: "json_object" } : undefined,
});
const choice = completion.choices[0];
return {
content: choice.message.content ?? "",
model: completion.model,
provider: "openai",
inputTokens: completion.usage?.prompt_tokens ?? 0,
outputTokens: completion.usage?.completion_tokens ?? 0,
latencyMs: Date.now() - start,
};
}
}
Step 3: Implement the Anthropic Provider
import Anthropic from "@anthropic-ai/sdk";
export class AnthropicProvider implements LLMProvider {
readonly name = "anthropic" as const;
private client: Anthropic;
constructor(apiKey: string) {
this.client = new Anthropic({ apiKey });
}
async call(req: LLMRequest): Promise<LLMResponse> {
const start = Date.now();
const systemPrompt = req.jsonMode
? `${req.systemPrompt ?? ""}\n\nRespond with valid JSON only.`.trim()
: req.systemPrompt;
const message = await this.client.messages.create({
model: "claude-3-5-sonnet-20241022",
max_tokens: req.maxTokens ?? 1024,
system: systemPrompt,
messages: [{ role: "user", content: req.userMessage }],
temperature: req.temperature ?? 0.7,
});
const content = message.content
.filter((b) => b.type === "text")
.map((b) => (b as Anthropic.TextBlock).text)
.join("");
return {
content,
model: message.model,
provider: "anthropic",
inputTokens: message.usage.input_tokens,
outputTokens: message.usage.output_tokens,
latencyMs: Date.now() - start,
};
}
}
Step 4: The Failover Middleware
This is the core — a middleware class that tries the primary provider and falls back to secondary on retryable errors.
// Errors that warrant failover (not permanent client errors)
const RETRYABLE_STATUS_CODES = new Set([429, 500, 502, 503, 504]);
export class LLMFailoverMiddleware {
private primary: LLMProvider;
private secondary: LLMProvider;
private onFailover?: (reason: string) => void;
constructor(
primary: LLMProvider,
secondary: LLMProvider,
options?: { onFailover?: (reason: string) => void }
) {
this.primary = primary;
this.secondary = secondary;
this.onFailover = options?.onFailover;
}
async call(req: LLMRequest): Promise<LLMResponse> {
try {
return await this.primary.call(req);
} catch (primaryErr) {
const shouldFailover = this.isRetryableError(primaryErr);
if (!shouldFailover) {
throw primaryErr; // Don't failover on auth errors, invalid requests, etc.
}
const reason = this.errorSummary(primaryErr);
this.onFailover?.(reason);
console.warn(`[LLM] Primary (${this.primary.name}) failed: ${reason}. Failing over to ${this.secondary.name}.`);
// Let the secondary error propagate naturally if it also fails
return await this.secondary.call(req);
}
}
private isRetryableError(err: unknown): boolean {
if (err instanceof Error) {
// OpenAI SDK error
if ("status" in err && typeof (err as any).status === "number") {
return RETRYABLE_STATUS_CODES.has((err as any).status);
}
// Network error (DNS failure, connection refused, timeout)
if (err.message.includes("fetch") || err.message.includes("ECONNREFUSED")) {
return true;
}
}
return false;
}
private errorSummary(err: unknown): string {
if (err instanceof Error) {
const status = "status" in err ? ` (HTTP ${(err as any).status})` : "";
return `${err.message}${status}`;
}
return String(err);
}
}
Step 5: Wire It Up
const llm = new LLMFailoverMiddleware(
new OpenAIProvider(process.env.OPENAI_API_KEY!),
new AnthropicProvider(process.env.ANTHROPIC_API_KEY!),
{
onFailover: (reason) => {
// Send to your observability system
metrics.increment("llm.failover", { reason });
alerts.warn(`LLM failover triggered: ${reason}`);
},
}
);
// Usage in your API handler
app.post("/api/generate", async (req, res) => {
try {
const result = await llm.call({
systemPrompt: "You are a helpful assistant.",
userMessage: req.body.message,
maxTokens: 512,
});
res.json({ text: result.content, provider: result.provider });
} catch (err) {
res.status(503).json({ error: "Both LLM providers unavailable" });
}
});
Adding a Circuit Breaker
If OpenAI has been returning errors for the last 10 minutes, you don't want to keep hammering it. A circuit breaker tracks failure rate and short-circuits to the secondary provider without even trying primary when it's in an open state.
class CircuitBreaker {
private failures = 0;
private lastFailure = 0;
private readonly threshold: number;
private readonly cooldownMs: number;
constructor(threshold = 5, cooldownMs = 60_000) {
this.threshold = threshold;
this.cooldownMs = cooldownMs;
}
get isOpen(): boolean {
if (this.failures < this.threshold) return false;
return Date.now() - this.lastFailure < this.cooldownMs;
}
recordSuccess(): void { this.failures = 0; }
recordFailure(): void {
this.failures++;
this.lastFailure = Date.now();
}
}
The circuit opens after 5 failures and stays open for 60 seconds, during which all traffic goes directly to the secondary provider.
Observability: What to Log
At minimum, emit these signals for every LLM call:
interface LLMCallEvent {
provider: string; // Which provider served the request
model: string; // Exact model version
latencyMs: number; // Wall-clock latency
inputTokens: number; // Prompt tokens
outputTokens: number; // Completion tokens
failoverOccurred: boolean; // Did we hit the secondary?
error?: string; // Error message if both failed
}
Track failoverOccurred rate over time — a spike tells you before the status page does that your primary provider is struggling.
This pattern has kept LLM-powered features available through multiple provider incidents. The implementation cost is one afternoon; the payoff is a feature that stays up when the AI infrastructure ecosystem has a bad day.