LLM APIs are the new infrastructure layer. If your product calls a model API in the critical path of a user interaction, its latency is your latency. After shipping production LLM features at DeveloperLab and consulting on several AI-powered products, we compiled structured benchmarks across the two dominant providers.
What We Measured
We ran 10,000 completion requests against each provider across four benchmark categories:
- Short completion (50 output tokens, simple factual prompt)
- Long completion (1,000 output tokens, detailed explanation)
- JSON mode (structured output extraction from a 2KB document)
- Streaming (time-to-first-token for streaming responses)
Models tested:
- OpenAI:
gpt-4oandgpt-4o-mini - Anthropic:
claude-3-5-sonnet-20241022andclaude-3-haiku-20240307
Temperature: 0 for all tests (deterministic, maximally cacheable).
Time-to-First-Token (TTFT)
TTFT is the most user-perceptible latency metric for streaming UIs. It's the time from "send request" to "first character appears."
| Model | p50 TTFT | p95 TTFT | p99 TTFT |
|---|---|---|---|
| gpt-4o | 620 ms | 1,840 ms | 3,120 ms |
| gpt-4o-mini | 290 ms | 780 ms | 1,340 ms |
| claude-3-5-sonnet | 780 ms | 2,100 ms | 4,200 ms |
| claude-3-haiku | 210 ms | 560 ms | 920 ms |
Key finding: gpt-4o-mini and claude-3-haiku are dramatically faster for TTFT. If you're building a conversational UI, the smaller models feel snappier even if output quality is slightly lower.
Total Completion Time (Non-Streaming)
For batch processing, background jobs, or synchronous API responses, total completion time matters more than TTFT.
Short Completion (50 tokens)
| Model | p50 | p95 | p99 |
|---|---|---|---|
| gpt-4o | 1,120 ms | 2,890 ms | 5,340 ms |
| gpt-4o-mini | 480 ms | 1,200 ms | 2,100 ms |
| claude-3-5-sonnet | 1,380 ms | 3,200 ms | 6,100 ms |
| claude-3-haiku | 340 ms | 890 ms | 1,600 ms |
Long Completion (1,000 tokens)
| Model | p50 | p95 | p99 |
|---|---|---|---|
| gpt-4o | 8,400 ms | 18,200 ms | 28,100 ms |
| gpt-4o-mini | 3,200 ms | 7,100 ms | 11,200 ms |
| claude-3-5-sonnet | 9,800 ms | 21,000 ms | 32,400 ms |
| claude-3-haiku | 2,100 ms | 4,800 ms | 8,200 ms |
At long completions, Anthropic's claude-3-5-sonnet shows higher latency than gpt-4o at every percentile. This is partly architectural — Claude Sonnet targets quality-per-token over raw throughput.
Timeout Rates and Error Patterns
Over 10,000 requests per model, using a 30-second hard timeout:
| Model | Timeout rate | 429 rate | 500 rate |
|---|---|---|---|
| gpt-4o | 0.08% | 0.31% | 0.12% |
| gpt-4o-mini | 0.02% | 0.18% | 0.04% |
| claude-3-5-sonnet | 0.14% | 0.22% | 0.09% |
| claude-3-haiku | 0.01% | 0.09% | 0.02% |
Both providers show sub-1% error rates under normal conditions. Claude Sonnet had slightly higher timeouts, consistent with its longer completion times pushing closer to the 30-second cutoff on complex prompts.
Rate Limits in Practice
This is where the operational differences become sharp:
OpenAI (Tier 2 account):
gpt-4o: 450 RPM / 150K TPMgpt-4o-mini: 2,000 RPM / 2M TPM- Rate limit errors are granular (
x-ratelimit-*headers with reset times)
Anthropic (Scale tier):
claude-3-5-sonnet: 400 RPM / 400K TPMclaude-3-haiku: 2,000 RPM / 1M TPM- Rate limit errors include
retry-afterheaders but less granular than OpenAI
For burst-heavy workloads (user-facing chat), OpenAI's gpt-4o-mini at 2,000 RPM is a clear advantage. For moderate-volume batch processing, both providers handle the load comfortably.
JSON Mode Reliability
We extracted a structured output from a 2KB input document, asking both models to return a specific 10-field JSON schema.
| Model | Valid JSON rate | Schema compliance rate |
|---|---|---|
| gpt-4o (json mode) | 99.97% | 98.4% |
| gpt-4o-mini (json mode) | 99.91% | 94.2% |
| claude-3-5-sonnet | 99.89% | 97.8% |
| claude-3-haiku | 98.7% | 89.4% |
OpenAI's structured JSON mode (enforced via their API) outperforms Anthropic's prompt-guided approach for strict schema compliance. Anthropic announced tool use and structured output improvements in late 2025; if you're building with Claude today, use function calling over raw JSON prompting.
Production Recommendation
| Use Case | Recommendation |
|---|---|
| Real-time user chat | gpt-4o-mini (lowest TTFT) |
| Complex reasoning tasks | claude-3-5-sonnet or gpt-4o (quality-focused) |
| Background batch jobs | claude-3-haiku (cost × speed) |
| Structured data extraction | gpt-4o with JSON mode |
| High-concurrency apps | gpt-4o-mini (highest RPM limit) |
Both providers have matured significantly in 2026. The "which is better" question has largely collapsed into "which is better for your specific use case" — a much more useful frame.
All benchmarks run from a single US-East probe location. Results will vary by region, account tier, prompt complexity, and time of day (load patterns differ significantly between business hours and off-peak).