Optimizing Concurrent API Health Checks: asyncio, Connection Pooling, and Batching

Probing a hundred API endpoints sounds trivial until you try to do it in under 30 seconds without hammering your own server's file descriptors, triggering rate limits on remote hosts, or — in Node.js — starving the event loop of cycles it needs for live user requests.

This post walks through the three layers of optimization we applied to DeveloperLab's health checker, along with the patterns that generalize to any high-throughput HTTP probing system.

Layer 1: From Serial to Concurrent

The baseline implementation is always serial — it's the simplest thing that could possibly work:

// Serial: one request at a time — painfully slow
for (const entry of entries) {
  const res = await fetch(entry.url, { signal: AbortSignal.timeout(5000) });
  entry.status = res.status;
}

With 199 endpoints and an average RTT of 300 ms, this takes ~60 seconds. Not viable.

The first upgrade is full concurrency — fire all requests at once:

// All-at-once: fast but dangerous
await Promise.allSettled(entries.map(probe));

This is faster (wall time drops to ~5 seconds) but it blasts 199 simultaneous connections. Many public APIs apply aggressive rate limiting at the IP level. You'll start seeing 429 Too Many Requests responses, which misrepresent the API's actual health.

Layer 2: Controlled Batching

The production approach is batched concurrency: issue N requests simultaneously, wait for the batch to complete, then move to the next batch.

const BATCH_SIZE = 25;
const DELAY_BETWEEN_BATCHES_MS = 200;

async function runHealthCheck(entries: ApiEntry[]): Promise<void> {
  for (let i = 0; i < entries.length; i += BATCH_SIZE) {
    const batch = entries.slice(i, i + BATCH_SIZE);
    await Promise.allSettled(batch.map(probeEntry));

    // Courtesy delay: avoid hammering rate limits on consecutive batches
    if (i + BATCH_SIZE < entries.length) {
      await new Promise((r) => setTimeout(r, DELAY_BETWEEN_BATCHES_MS));
    }
  }
}

Why Promise.allSettled over Promise.all? Because Promise.all rejects immediately when any promise rejects — one timed-out endpoint kills the entire batch. allSettled waits for every promise to either resolve or reject, then gives you all results. Essential for fault-tolerant probing.

Layer 3: Timeout Architecture

Timeouts have two failure modes that are easy to confuse:

Connection timeout — how long to wait for the TCP connection to establish. Read timeout — how long to wait for the server to send a response after connecting.

Node.js's fetch + AbortSignal.timeout() applies a single wall-clock timeout covering both phases:

async function probeEntry(entry: ApiEntry): Promise<void> {
  const start = Date.now();
  try {
    const res = await fetch(entry.Link, {
      method: "GET",
      signal: AbortSignal.timeout(5000), // 5s total: connect + read
      redirect: "follow",               // follow up to 20 redirects
      headers: { "User-Agent": "DeveloperLab-HealthChecker/1.0" },
    });
    entry.last_checked_status = res.status;
    entry.latency_ms = Date.now() - start;
    // Extract response headers for display
    entry.response_headers = extractHeaders(res.headers);
  } catch (err) {
    // AbortError (timeout), TypeError (network error), etc.
    entry.last_checked_status = null;
    entry.latency_ms = null;
  }
}

5 seconds is generous for a health check. In practice, 95% of reachable APIs respond in under 1 second. The long tail — APIs on cold-start serverless infrastructure, slow government endpoints — occasionally hits 3–4 seconds.

Connection Pooling in the Browser Context

One advantage of running health checks server-side (vs. browser-side) is persistent TCP connection reuse. The Node.js fetch implementation (via undici in Node 18+) maintains connection pools per origin automatically.

For probing many different origins (every API is a different domain), connection pooling helps less than you'd expect — each batch likely hits 25 different domains, meaning 25 fresh TCP handshakes. The optimizations that do help:

1. DNS caching: Node.js caches DNS lookups per session. Subsequent probes to the same domain in later batches skip the DNS round-trip.
2. HTTP/2 multiplexing: For endpoints that support HTTP/2, multiple requests to the same origin can share a single TCP connection. Rare for public APIs but increasingly common.
3. Keepalive: Reuse TCP connections across batch runs (useful when the checker runs every 20 minutes to the same endpoint set).

Measuring Latency Accurately

The naive approach — Date.now() before and after fetch() — captures total wall time including:

• DNS resolution
• TCP handshake
• TLS negotiation (for HTTPS)
• Server processing time
• Response transmission

For our purposes (displaying "is this API slow?"), total wall time is the right metric. It reflects what a real client experiences. For diagnosing why an API is slow, you'd want to instrument each phase separately using the Performance API or a tool like Wireshark.

// Wall-clock latency: what the user actually experiences
const start = performance.now();
const res = await fetch(url, { signal: AbortSignal.timeout(5000) });
const latencyMs = Math.round(performance.now() - start);

Prefer performance.now() over Date.now() — it's monotonic and has sub-millisecond resolution, so it won't produce negative deltas if the system clock is adjusted mid-request.

Result Interpretation

A 404 on a root URL (https://api.example.com/) doesn't mean the API is down — it often means the root isn't a valid endpoint. We mark any 2xx or 4xx response as "reachable" and only flag 5xx and timeouts as potential problems.

The patterns here — batched Promise.allSettled, per-entry timeouts with AbortSignal, and wall-clock latency measurement — apply directly to any monitoring system you build, regardless of language or framework.