Bound AI Generation Concurrency with a Node.js Worker Pool

A summarization batch should not start every generation request at once. A worker pool lets the application control how many requests are active while collecting one outcome per input. This example also preserves input order when jobs finish in a different order and keeps an individual failure from discarding successful results.

The generation adapter is a timed fixture. It represents the asynchronous boundary where an application would invoke its separately configured AI client. No model request, account credential or paid API is involved.

Prerequisites

Use a supported Node.js LTS release. This program was executed on Node.js v26.13.0. Save it as pool.mjs so Node loads its imports as an ES module. All imports are built in.

import assert from 'node:assert/strict';
import { setTimeout as sleep } from 'node:timers/promises';
async function pool(jobs, limit, invoke) {
  if (!Number.isInteger(limit) || limit < 1) throw new Error('bad limit');
  let cursor = 0;
  const results = new Array(jobs.length);
  async function worker() {
    while (cursor < jobs.length) {
      const index = cursor++;
      try {
        results[index] = {ok: true, value: await invoke(jobs[index])};
      } catch {
        results[index] = {ok: false, error: 'generation failed'};
      }
    }
  }
  await Promise.all(Array.from({length: Math.min(limit, jobs.length)}, worker));
  return results;
}
let active = 0, peak = 0;
async function fixture(job) {
  active++; peak = Math.max(peak, active);
  try {
    await sleep(job === 'slow' ? 20 : 2);
    if (job === 'bad') throw new Error('fixture failure');
    return `summary:${job}`;
  } finally { active--; }
}
const result = await pool(['slow','bad','fast'], 2, fixture);
assert.deepEqual(result, [
  {ok:true,value:'summary:slow'},
  {ok:false,error:'generation failed'},
  {ok:true,value:'summary:fast'}
]);
assert.equal(peak, 2);
assert.equal(active, 0);
assert.deepEqual(await pool([], 2, fixture), []);
await assert.rejects(() => pool([], 0, fixture));
console.log(JSON.stringify(result));
console.log(`peak=${peak} active=${active}`);

Run and expected output

node pool.mjs

[{"ok":true,"value":"summary:slow"},{"ok":false,"error":"generation failed"},{"ok":true,"value":"summary:fast"}]
peak=2 active=0

Why the limit holds

Each worker claims an index synchronously before awaiting the adapter. JavaScript executes that claim without another worker interleaving inside it. A worker requests another job only after its current invocation settles. At most two workers exist for this batch, so at most two fixture invocations are active.

Results are assigned by claimed index, rather than appended when jobs finish. The first slot therefore remains the slow job. A caught failure occupies its own slot, and the worker continues. The fixture's finally block decrements the active counter on both success and failure.

Edge cases and troubleshooting

An empty batch creates no workers. A zero or noninteger limit is rejected. Keep the input array unchanged during processing; this small interface assumes a finite, stable batch. It stores all results in memory and is not a streaming queue.

A concurrency ceiling is different from a requests-per-minute or token budget. A real adapter also needs transport deadlines and cancellation. A promise that never settles can occupy a worker indefinitely; the fixture cannot reproduce that production failure. Blind retries can duplicate billable work, so define retry and idempotency behavior separately. Timer durations here arrange asynchronous work and are not latency benchmarks.

Primary references

See the official Node.js timer API, assertion API and release support table. More examples appear in node.js and AI.