Terminal-Proof API Retries in BullMQ Workers

A completed request is not a completed job

The awkward failure happens after generation: your worker receives an answer, saves it, then crashes before acknowledging the job. The retry must reuse that saved answer rather than publish another result. BullMQ recommends designing retries around an unchanged final application state. That is the guarantee we will build—not exactly-once API execution. ([docs.bullmq.io](https://docs.bullmq.io/patterns/idempotent-jobs?utm_source=openai))

1. Prepare Redis and the Fluxo client

Use Bash, Node.js 24, and Docker. Run each command paragraph separately.

mkdir fluxo-retry-lab && cd fluxo-retry-lab && npm init -y && npm install bullmq@5 openai better-sqlite3

docker run -d --name fluxo-retry-redis -p 127.0.0.1:6379:6379 redis:7-alpine redis-server --appendonly yes

Fluxo provides one key and one base URL, https://api.fluxorelay.com, across its model catalog. We will use GPT-6 Astra; copy its exact API identifier from your account rather than guessing a slug. Enter the complete OpenAI SDK base shown in your setup, including any required path prefix.

read -rs -p 'Fluxo key: ' FLUXO_API_KEY; echo; export FLUXO_API_KEY

read -r -p 'OpenAI-compatible SDK base: ' FLUXO_BASE_URL; export FLUXO_BASE_URL

read -r -p 'GPT-6 Astra API identifier: ' FLUXO_MODEL; export FLUXO_MODEL

If you want to follow along, you can create a Fluxo account. The same service works with Claude Code, Codex, Gemini CLI, and any OpenAI SDK. Billing follows real tokens, including cache reads and writes, without a monthly fee; invited users' payments also earn a referral share.

2. Hash the immutable operation into its job ID

Create the producer with this command. The hash covers the tenant, prompt version, model, and complete request. Keep those fields immutable across retries. BullMQ ignores duplicate IDs while the original job remains retained; removing that job removes this protection. ([docs.bullmq.io](https://docs.bullmq.io/guide/jobs/job-ids?utm_source=openai))

printf '%s\n' 'import { Queue } from "bullmq";' 'import { createHash } from "node:crypto";' 'const q = new Queue("summaries", { connection: { host: "127.0.0.1", port: 6379 } });' 'const data = { tenant: "demo", version: 1, request: { model: process.env.FLUXO_MODEL, messages: [{ role: "user", content: "Summarize: The deployment passed all checks." }] } };' 'const jobId = createHash("sha256").update(JSON.stringify(data)).digest("hex");' 'await q.add("summarize", data, { jobId, attempts: 4, backoff: { type: "exponential", delay: 1000, jitter: 0.5 }, removeOnComplete: false });' 'console.log(jobId);' 'await q.close();' > enqueue.mjs

Four attempts means four total executions, not four additional retries. Exponential backoff with jitter spreads retry traffic. ([docs.bullmq.io](https://docs.bullmq.io/guide/retrying-failing-jobs?utm_source=openai))

3. Persist the answer before acknowledging BullMQ

The SDK gets zero automatic retries; BullMQ owns the retry budget. Otherwise, SDK retries multiply queue attempts. The client also uses an explicit request timeout. ([github.com](https://github.com/openai/openai-node/blob/main/docs/configuration.md?utm_source=openai))

printf '%s\n' 'import { Worker, UnrecoverableError } from "bullmq";' 'import OpenAI from "openai";' 'import Database from "better-sqlite3";' 'const db = new Database("results.sqlite");' 'db.exec("CREATE TABLE IF NOT EXISTS results (id TEXT PRIMARY KEY, body TEXT NOT NULL)");' 'const get = db.prepare("SELECT body FROM results WHERE id = ?");' 'const save = db.prepare("INSERT INTO results VALUES (?, ?) ON CONFLICT(id) DO NOTHING");' 'const api = new OpenAI({ apiKey: process.env.FLUXO_API_KEY, baseURL: process.env.FLUXO_BASE_URL, maxRetries: 0, timeout: 30000 });' 'const worker = new Worker("summaries", async job => {' ' const cached = get.get(job.id);' ' if (cached) { console.log("reused", job.id); return JSON.parse(cached.body); }' ' let result;' ' try { result = await api.chat.completions.create(job.data.request); }' ' catch (error) {' ' if ([400, 401, 403, 404, 422].includes(error.status)) throw new UnrecoverableError(error.message);' ' throw error;' ' }' ' save.run(job.id, JSON.stringify(result));' ' if (process.env.DEMO_FAIL_AFTER_SAVE === "1" && job.attemptsMade === 0) throw new Error("demo failure after save");' ' return JSON.parse(get.get(job.id).body);' '}, { connection: { host: "127.0.0.1", port: 6379 }, concurrency: 1 });' 'worker.on("failed", (job, error) => console.error(job?.id, error.message));' 'worker.on("error", console.error);' > worker.mjs

The selected permanent HTTP errors fail immediately through UnrecoverableError instead of exhausting the queue budget. ([docs.bullmq.io](https://docs.bullmq.io/patterns/stop-retrying-jobs))

4. Trigger the post-save failure and inspect recovery

node enqueue.mjs && node enqueue.mjs && DEMO_FAIL_AFTER_SAVE=1 node worker.mjs

Expect matching producer IDs, a deliberate failure, then a reused log. Stop the worker with Ctrl-C and inspect the durable output.

node --input-type=module -e 'import Database from "better-sqlite3"; console.log(new Database("results.sqlite").prepare("SELECT id, length(body) AS bytes FROM results").all());'

5. Keep the remote-call boundary explicit

• A timeout before local persistence remains ambiguous, so retrying may generate and bill again.
• Do not assume an arbitrary idempotency header prevents duplicate provider execution.
• For multiple hosts, replace this local ledger with shared transactional storage.
• Commit downstream changes and outbox records together before acknowledging successful processing.

This lab guarantees one stored result per operation key. It deliberately does not promise one remote invocation. That distinction keeps a retry mechanism from becoming a billing surprise.