Node.js SSE: Stop Paying for Abandoned Streams
Node.js SSE: Stop Paying for Abandoned Streams
The expensive streaming bug is not usually a slow first token. It is a request that keeps generating after the developer closes a terminal, changes a prompt, or navigates away. Server-sent events make output feel immediate, but streaming alone does not reduce token charges. Saving money requires cancellation, bounded output, and reliable accounting.
My default architecture is simple: the browser or terminal calls a Node.js backend, and that backend opens the upstream model stream. Credentials stay server-side. The backend forwards useful events and owns the cancellation lifecycle.
Connect the Node.js relay before tuning tokens
Fluxo gives this backend one key and one base URL, https://api.fluxorelay.com, across Claude, GPT, Gemini, DeepSeek, Kimi, GLM, Qwen, Grok, and more. It works with any OpenAI SDK, plus Claude Code, Codex, and Gemini CLI. Keeping the transport consistent makes cost experiments easier to reproduce from the tools already open on your desktop.
Start with DeepSeek V4 Flash for a bounded summarization task, then compare Claude Sonnet 5.5 on the same fixtures. Use the exact API model identifiers exposed for your account rather than assuming display names are request identifiers.
If you want to run these measurements yourself, create a Fluxo account and keep the first experiment small.
Send valid SSE frames from your Node.js route
In an Express route, set Content-Type to text/event-stream and Cache-Control to no-cache. Flush headers when you are ready to commit to streaming. Forward text with res.write('data: ' + JSON.stringify({ text: delta }) + '\n\n'); so embedded newlines cannot accidentally create extra events.
For an OpenAI-compatible chat stream, iterate with for await (const chunk of stream) and read chunk.choices?.[0]?.delta?.content. Some chunks contain metadata rather than text. Do not assume every event has choices or a printable delta.
A browser client can use fetch with a POST body and read response.body incrementally. Native EventSource does not provide an arbitrary POST body or authorization headers. Decode bytes with a streaming TextDecoder, retain incomplete frames between reads, and split only at complete SSE boundaries. Network chunks are not event boundaries.
Wire disconnects into the upstream AbortController
Create one AbortController per request. With the OpenAI Node SDK, pass { signal: controller.signal } as the second argument to client.chat.completions.create, alongside a first argument containing stream: true and your request parameters.
Track completion explicitly. Register res.on('close', () => { if (!completed) controller.abort(); }); before opening the upstream request. Once iteration finishes normally, set completed = true and call res.end(). In the error path, abort upstream and finish cleanup too.
Also handle request abortion, SDK failures, and your own deadline. Catch abort errors separately from provider failures so normal tab closures do not flood your error tracker. Cancellation limits additional generation where supported; it does not erase tokens already generated or guarantee instantaneous upstream shutdown.
Prevent retries from doubling partial answers
Check your SDK's retry defaults before adding another retry loop. Before any output reaches the client, a bounded retry may be reasonable for transient failures. After partial output, silently starting again can buy two answers while displaying one confusing result.
- Mark interrupted responses explicitly instead of pretending generation completed successfully.
- Require an intentional retry after visible output has already arrived.
- Set an output ceiling using the parameter supported by your model.
- Stop forwarding when writable buffers fill, using backpressure or bounded buffering.
Disable buffering in your reverse proxy where supported, and test through that proxy rather than only on localhost. Send occasional SSE comment heartbeats for idle connections, but enforce a deadline; a healthy socket should not authorize unlimited generation.
Measure billed tokens, not animation speed
Record model, request identifier, elapsed time, disconnect status, and reported usage. Request streamed usage when the selected route supports it, but remember an interrupted stream may never deliver its final usage event. Reconcile gaps against billing records instead of recording zero.
Fluxo bills per real token, including cache reads and writes, with no monthly fee. Measure cache behavior rather than assuming repeated context is free. Referral shares from invited users' payments are separate from inference savings, so keep them outside your request-cost comparisons.
The useful benchmark is cost per accepted result. Test cancellation during generation, proxy disconnects, and retry failures before shipping. Fewer abandoned tokens beat a prettier loading animation.