37 lines
1.9 KiB
Markdown
37 lines
1.9 KiB
Markdown
|
---
|
||
|
|
name: NDJSON streaming under Express behind the Replit proxy
|
||
|
|
description: Pitfalls for live/streaming HTTP responses (NDJSON/SSE) from an Express API on Replit — client-disconnect detection, persistence, and proxy buffering
|
||
|
|
---
|
||
|
|
|
||
|
|
# Streaming responses (NDJSON) from Express on Replit
|
||
|
|
|
||
|
|
For a long-lived POST that streams progress (e.g. `application/x-ndjson` with
|
||
|
|
`res.flushHeaders()` then line-delimited JSON writes):
|
||
|
|
|
||
|
|
**Detect a real client abort with `res.on("close")` + `res.writableFinished` — never
|
||
|
|
`req.on("close")`.**
|
||
|
|
`req.on("close")` fires as soon as the POST *request body* has been fully consumed,
|
||
|
|
which is normal and happens immediately. Gating writes on that wrongly suppresses the
|
||
|
|
entire stream. Use the *response* close event, and treat it as an abort only when
|
||
|
|
`!res.writableFinished`.
|
||
|
|
|
||
|
|
**Persist the result regardless of disconnect.** Run the analysis and the DB write
|
||
|
|
inside the handler's `try` unconditionally; only guard the `res.write()` calls on
|
||
|
|
"connection still open". This way a client that disconnects mid-stream still gets its
|
||
|
|
result persisted (and the client must NOT re-submit on a mid-stream error, or it
|
||
|
|
creates a duplicate row — see below).
|
||
|
|
|
||
|
|
**The Replit path-proxy does NOT buffer the stream.** Incremental flushes arrive line
|
||
|
|
by line through `$REPLIT_DEV_DOMAIN`, same as `localhost`. No special proxy header or
|
||
|
|
chunk-padding needed.
|
||
|
|
|
||
|
|
**Client fetch must be root-relative (`/api/...`), not `import.meta.env.BASE_URL`.**
|
||
|
|
That matches the generated API client and is what routes correctly through the proxy.
|
||
|
|
|
||
|
|
**Fallback gating to avoid duplicate persists:** a client that streams and also falls
|
||
|
|
back to a non-streaming POST on error must only fall back when the server never
|
||
|
|
processed the request (fetch rejected, or `!res.ok` with a 5xx). Once the response
|
||
|
|
body has started streaming, any read error means the server is already persisting —
|
||
|
|
falling back then duplicates the record. Distinguish 4xx (show message, no retry)
|
||
|
|
from transport failures.
|