434ec07885
Introduce streaming endpoint for NDJSON scan progress, incorporate scan checkpoints into scan details, and update UI components to display this new information. Replit-Commit-Author: Agent Replit-Commit-Session-Id: 0d01f99a-ea6a-447d-82fd-311715434a39 Replit-Commit-Checkpoint-Type: full_checkpoint Replit-Commit-Event-Id: 2852b526-3bf8-4a93-a62a-a50e26291074 Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/e32d2b99-1721-47dd-833c-98b372f48008/0d01f99a-ea6a-447d-82fd-311715434a39/8MCgDZm Replit-Helium-Checkpoint-Created: true
36 lines
1.9 KiB
Markdown
36 lines
1.9 KiB
Markdown
---
|
|
name: NDJSON streaming under Express behind the Replit proxy
|
|
description: Pitfalls for live/streaming HTTP responses (NDJSON/SSE) from an Express API on Replit — client-disconnect detection, persistence, and proxy buffering
|
|
---
|
|
|
|
# Streaming responses (NDJSON) from Express on Replit
|
|
|
|
For a long-lived POST that streams progress (e.g. `application/x-ndjson` with
|
|
`res.flushHeaders()` then line-delimited JSON writes):
|
|
|
|
**Detect a real client abort with `res.on("close")` + `res.writableFinished` — never
|
|
`req.on("close")`.**
|
|
`req.on("close")` fires as soon as the POST *request body* has been fully consumed,
|
|
which is normal and happens immediately. Gating writes on that wrongly suppresses the
|
|
entire stream. Use the *response* close event, and treat it as an abort only when
|
|
`!res.writableFinished`.
|
|
|
|
**Persist the result regardless of disconnect.** Run the analysis and the DB write
|
|
inside the handler's `try` unconditionally; only guard the `res.write()` calls on
|
|
"connection still open". This way a client that disconnects mid-stream still gets its
|
|
result persisted (and the client must NOT re-submit on a mid-stream error, or it
|
|
creates a duplicate row — see below).
|
|
|
|
**The Replit path-proxy does NOT buffer the stream.** Incremental flushes arrive line
|
|
by line through `$REPLIT_DEV_DOMAIN`, same as `localhost`. No special proxy header or
|
|
chunk-padding needed.
|
|
|
|
**Client fetch must be root-relative (`/api/...`), not `import.meta.env.BASE_URL`.**
|
|
That matches the generated API client and is what routes correctly through the proxy.
|
|
|
|
**Fallback gating to avoid duplicate persists:** a client that streams and also falls
|
|
back to a non-streaming POST on error must only fall back when the server never
|
|
processed the request (fetch rejected, or `!res.ok` with a 5xx). Once the response
|
|
body has started streaming, any read error means the server is already persisting —
|
|
falling back then duplicates the record. Distinguish 4xx (show message, no retry)
|
|
from transport failures.
|