HTTP Streaming Example in NodeJS

Note Statistics

Note Statistics

  • Viewed 14 times
Sun, 09/12/2021 - 20:49

With HTTP streaming the connection to clients stays open, allowing a server to push data to connected clients. Popular use cases are when there is large data, and because of its size is sent in smaller part or a server needs to push updates to connected clients. For example sending the content of a very large log file or sending stock updates to connected Apps.

HTTP Chunked data

To send data as chunk the server must send Transfer Encoding: chunked header in the first response.

app.get("some-big-file", (req, res) => {
    // Indicate we're sending a multipart response
    res.writeHead(200, {
        Connection: "keep-alive",
        "Content-Type": 'multipart/mixed; boundary="-"',
        "Transfer-Encoding": "chunked",
    });

    // If the request is closed by the client we can do some cleanup
    req.on("close", () => {
        //... 
    });

    res.write("---");

    // Subscribe and send back each result as a separate chunk. We await the readData
    // call. Once we're done executing the request and there are no more results to send
    // to the client, the Promise returned by subscribe will resolve and we can end the response.
    await readData((result) => {
        const chunk = Buffer.from(JSON.stringify(result), "utf8");
        const data = [
            "",
            "Content-Type: application/json; charset=utf-8",
            "Content-Length: " + String(chunk.length),
            "",
            chunk,
        ];

        if (result.hasMoreData) {
            data.push("---");
        }

        res.write(data.join("\r\n"));
    });

    res.write("\r\n-----\r\n");
    res.end();
})

Note: One downside of this solution is that it is not supported in HTTP 2. It is an HTTP 1 only feature.

Server Sent Events

Server Sent Events(SSE) are another streaming method. SSE also has a standardized HTML5 API called EventSource. With SSE, the data is encoded as text/event-stream in the header.

app.get("subscribe-to-events", (req, res) => {
    // Indicate we're sending an event stream to the client
    res.writeHead(200, {
        "Content-Type": "text/event-stream",
        Connection: "keep-alive",
        "Cache-Control": "no-cache",
    });

    // If the request is closed by the client, we do some cleanup
    req.on("close", () => {
        // ... some cleanup code
    });

    // We subscribe to the event stream and push any new events to the client
    onSomeEvent((result) => {
        res.write(`data: ${JSON.stringify(result)}\n\n`);
    });
})

Authored by
Tags