1367375 - Streams API needs upload chunked transfer encoding in HTTP channel

Status: RESOLVED WONTFIX

(Core :: Networking: HTTP, enhancement, P3)

Description

[Andrea Marchesini [:baku]]

Streams API allows the creation of ReadableStream objects and these objects can be used as body for Fetch/XHR/SendBeacon/WebSocket requests. When this happens, firefox must send data using chunked transfer encoding.

Currently, the interaction between necko and DOM, when data is sent to the network, is done using nsIInputStream. nsIInputStream::Available() is called to know the full size of the stream and, this information is used in the HTTP header Content-Length.

ReadableStream doesn't work in this way and it's possible that the full size of the stream is unknown when used. What I would like to have is the possibility to enable the chunked transfer encoding in nsIHttpChannel (or elsewhere). When enabled, consecutive nsIInputStream::Read() and nsIAsyncInputStream::AsyncWait() calls should be done in order to retrieve data from the stream.

Unfortunately, all of this is not specified in the Fetch API spec yet: https://fetch.spec.whatwg.org/#streams

Note that, this bug doesn't block the first implementation of Streams API: we will probably ship Streams API without this feature initially.

Comment 1

[Anne (:annevk)]

Step 4 of https://fetch.spec.whatwg.org/#concept-http-network-fetch is roughly what's supposed to define this. (See also the condition in step 5 for HTTP/1.0.)

The HTTP standard defines the details of chunked encoding. But it's not always super clear how to wrap the HTTP standard. It doesn't have a clear API.

Comment 2

[Honza Bambas (:mayhemer)]

When this is about to be implemented, one has to think about our transaction restart logic.  The input stream, regardless if it's async or not, has to be seekable back to start position (0).  We may only need some kind of buffering of the head of the request body (and limit the first chunk of data before we know the connection has not been closed before the request reached the server), nothing more.  Just keep this in mind.

Comment 3

[Jason Duell]

This isn't part of v1 of the Streams API.  The part that needs chunked encoding seems to be future work for now (not implemented in Chrome yet for instance).  So taking off the 'active' label for now.

Comment 4

[Firefox Bug Husbandry Bot]

Bulk change to priority: https://bugzilla.mozilla.org/show_bug.cgi?id=1399258

Comment 5

[Firefox Bug Husbandry Bot]

Bulk change to priority: https://bugzilla.mozilla.org/show_bug.cgi?id=1399258

Comment 6

[Selena Deckelmann :selenamarie :selena]

Is this work part of the Streams API at this point?

Comment 7

[Dragana Damjanovic [:dragana]]

When Streams are used for a request the length of data is not know at the time the headers are written. Therefore we cannot add Content-Length. We need to be careful with this because it will most probably break on some servers and middleboxes. I would say we do not implement this for now.

Comment 8

[Selena Deckelmann :selenamarie :selena]

Probably need to have a separate conversation about streams. Closing this WONTFIX for now.