1387483 - [meta] Support ReadableStream as Request.body in fetch API

Status: NEW

(Core :: DOM: Networking, enhancement, P2)

Description

[Ben Kelly [:bkelly, not reviewing]]

Bug 1128959 is implementing support for ReadableStream as Response.body, but we are not implementing request bodies yet.  Request bodies are harder for a number of reasons:

1. HTTP/1.x effectively require fixed length upload bodies.
2. HTTP/2 can use chunked-encoding upload bodies, but we haven't implemented it yet.

Chrome has not yet shipped ReadableStream upload bodies either.

Comment 1

[dtippett]

Is this the reason why the ReadableStream example in https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream doesn't work in Firefox even after changing the URL to a resource with properly defined CORS headers? It appears to work in Chrome and Edge.

Comment 2

[Ben Kelly [:bkelly, not reviewing]]

(In reply to dtippett from comment #1)
> Is this the reason why the ReadableStream example in
> https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream doesn't work
> in Firefox even after changing the URL to a resource with properly defined
> CORS headers? It appears to work in Chrome and Edge.

I think that uses response.body.  This bug is about request body.  The code is probably not working in firefox because we have not exposed ReadableStream by default yet.  See bug 1389628.

Comment 3

[dtippett]

Okay, thanks for the explanation.

Comment 5

[Irakli Gozalishvili [:irakli] [:gozala] [@gozala]]

At the moment passing ReadableStream as body to a fetch or a Request constructor produces request that has body that is just body.toStrin():

const encoder = new TextEncoder()
const request = new Request('about:blank', {
  method: "PUT",
  body: new ReadableStream({
    pull(c) {
      c.enqueue(encoder.encode("hello"))
      c.close()
    }
  }) 
})
await request.text() // -> "[object ReadableStream]"

I think we should at least error until support for body has landed, especially given that Response accepts ReadableStream just fine.

const encoder = new TextEncoder()
const response = new Response(new ReadableStream({
  pull(c) {
    c.enqueue(encoder.encode("hello"))
    c.close()
  }
}))
await response.text() // -> "hello"

Comment 6

[Firefox Bug Husbandry Bot]

Migrating Webcompat whiteboard priorities to project flags. See bug 1547409.

Comment 7

[Firefox Bug Husbandry Bot]

See bug 1547409. Migrating whiteboard priority tags to program flags.

Comment 8

[github]

Are there any plans to implement this in FF? whatwg seems to be considering removing it from the standard as no browser is implementing it.

I think this would be a huge loss for the web, as even things like firefox send need to resort to hacks like uploading large files via websockets.

Comment 9

[Ryo Ota]

Hi,

Chrome 85 includes this feature.

Here are links.

• Chrome Platform Status: fetch() upload streaming
• web.dev: Streaming requests with the fetch API
• Chromium Blog:
  Chrome 85: Upload Streaming, Human Interface Devices, Custom Properties with Inheritance and More

Comment 10

[Kagami Rosylight 🏳️‍🌈🏳️‍⚧️ [:saschanaz] (they/them)]

It seems Chrome people now think this may not be so useful.

Comment 11

[max]

I read through the (now closed) issue on github and it seems that they'll keep the feature, as many people found it very useful (me included). I did give it a try in Chrome 105 and found it to be working pretty well - with the limitations that the web server needs to run on HTTP2, HTTPS must be used, and only the "half-duplex" mode is supported. But even so, it's very useful and I would really love to see this in Firefox as well.

As many people can't imagine why this would be worth the effort, let me give you an example: I am currently working on a end-to-end encryption of files, completely done in the browser. You can imagine why loading the entire file into memory, encrypting it there and then sending it to a server in one piece isn't feasible - for anything but small image files, that is. I managed to come up with a relatively simple proof-of-concept that encrypts any file on the client side and sends this with a single fetch (POST) to a Kestrel server - and later downloads and decrypts the same file again. It works perfectly in Chrome (both on desktop and smartphone) but not on Firefox, which is a shame. As the whole idea behind this project is privacy, I wouldn't want to depend on Chrome for using it.

tl;dr: Please bring this to Firefox - even with limitations (like half-duplex HTTPS 2 only), it's pretty useful.

Comment 12

[Jimmy Warting]

Oh, c'mon

new Request('http://localhost', { method: 'post', body: '123' }).body // undefined
await new Request('http://localhost', { method: 'post', body: '123' }).text() // 123

what a # developer-pain this is.

Comment 14

[Sean Morris]

I've been waiting on this one for a while. I'm implementing an httpd-like interface that runs WASM binaries in a service worker and I'd like to be able to respond correctly to POST requests.

I've got an example running at https://seanmorris.github.io/php-wasm. Although github pages is a static filehost, I'm running PHP sites on it by intercepting the requests in a service worker and interpreting the requests with php compiled to web assembly.

The login function in the drupal example works great on chrome, but you can't login on firefox since the POST request doesn't contain a body.

My vision for the future is that users should be able to use a service worker as a full backend. I think it would be useful for developers and perhaps for reasons currently unforeseen.

Its really disappointing that my demo doesn't operate fully in firefox. I feel like this feature would be a huge win for the web as a whole.

Comment 15

[Kagami Rosylight 🏳️‍🌈🏳️‍⚧️ [:saschanaz] (they/them)]

Fetch API thing probably fits better in DOM: Networking.

Comment 16

[Randell Jesup [:jesup] (needinfo me)]

Note: potential webcompat issue

Comment 18

[Luc Fauvel]

Hi, I'm currently working on something that would benefit from this and would like to eventually submit a patch. Any reason I shouldn't start work on this?

Comment 19

[Olli Pettay [:smaug][[e-mail]]]

Patches are very welcome.

Comment 20

[Kagami Rosylight 🏳️‍🌈🏳️‍⚧️ [:saschanaz] (they/them)]

I think Necko team wants to write a patch very soon, so it might be better to talk with :smayya first.

Comment 21

[Mauro B]

Hi! Any news on this?

Comment 22

[Mauro B]

Hi! I've been talking to Luc Fauvel about a solution. I wonder if I could get assigned to this issue. If he's alright with it. Thanks!

Comment 23

[Sunil Mayya]

Hello Mauro B,
Thank you so much for your interest in contributing to Firefox! 🎉
Apologies for the delay in responding.

We’d love to see patches for this, as it's not something we have planned to work on anytime soon.
It would be great to hear how you plan to approach the implementation, and we’re happy to support you in any way we can.

Let us know if you'd like to discuss this over a Zoom call—we’d be more than happy to chat!
you can reach out to us via matrix at #necko:mozilla.org as well.

Looking forward to hear back from you!

Comment 24

[Niko Cantero]

(In reply to Mauro B from comment #22)

Hi! I've been talking to Luc Fauvel about a solution. I wonder if I could get assigned to this issue. If he's alright with it. Thanks!

Bump if you didn't see the above comment :3

Comment 25

[Mauro V [:cheff]]

Whoops! Sorry about that, i've missed it... I did try to do a fix for this but ended up not 100% succeeding, im very sorry.

Comment 26

[Sunil Mayya]

No worries Mauro, if you are still interested in contributing this, we would love to support you.

Comment 27

[Armen Michaeli]

I don't think HTTP 1.x forbids use of e.g. Transfer-Encoding (enabling the so-called chunked transfer encoding) for specifically requests, reading the relevant section of the HTTP 1.1 specification? There is the following sentence which alludes to the fact that it's supported insofar the server is known to support it for handling requests:

A client MUST NOT send a request containing Transfer-Encoding unless it knows the server will handle HTTP/1.1 requests (or later minor revisions); such knowledge might be in the form of specific user configuration or by remembering the version of a prior received response.

Am I missing something?

Comment 28

[Luc Fauvel]

(In reply to Armen Michaeli from comment #27)

I don't think HTTP 1.x forbids use of e.g. Transfer-Encoding (enabling the so-called chunked transfer encoding) for specifically requests, reading the relevant section of the HTTP 1.1 specification? There is the following sentence which alludes to the fact that it's supported insofar the server is known to support it for handling requests:

A client MUST NOT send a request containing Transfer-Encoding unless it knows the server will handle HTTP/1.1 requests (or later minor revisions); such knowledge might be in the form of specific user configuration or by remembering the version of a prior received response.

Am I missing something?

From the google team that implemented ReadableStreams for fetch in chrome:

Chunked encoding is pretty common when it comes to HTTP/1.1 responses, but very rare when it comes to requests, so it's too much of a compatibility risk.

Comment 29

[Brad Isbell]

I assert that the Google team was incorrect. No evidence was provided about a compatibility risk, and whether it's "too much of" anything is subjective. With a single offhand comment, they ended the chance of web client compatibility with entire classes of servers that will never support HTTP/2.

In my work, this includes most any HTTP-compatible media servers such as Icecast and FFmpeg, as well as generic storage like S3-compatibles. I can imagine a need with other devices such as NAS web interfaces, IoT devices, etc.

This is essentially new functionality. Any hypothetical compatibility risk is strongly outweighed by the 0% compatibility we have today.

I'd urge Mozilla to take the lead here and enable Fetch API streaming request bodies over HTTP/1.1.

Comment 30

[Armen Michaeli]

From the google team that implemented ReadableStreams for fetch in chrome:

Chunked encoding is pretty common when it comes to HTTP/1.1 responses, but very rare when it comes to requests, so it's too much of a compatibility risk.

But Google Chrome, unlike Firefox (which is what this is about, after all), has implemented ReadableStream for body on Request, has it not? Does this mean the quote is outdated?

Also, looking aside from Google exercising authority over HTTP, my earlier comment about chunked transfer coding was prompted by the following bullet point in the bug description:

1. HTTP/1.x effectively require fixed length upload bodies.

I still am not able to seemingly understand what "effectively"means above? I have used Transfer-Encoding with HTTP 1.1 on a number of occasions, and it's no more complicated (or even different, really) -- from the perspective of a HTTP client -- than receiving chunked responses. I mean I'd understand if the description said "support among other browsers [at the time] is sparse [so we're post-poning this]", but blaming this on HTTP 1.1 does this "bug" a disservice -- as most perhaps intimately familiar with HTTP would be inclined to take the description "at face value".

I mean let's call the shovel a shovel -- support in Firefox is not there, HTTP 1.1 trivially supports it, Google Chrome has not implemented it, and all there is to it is someone (like me, in all fairness) grabbing the code and getting to work on it. I understand the latter part, so please forgive me if I am being pedantic about the description -- it helps to be on the same page as far as reasons and blockers go, I'd say.

Comment 31

[Brad Isbell]

But Google Chrome, unlike Firefox (which is what this is about, after all), has implemented ReadableStream for body on Request, has it not? Does this mean the quote is outdated?

They have, but only for HTTP/2... not for HTTP/1.1.

Comment 32

[Luc Fauvel]

HTTP 1.1 trivially supports it, Google Chrome has not implemented it.

Like Brad mentioned above, Google Chrome has implemented ReadableStream on request bodies only for HTTP2 in half duplex mode. For parity, the priority should be HTTP2 in my opinion. Especially since sites that have already implemented ReadableStream request bodies have to use the duplex: 'half' parameter and thus should be supported.

There's nothing preventing Mozilla or other from implementing HTTP1.1 as well in a happy eyeballs fashion, but I don't think Google's claim that Transfer-Encoding: chunked is not widespread is false. Example, nginx, which powers around 30% of the web right now, does not support HTTP1.1 Transfer-Encoding: chunked for request bodies out of the box as far as I know, you need to install an extension (though from the docs, its unclear if that has changed in recent times).

Comment 33

[Rasmus Schultz]

So this is why upload progress is not possible with fetch in Firefox.

https://github.com/teil-one/fetch-api-progress/issues/13

Can someone please reclassify this as a bug, and increase the severity?

Browser support for fetch is literally going on a decade, this issue has been open for 8 years - and this is likely the sole reason why literally all upload components/libraries exclusively use XmlHttpRequest, a relic from the age of Internet Explorer, still to this day.

Please get this on the docket and get it done.