1502355 - Implement ReadableStream.prototype.pipeTo and pipeThrough

Status: RESOLVED WONTFIX

(Core :: JavaScript Engine, enhancement, P3)

Comment 1

[Mike Conca [:mconca]]

Added dependency on Writable Streams. We are hearing from web platforms that piping is a critical missing feature of Readable Streams right now. It should be implemented once Writable Sreams are available.

Comment 2

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Implement the user-visible portion of RS.prototype.pipeTo, behind a RealmOption, with the underlying piping algorithm *only* just stubbed out. r=arai

Comment 3

[Jeff Walden [:Waldo]]

The one patch added implements basically the surface of piping, but not any of the internals -- if you invoke them, an error is thrown. A lot more patching is going to have to happen to actually make piping work.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d3ab478c8e73
Implement the user-visible portion of RS.prototype.pipeTo, behind a RealmOption, with the underlying piping algorithm *only* just stubbed out.  r=arai

Comment 5

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/d3ab478c8e73

Comment 6

[Eric Shepherd [:sheppy]]

Is this something expected to be actually completed and functional for 73 or is it going to move off to later? Docs planning for 73 going on. Thanks!

Comment 7

[Jeff Walden [:Waldo]]

There is zero chance of this being finished by Monday for the next merge date. I'm currently bogged down in a separate bug and largely haven't started any meaningful work on this yet.

Comment 8

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Implement a very barebones start of a |PipeToState| class to hold the state of an ongoing |pipeTo| operation. r=arai

Depends on D59232

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d1408b4cf53a
Implement a very barebones start of a |PipeToState| class to hold the state of an ongoing |pipeTo| operation.  r=arai

Comment 10

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/d1408b4cf53a

Comment 11

[Jeff Walden [:Waldo]]

Arno Renevier has posted elaborations on the hyper-partial stabbing I have done so far, FWIW. Still looking into what of that I can integrate into my own work here.

https://phabricator.services.mozilla.com/D59258

Comment 12

[Chris Mills [:cmills]]

Since this is just early implementation details, and nowhere near a shipping feature, I'll remove dev-doc-needed from here. We'll track it again in future when it is a bit nearer to completion.

Comment 13

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Make |ReadableStreamPipeTo| return |PromiseObject*|. r=arai

Comment 14

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Fill in more of |ReadableStreamPipeTo|, as regards consing up a |PipeToState| closure to store state variables. r=arai

Depends on D65057

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8a104e44cbe6
Make |ReadableStreamPipeTo| return |PromiseObject*|.  r=arai
https://hg.mozilla.org/integration/autoland/rev/7b7fb2498eb6
Fill in more of |ReadableStreamPipeTo|, as regards consing up a |PipeToState| closure to store state variables.  r=arai

Comment 16

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/8a104e44cbe6
https://hg.mozilla.org/mozilla-central/rev/7b7fb2498eb6

Comment 17

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Make the |ForAuthorCodeBool| argument to |CreateReadableStreamDefaultReader| not defaulted, for explicitness at calling sites. r=arai!

Comment 18

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Support wrapping a |JS::MutableHandle<Maybe<JS::Value>>|. r=arai!

Depends on D68472

Comment 19

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Add skeletal code checking for source/dest closure at start of a pipe-to operation. r=arai!

Depends on D68473

Comment 20

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Add reactions handling source/dest becoming closed or errored. r=arai!

Depends on D68474

Comment 21

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Begin defining the shutdown and shutdown-with-action algorithms, starting with the "only perform shutdown once" step. r=arai!

Depends on D68475

Comment 22

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Implement writable-and-not-closing checks in shutdown steps. r=arai!

Depends on D68476

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/99f6870e3611
Make the |ForAuthorCodeBool| argument to |CreateReadableStreamDefaultReader| not defaulted, for explicitness at calling sites.  r=arai
https://hg.mozilla.org/integration/autoland/rev/ac82299cbd75
Support wrapping a |JS::MutableHandle<Maybe<JS::Value>>|.  r=arai
https://hg.mozilla.org/integration/autoland/rev/9fd9dc3b2edb
Add skeletal code checking for source/dest closure at start of a pipe-to operation.  r=arai
https://hg.mozilla.org/integration/autoland/rev/fed0a5f809d2
Add reactions handling source/dest becoming closed or errored.  r=arai
https://hg.mozilla.org/integration/autoland/rev/ae4985a75c89
Begin defining the shutdown and shutdown-with-action algorithms, starting with the "only perform shutdown once" step.  r=arai
https://hg.mozilla.org/integration/autoland/rev/93d8076776df
Implement writable-and-not-closing checks in shutdown steps.  r=arai

Comment 24

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/99f6870e3611
https://hg.mozilla.org/mozilla-central/rev/ac82299cbd75
https://hg.mozilla.org/mozilla-central/rev/9fd9dc3b2edb
https://hg.mozilla.org/mozilla-central/rev/fed0a5f809d2
https://hg.mozilla.org/mozilla-central/rev/ae4985a75c89
https://hg.mozilla.org/mozilla-central/rev/93d8076776df

Comment 25

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Don't use |PromiseObject::unforgeableResolve| where the passed-in value is known to not be a promise, and add/use |PromiseResolvedWithUndefined| for cases where the value is |undefined|. r=arai!

Comment 26

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Rename |CanPipeStreams| to |SourceOrDestErroredOrClosed| for more clarity. r=arai!

Depends on D69070

Comment 27

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Make |ReadableStreamDefaultControllerPullSteps| return a narrower |PromiseObject*| type. r=arai!

Depends on D69071

Comment 28

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Make |PromiseRejectedWithPendingError| return a narrower |PromiseObject*| type. r=arai!

Depends on D69072

Comment 29

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Make |ReadableByteStreamControllerPullSteps| return a narrower |PromiseObject*| type. r=arai!

Depends on D69073

Comment 30

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Make |ReadableStreamControllerPullSteps| return a narrower |PromiseObject*| type. r=arai!

Depends on D69074

Comment 31

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Make |ReadableStreamDefaultReaderRead| return a narrower |PromiseObject*| type. r=arai!

Depends on D69075

Comment 32

[Jeff Walden [:Waldo]]

FYI, the type-narrowing patches just posted are mostly intended to permit making the types of values in some of the piping algorithms 1) clearer that an actual promise is returned, and 2) clearer that a promise from the current realm/compartment is being returned. (I have the start of a first patch that actually puts this to some use, but it's just slightly too sketchy to be reviewable yet.)

Comment 33

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6590b48d7450
Don't use |PromiseObject::unforgeableResolve| where the passed-in value is known to not be a promise, and add/use |PromiseResolvedWithUndefined| for cases where the value is |undefined|.  r=arai
https://hg.mozilla.org/integration/autoland/rev/b993ebbff7e4
Rename |CanPipeStreams| to |SourceOrDestErroredOrClosed| for more clarity.  r=arai
https://hg.mozilla.org/integration/autoland/rev/03bf4a6f38a7
Make |ReadableStreamDefaultControllerPullSteps| return a narrower |PromiseObject*| type.  r=arai
https://hg.mozilla.org/integration/autoland/rev/10f67be0415c
Make |PromiseRejectedWithPendingError| return a narrower |PromiseObject*| type.  r=arai
https://hg.mozilla.org/integration/autoland/rev/14f74f970269
Make |ReadableByteStreamControllerPullSteps| return a narrower |PromiseObject*| type.  r=arai
https://hg.mozilla.org/integration/autoland/rev/fcf4151b9e07
Make |ReadableStreamControllerPullSteps| return a narrower |PromiseObject*| type.  r=arai
https://hg.mozilla.org/integration/autoland/rev/d1287e914d42
Make |ReadableStreamDefaultReaderRead| return a narrower |PromiseObject*| type.  r=arai

Comment 34

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/6590b48d7450
https://hg.mozilla.org/mozilla-central/rev/b993ebbff7e4
https://hg.mozilla.org/mozilla-central/rev/03bf4a6f38a7
https://hg.mozilla.org/mozilla-central/rev/10f67be0415c
https://hg.mozilla.org/mozilla-central/rev/14f74f970269
https://hg.mozilla.org/mozilla-central/rev/fcf4151b9e07
https://hg.mozilla.org/mozilla-central/rev/d1287e914d42

Comment 35

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Make |WritableStreamDefaultWriterWrite| return a narrower |PromiseObject*| type for more clarity. r=arai!

Comment 36

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2084b0479a61
Make |WritableStreamDefaultWriterWrite| return a narrower |PromiseObject*| type for more clarity.  r=arai

Comment 37

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/2084b0479a61

Comment 38

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Read from the source stream once permitted to do so. r=arai!

Comment 39

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - React to a stream reader read result. r=arai!

Depends on D71855

Comment 40

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a5997699b481
Read from the source stream once permitted to do so.  r=arai
https://hg.mozilla.org/integration/autoland/rev/b71b149b451c
React to a stream reader read result.  r=arai

Comment 41

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/a5997699b481
https://hg.mozilla.org/mozilla-central/rev/b71b149b451c

Comment 42

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Allow handler functions to store an extra *value*, not merely an extra T* that's an object. r=arai!

Comment 43

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Implement |WritableStreamDefaultWriterCloseWithErrorPropagation|. r=arai!

Depends on D80778

Comment 44

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - React to a read promise that rejects. r=arai!

Depends on D80779

Comment 45

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Store the value optionally passed to finalizing as an extra value in a handler function. r=arai!

Depends on D80780

Comment 46

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Save to the side the action specified when we're supposed to "shutdown with an action". r=arai!

Depends on D80781

Comment 47

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Wait for completed reads to finish writing before finalizing, during pipe-to shutdown with an action. r=arai!

Depends on D80782

Comment 48

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Implement some of the process of performing an action and then finalizing, in the shutdown with an action process. r=arai!

Depends on D80783

Comment 49

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Fill in most of the body of the "finalize" operation in the piping algorithm. r=arai!

Depends on D80784

Comment 50

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/42fb5205d051
Allow handler functions to store an extra *value*, not merely an extra T* that's an object.  r=arai
https://hg.mozilla.org/integration/autoland/rev/146e9fa5e758
Implement |WritableStreamDefaultWriterCloseWithErrorPropagation|.  r=arai
https://hg.mozilla.org/integration/autoland/rev/d4b8ef062b42
React to a read promise that rejects.  r=arai
https://hg.mozilla.org/integration/autoland/rev/a7d2cf07580f
Store the value optionally passed to finalizing as an extra value in a handler function.  r=arai
https://hg.mozilla.org/integration/autoland/rev/c2066ce91465
Save to the side the action specified when we're supposed to "shutdown with an action".  r=arai
https://hg.mozilla.org/integration/autoland/rev/cb9ba1647a9a
Wait for completed reads to finish writing before finalizing, during pipe-to shutdown with an action.  r=arai
https://hg.mozilla.org/integration/autoland/rev/22e38ee59de3
Implement some of the process of performing an action and then finalizing, in the shutdown with an action process.  r=arai
https://hg.mozilla.org/integration/autoland/rev/148d65d325da
Fill in most of the body of the "finalize" operation in the piping algorithm.  r=arai

Comment 51

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/42fb5205d051
https://hg.mozilla.org/mozilla-central/rev/146e9fa5e758
https://hg.mozilla.org/mozilla-central/rev/d4b8ef062b42
https://hg.mozilla.org/mozilla-central/rev/a7d2cf07580f
https://hg.mozilla.org/mozilla-central/rev/c2066ce91465
https://hg.mozilla.org/mozilla-central/rev/cb9ba1647a9a
https://hg.mozilla.org/mozilla-central/rev/22e38ee59de3
https://hg.mozilla.org/mozilla-central/rev/148d65d325da

Comment 52

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Declare |JSClass| in consistent fashion everywhere. r=arai!

Comment 53

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Remove various unused detritus from jsfriendapi.h. r=arai!

Depends on D87376

Comment 54

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Add support for unwrapping a value/object, known at one time to have a given class, as an object of that class. r=arai!

Depends on D87377

Comment 55

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Correctly detect AbortSignal instances using a |const JSClass*| supplied by the embedding. r=arai!

Depends on D87378

Comment 56

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Recognize already-aborted signals passed to |ReadableStreamPipeTo| and don't pipe in this case. r=arai!

Depends on D87379

Comment 57

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/207f8942ffd2
Declare |JSClass| in consistent fashion everywhere.  r=arai
https://hg.mozilla.org/integration/autoland/rev/eeff6597f16f
Remove various unused detritus from jsfriendapi.h.  r=arai
https://hg.mozilla.org/integration/autoland/rev/deb00e023f3b
Add support for unwrapping a value/object, known at one time to have a given class, as an object of that class.  r=arai
https://hg.mozilla.org/integration/autoland/rev/254c794085e1
Correctly detect AbortSignal instances using a |const JSClass*| supplied by the embedding.  r=arai
https://hg.mozilla.org/integration/autoland/rev/77027a06c438
Recognize already-aborted signals passed to |ReadableStreamPipeTo| and don't pipe in this case.  r=arai

Comment 58

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Fix Rust build to not depend on previously-bootlegged symbols.

Comment 59

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/78982a089258
Fix Rust build to not depend on previously-bootlegged symbols.

Comment 60

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/207f8942ffd2
https://hg.mozilla.org/mozilla-central/rev/eeff6597f16f
https://hg.mozilla.org/mozilla-central/rev/deb00e023f3b
https://hg.mozilla.org/mozilla-central/rev/254c794085e1
https://hg.mozilla.org/mozilla-central/rev/77027a06c438
https://hg.mozilla.org/mozilla-central/rev/78982a089258

Comment 61

[Florian Ernst]

(In reply to Mike Conca [:mconca] from comment #1)

Added dependency on Writable Streams. We are hearing from web platforms that piping is a critical missing feature of Readable Streams right now. It should be implemented once Writable Sreams are available.

I agree on this idea, though both of them should be implemented in parallel if possible.

Comment 62

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Allocate |PipeToState| as tenured, for perhaps marginally greater efficiency. r=arai!

Depends on D92327

Comment 63

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Rename |JS::InitAbortSignalHandling| to |JS::InitPipeToHandling|, because this embedder interface is going to get a lot more pipeTo-specific shortly... r=arai!

Depends on D92350

Comment 64

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Expand |JS::InitPipeToHandling| to support creating, adding, removing, and tracing |abortAlgorithm|. r=arai!,smaug!

Depends on D92351

Comment 65

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Expand |JS::InitPipeToHandling| to let the embedder create the |AbortError| thrown if piping's |abortAlgorithm| is run. r=arai!,smaug!

Depends on D92352

Comment 66

[Jeff Walden [:Waldo]]

Attachment: Bug 1502355 - Implement the action passed to "shutdown with an action" in |abortAlgorithm|. r=arai!

Depends on D92353

Comment 67

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/195dfba58e22
Allocate |PipeToState| as tenured, for perhaps marginally greater efficiency.  r=arai

Comment 68

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/195dfba58e22

Comment 69

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2a2548dc3ded
Rename |JS::InitAbortSignalHandling| to |JS::InitPipeToHandling|, because this embedder interface is going to get a lot more pipeTo-specific shortly...  r=arai

Comment 70

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/2a2548dc3ded

Comment 71

[Tom S [:evilpie]]

pipeTo is going to be implemented by bug 1734241 and pipeThrough by bug 1734243.