Add backpressure support for an async TransformStream
Categories
(Core :: DOM: Streams, defect)
Tracking
()
People
(Reporter: minfrin, Unassigned)
Details
User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.6.1 Safari/605.1.15
Steps to reproduce:
When an attempt is made to create a TransformStream that needs to process data asynchronously, there appears to be no support for backpressure.
Actual results:
With the following "transform" function, a chunk is received and then sent as a message to a webextension background script. What is supposed to happen next is for a message to be received in response, and the response enqueued.
transformer.transform = exportFunction(
function(chunk, controller) {
queueingStrategy.highWaterMark = -1;
port.postMessage({
protocol: location.protocol,
hostname: location.hostname,
contentType: 'text/plain',
uuid: uuid,
request: chunk
});
}, window.wrappedJSObject
);
The problem with above code is that there is no support for backpressure, and so data comes tumbling in, followed by the close. The close has the effect of closing the downstream writer (this is also broken behaviour, but seems mandated by the spec), and so when the message finally arrives from the webextension the writablestream is now closed and the the chain fails.
What is needed is the ability to "pause" the readablestream to stop the data tumbling in in a pile, to be "unpaused" when the webextension responds.
The streams spec makes reference to backpressure support existing, and then makes a sideways reference to "signals" being sent, but the spec then doesn't describe how any of this works, or how any of it could be controlled to achieve actual backpressure support.
The closest thing that seems to exist is a controller.desiredSize property. This property is read only, and there is nothing concrete in the spec to define how to control this value during stream processing, or whether this value can be controlled at all.
If this value was somehow set to zero, there is nothing clearly specified to show how the system might be kicked to start the data flowing again.
Expected results:
Proper control over backpressure during the running of the stream, so that the code inside the transform function can pause and resume data flow.
This also needs to be clearly and unambiguously documented - nodejs has a competing and incompatible streams standard with the same name, meaning that it is impossible to search for help, and you have to rip apart specs to get clarification.
|
||
Comment 1•3 years ago
|
||
The Bugbug bot thinks this bug should belong to the 'Core::Web Painting' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.
|
||
Updated•3 years ago
|
|
||
Updated•3 years ago
|
|
||
Comment 2•3 years ago
|
||
Hi Graham, thanks for the report!
Do you have a minimal repro for this, or do you know whether Chrome has the same behavior? It seems this is worth discussing on https://github.com/whatwg/streams/ if we have a problem in the spec.
S3 on the logic that transform streams aren't yet central to the Web, but in terms of functionality and workaround availability close to S2.
|
||
Comment 4•3 years ago
|
||
(Potentially related is this Streams bug: https://github.com/whatwg/streams/issues/1235)
|
|
||
Comment 5•3 years ago
|
||
A needinfo is requested from the reporter, however, the reporter is inactive on Bugzilla. Closing the bug as incomplete.
For more information, please visit BugBot documentation.
|
|
||
Comment 6•3 years ago
|
||
Just to be clear, the pull function is correctly called only once every 4 seconds in the following example, so I don't see a problem in TransformStream.
new ReadableStream({
pull(c) {
console.log('foo')
c.enqueue('foo');
}
}, { highWaterMark: 0 }).pipeThrough(
new TransformStream({
async transform(chunk, controller) {
console.log('transform', chunk)
await new Promise(r => setTimeout(r, 4000)); //
controller.enqueue(chunk);
}
})
).pipeTo(new WritableStream())
Description
•