CanvasRenderingContext2D with live captureStream captures a frame on first EnsureTarget
Categories
(Core :: Graphics: Canvas2D, defect, P2)
Tracking
()
People
(Reporter: jib, Assigned: pehrsons)
Details
Attachments
(3 files)
STRs:
- Open https://jsfiddle.net/jib1/kzsh9wgx/17/
- DON'T hit the "Start" button.
- Hit JSFiddle's
▶ Run
button 10 times a couple of seconds apart.
Expected result every time:
checking
connected
HERE
Actual result ~1/5 of the time:
checking
connected
Reporter | ||
Comment 1•4 years ago
|
||
See https://github.com/w3c/webrtc-pc/issues/2506#issuecomment-616610282 for context.
Reporter | ||
Comment 2•4 years ago
•
|
||
Looks like this might be racing with getImageData
https://jsfiddle.net/jib1/kzsh9wgx/32/ because without that call (34) there's no frame.
Comment 3•4 years ago
|
||
getImageData()
is not the issue. muted
behaviour is inconsistent.
BTW, Nightly is crashing the tab when attempting to load jsfiddle. Used plnkr to reproduce.
Comment 4•4 years ago
|
||
Per https://bugs.chromium.org/p/chromium/issues/detail?id=941740#c6
The mute events on a remote track are only supposed to fire from negotiation changes (SRD/SLD) and real network issues (timeout or BYE), not content.
The MediaStreamTrack
from canvas.captureStream()
should not be muted
at all once network connection is established.
Is this a case of WebRTC not being observable with regard to the exact timing of network connections, where the only observable behaviour from that connection is the inconsistency as to when the connection is established?
Could be 1 second this run, could be 45 seconds next run when unmute
is dispatched - and not just from a canvas.captureStream()
?
That is precisely what occurs at this code https://plnkr.co/edit/Axkb8s?preview (change codec at MediaRecorder
for Nightly to run). Could take 2 seconds after first run (if cache is enabled), could take 30 seconds third run, just have to await unmute
and resize
events - because WebRTC is not guaranteed to transmit accurate width and height of videos for every frame either.
Comment 5•4 years ago
|
||
More importantly, since the function at the code returns a MediaStream
from a canvas
Expected result every time:
checking
connected
HERE
For clarity, if gather the correctly, the MediaStreamTrack
from canvas.captureStream()
is expected to not be muted
, irrespective whether or not any images have been drawn onto the canvas? Because muted
is within the domain of only networking (WebRTC), and, in fact, the canvas
itself is an image that should at least account for 1 image: meaning the MediaStreamTrack
should be enabled === true
and muted === false
because we always begin the network connect with at least 1 image, whether "black frames" or transparent frames?
Comment 6•4 years ago
|
||
Hmm, looks like it's intermittent actually. I've filed a bug.
Not intermittent. At Nightly if an image is not drawn onto the canvas
where the MediaStream
and MediaStreamTrack
therefrom is passed to addTrack()
unmute
will never fire
const pc1 = new RTCPeerConnection(),
pc2 = new RTCPeerConnection();
(async () => {
try {
const {ctx, imageData, stream} = whiteNoise();
console.log(stream.getTracks()[0].muted);
stream.getTracks()[0].onmute = stream.getTracks()[0].onunmute = e => {
console.log(e.type);
}
pc1.addTrack(stream.getTracks()[0], stream);
const { track, streams } = await new Promise(r => (pc2.ontrack = r));
// uncomment below for `unmute` to be dispatched
// ctx.putImageData(imageData, 0, 0);
// this has no effect, probably should be a Promise, not observable
// stream.requestFrame();
track.onmute = e => console.log(e);
track.onunmute = () => {
console.log('HERE');
video.srcObject = stream;
};
} catch (e) {
console.log(e);
}
})();
pc1.onicecandidate = e => pc2.addIceCandidate(e.candidate);
pc2.onicecandidate = e => pc1.addIceCandidate(e.candidate);
pc1.oniceconnectionstatechange = e => console.log(pc1.iceConnectionState);
pc1.onnegotiationneeded = async e => {
try {
await pc1.setLocalDescription(await pc1.createOffer());
await pc2.setRemoteDescription(pc1.localDescription);
await pc2.setLocalDescription(await pc2.createAnswer());
await pc1.setRemoteDescription(pc2.localDescription);
} catch (e) {
console.log(e);
}
};
function whiteNoise(width = 160, height = 120) {
const canvas = Object.assign(document.createElement('canvas'), {
width,
height,
});
const ctx = canvas.getContext('2d');
const data = new Uint8ClampedArray(
Array(canvas.width ** 2)
.fill([0, 0, 0, 255])
.flat()
);
const imageData = new ImageData(data, canvas.width);
/*
requestAnimationFrame(function draw() {
ctx.putImageData(imageData, 0, 0);
requestAnimationFrame(draw);
});
*/
return {ctx, imageData, stream: canvas.captureStream(0)};
}
Comment 7•4 years ago
|
||
To exclude for
loop and getImageData()
as a potential reason for not firing unmute
// uncomment below for `unmute` to be dispatched
ctx.drawImage(document.createElement('canvas'));
a canvas
with no context set passed to drawImage()
also results in unmute
being fired.
A MediaStreamTrack
from canvas.captureStream()
could at least have one frame of content, if not a continuous stream of the same content resulting in not muted === true
at all, equivalent to a <canvas>
with no context, a transparent image. WebRTC should interpret that data as it is, a transparent image set to default canvas
width
and height
, unless set otherwise https://github.com/web-platform-tests/wpt/pull/23091/commits/854e994232f0529d752e39859b2b4d361eb76968: capture the canvas at default frames per second or value passed to captureStream()
, irrespective of whether or not requestFrame()
(not defined as being observable) or other Canvas API is used to draw a new frame onto the canvas.
Comment 8•4 years ago
|
||
The original track is not muted
.
const { track, streams } = await new Promise(r => (pc2.ontrack = r));
console.log(stream.getTracks()[0].muted, track.muted); // false true
A WebRTC issue, not a canvas.captureStream()
issue; or are both implementations the cause of the input, output inconsistency?
Comment 9•4 years ago
|
||
At Firefox, if the context is local testing suspend
and emptied
events of HTMLMediaElement
could be used to detect a remote MediaStreamTrack
from RTCPeerConnection()
video.addEventListener(
'emptied',
e => {
alert(e.type);
if (track.muted) {
alert('emptied:'+track.muted);
ctx.drawImage(new Image(), 0, 0);
}
},
{ once: true }
);
video.addEventListener(
'suspend',
e => {
alert(e.type);
if (track.muted) {
alert('suspend:'+ track.muted);
ctx.drawImage(new Image(), 0, 0);
}
},
{ once: true }
);
video.srcObject = streams[0];
track.onmute = e => alert(e.type);
track.onunmute = e => {
if (video.srcObject.getVideoTracks()[0].id === track.id) {
alert(video.srcObject.getVideoTracks()[0].muted)
}
};
which does not output the same behaviour at Chromium, where mute
and unmute
states are toggled, am not certain if a timer or other means is used to determine when the states are changed and when the events are fired; out of the control of the application, save for drawing an image onto the the known canvas source of the stream, which might present challenges to coordinate if the stream is in fact remote. Firefox does not fire mute
event using the same code.
Updated•4 years ago
|
Comment 10•4 years ago
•
|
||
(In reply to guest271314 from comment #4)
Per https://bugs.chromium.org/p/chromium/issues/detail?id=941740#c6
The mute events on a remote track are only supposed to fire from negotiation changes (SRD/SLD) and real network issues (timeout or BYE), not content.
The
MediaStreamTrack
fromcanvas.captureStream()
should not bemuted
at all once network connection is established.
Sadly, no. webrtc-pc says that we unmute when we actually receive media packets from the other side, not when a connection is established. Therefore, the problem here is that sometimes no media packets are being transmitted when the media source is a capture stream.
Comment 11•4 years ago
|
||
Does not appear to be a (recent) regression.
Comment 12•4 years ago
|
||
Therefore, the problem here is that sometimes no media packets are being transmitted when the media source is a capture stream.
The user called captureStream()
on a canvas. canvas.captureStream()
captures the canvas, which is media, an image:
- When a user right-clicks on a
canvas
with no context they can still download the image. document.createElement('canvas').toDataURL()
returns adata URI
of a valid image.document.createElement('canvas').toBlob(blob => console.log(blob))
logsBlob {size: 1179, type: "image/png"}
, a valid image.
The first frame from of the MediaStreamTrack
should be the canvas
itself, whether the implementation set the image to transparent or "black" frame.
Further, canvas.captureStream()
could stream transparent or "black frames" at FPS.
Is the specification clear on when exactly "black frames" are streamed?
Executing clearRect()
or fillRect()
does not necessarily "change" the image appreciably from before either call.
At least the first frame should be transferred over the network, to conform with the fact that canvas
already is an image: media.
canvas.captureStream()
or WebRTC or both specifications probably need to address the issue for clarity.
Comment 13•4 years ago
|
||
Consider language in https://w3c.github.io/mediacapture-fromelement/#html-canvas-element-media-capture-extensions
In order to support manual control of frame capture with the
requestFrame()
method, browsers MUST support a value of 0 forframeRequestRate
. However, a captured stream MUST request capture of a frame when created, even ifframeRequestRate
is zero.
emphasis added, implying that "when created" a "a captured stream MUST request capture of a frame".
After describing what should occur when the stream is created, capture of a frame, without stating any paint must occur during or after that creation process, "paint" is mentioned
A new frame is requested from the canvas when
frameCaptureRequested
is true and the canvas is painted.
The on creation language states what MUST occur: the only frame that can be captured at that point is the canvas itself, which is by default, an image, media, the same initial format of media which comprises the stream after a paint, the only difference is a paint in this case can constitute a change of a pixel from value 0 to value 0 with clearRect()
or putImageData()
with the same initial image data, thus nothing has actually been painted, and the "frame" at that point is no different from the initial "frame", the default transparent or "black frame" canvas. Therefore, WebRTC connection should not be muted, because we have at least that first frame, and, technically, transparent frames could stream at FPS even if no "change" occurs at the canvas, due to the fact canvas is already an image, without a need to paint anything.If no "paint" is made after first frame, then, perhaps mute
could be fired, set, not before, because we already have media to signal. In this case we disregard what canvas is in every other case though should not.
Comment 14•4 years ago
|
||
Right, the problem lies somewhere in the interface between the webrtc code and the image capture code on the sender side. It is probably some subtle timing flaw or race.
Assignee | ||
Comment 15•4 years ago
|
||
(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #2)
Looks like this might be racing with
getImageData
https://jsfiddle.net/jib1/kzsh9wgx/32/ because without that call (34) there's no frame.
By spec there should be no frame since nothing has been drawn to the canvas.
A new frame is requested from the canvas when [[frameCaptureRequested]] is true and the canvas is painted. Each time that the captured canvas is painted, the following steps are executed:
1. For each track capturing from the canvas execute the following steps:
1. If new content has been drawn to the canvas since it was last painted, and if the [[frameCaptureRequested]] internal slot of track is set, add a new frame to track containing what was painted to the canvas.
The text is a bit confusing because a canvas is painted, we don't paint to a canvas. We draw to a canvas. And there's no good definition for these terms either, see #29, but the gist of them is clear IMO.
Assignee | ||
Comment 16•4 years ago
|
||
Assignee | ||
Updated•4 years ago
|
Assignee | ||
Comment 17•4 years ago
|
||
Assignee | ||
Comment 18•4 years ago
|
||
Assignee | ||
Comment 19•4 years ago
|
||
Assignee | ||
Updated•4 years ago
|
Comment hidden (offtopic) |
Comment hidden (offtopic) |
Comment 22•4 years ago
|
||
Pushed by pehrsons@gmail.com: https://hg.mozilla.org/integration/autoland/rev/ca9c321d4225 Add WPT checking that HTMLCanvasElement doesn't capture a frame on getImageData. r=jib https://hg.mozilla.org/integration/autoland/rev/22d52c012fd0 Don't let CanvasRenderingContext2D::EnsureTarget lead to frame capture. r=nical
Comment hidden (offtopic) |
Comment 24•4 years ago
|
||
Backed out for causing reftest failures.
Backout link: https://hg.mozilla.org/integration/autoland/rev/3361bb25ac75b33bbe9d7baece2ef3f7d61b1d2e
Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&searchStr=crash&revision=22d52c012fd00c639c0b8df4441d29ab5f05c070
Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=319001794&repo=autoland&lineNumber=7797
REFTEST TEST-START | dom/media/test/crashtests/1560215.html
[task 2020-10-19T11:26:31.538Z] 11:26:31 INFO - REFTEST TEST-LOAD | file:///builds/worker/workspace/build/tests/reftest/tests/dom/media/test/crashtests/1560215.html | 690 / 3855 (17%)
[task 2020-10-19T11:26:31.595Z] 11:26:31 INFO - [Child 1755, Main Thread] WARNING: DispatchEvent called on non-current inner window, dropping. Please check the window in the caller instead.: file /builds/worker/checkouts/gecko/dom/base/nsGlobalWindowInner.cpp:4038
[task 2020-10-19T11:26:31.596Z] 11:26:31 INFO - JavaScript error: , line 0: InvalidStateError: Navigated away from page
[task 2020-10-19T11:26:32.623Z] 11:26:32 INFO - [Child 1755, Main Thread] WARNING: Wrong inner/outer window combination!: file /builds/worker/checkouts/gecko/dom/base/Document.cpp:7134
[task 2020-10-19T11:26:32.624Z] 11:26:32 INFO - [Child 1755, Main Thread] WARNING: Wrong inner/outer window combination!: file /builds/worker/checkouts/gecko/dom/base/Document.cpp:7134
[task 2020-10-19T11:26:32.625Z] 11:26:32 INFO - [Child 1755, Main Thread] WARNING: Wrong inner/outer window combination!: file /builds/worker/checkouts/gecko/dom/base/Document.cpp:7134
[task 2020-10-19T11:31:31.558Z] 11:31:31 INFO - REFTEST TEST-UNEXPECTED-FAIL | dom/media/test/crashtests/1560215.html | load failed: timed out waiting for reftest-wait to be removed
[task 2020-10-19T11:31:31.559Z] 11:31:31 INFO - REFTEST INFO | Saved log: START file:///builds/worker/workspace/build/tests/reftest/tests/dom/media/test/crashtests/1560215.html
[task 2020-10-19T11:31:31.559Z] 11:31:31 INFO - REFTEST INFO | Saved log: [CONTENT] OnDocumentLoad triggering WaitForTestEnd
[task 2020-10-19T11:31:31.559Z] 11:31:31 INFO - REFTEST INFO | Saved log: [CONTENT] WaitForTestEnd: Adding listeners
[task 2020-10-19T11:31:31.560Z] 11:31:31 INFO - REFTEST INFO | Saved log: Initializing canvas snapshot
[task 2020-10-19T11:31:31.560Z] 11:31:31 INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress
[task 2020-10-19T11:31:31.560Z] 11:31:31 INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress: STATE_WAITING_TO_FIRE_INVALIDATE_EVENT
[task 2020-10-19T11:31:31.560Z] 11:31:31 INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress: dispatching MozReftestInvalidate
[task 2020-10-19T11:31:31.560Z] 11:31:31 INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress
[task 2020-10-19T11:31:31.560Z] 11:31:31 INFO - REFTEST INFO | Saved log: [CONTENT] AfterPaintListener in file:///builds/worker/workspace/build/tests/reftest/tests/dom/media/test/crashtests/1560215.html
[task 2020-10-19T11:31:31.560Z] 11:31:31 INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress: STATE_WAITING_FOR_REFTEST_WAIT_REMOVAL
[task 2020-10-19T11:31:31.560Z] 11:31:31 INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress: waiting for reftest-wait to be removed
[task 2020-10-19T11:31:31.560Z] 11:31:31 INFO - REFTEST INFO | Saved log: [CONTENT] HandlePendingTasksAfterMakeProgress updating canvas
[task 2020-10-19T11:31:31.560Z] 11:31:31 INFO - REFTEST INFO | Saved log: [CONTENT] SendUpdateCanvasForEvent with 1 rects
[task 2020-10-19T11:31:31.561Z] 11:31:31 INFO - REFTEST INFO | Saved log: [CONTENT] Rect: 0 0 800 1000
[task 2020-10-19T11:31:31.561Z] 11:31:31 INFO - REFTEST INFO | Saved log: Updating canvas for invalidation
[task 2020-10-19T11:31:31.561Z] 11:31:31 INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress
[task 2020-10-19T11:31:31.561Z] 11:31:31 INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress: STATE_WAITING_FOR_REFTEST_WAIT_REMOVAL
[task 2020-10-19T11:31:31.561Z] 11:31:31 INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress: waiting for reftest-wait to be removed
[task 2020-10-19T11:31:31.561Z] 11:31:31 INFO - REFTEST TEST-END | dom/media/test/crashtests/1560215.html
[task 2020-10-19T11:31:31.599Z] 11:31:31 INFO - REFTEST TEST-START | dom/media/test/crashtests/1547784.html
Assignee | ||
Comment 25•4 years ago
|
||
Thanks! I'll update the test that times out.
Assignee | ||
Comment 26•4 years ago
|
||
With the fix for this bug context.scale(64, 64) no longer results in a frame
capture. I have tested 19cf79b6f07d as reported in bug 1560215 and it reproduces
the original crash with this test change.
Comment 27•4 years ago
|
||
Pushed by pehrsons@gmail.com: https://hg.mozilla.org/integration/autoland/rev/ac4b468736bb Add WPT checking that HTMLCanvasElement doesn't capture a frame on getImageData. r=jib https://hg.mozilla.org/integration/autoland/rev/883757387f9d Don't let CanvasRenderingContext2D::EnsureTarget lead to frame capture. r=nical https://hg.mozilla.org/integration/autoland/rev/080bbf659bc5 Update crashtest to use canvas method that actually draws something. r=jib
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/26311 for changes under testing/web-platform/tests
Comment 29•4 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/ac4b468736bb
https://hg.mozilla.org/mozilla-central/rev/883757387f9d
https://hg.mozilla.org/mozilla-central/rev/080bbf659bc5
Updated•3 years ago
|
Description
•