Closed Bug 1631476 Opened 4 years ago Closed 4 years ago

CanvasRenderingContext2D with live captureStream captures a frame on first EnsureTarget

Categories

(Core :: Graphics: Canvas2D, defect, P2)

defect

Tracking

()

RESOLVED FIXED
84 Branch
Tracking Status
firefox-esr68 --- wontfix
firefox75 --- wontfix
firefox76 --- wontfix
firefox77 --- wontfix
firefox78 --- wontfix
firefox84 --- fixed

People

(Reporter: jib, Assigned: pehrsons)

Details

Attachments

(3 files)

STRs:

  1. Open https://jsfiddle.net/jib1/kzsh9wgx/17/
  2. DON'T hit the "Start" button.
  3. Hit JSFiddle's ▶ Run button 10 times a couple of seconds apart.

Expected result every time:

checking
connected
HERE

Actual result ~1/5 of the time:

checking
connected

Looks like this might be racing with getImageData https://jsfiddle.net/jib1/kzsh9wgx/32/ because without that call (34) there's no frame.

getImageData() is not the issue. muted behaviour is inconsistent.

BTW, Nightly is crashing the tab when attempting to load jsfiddle. Used plnkr to reproduce.

Per https://bugs.chromium.org/p/chromium/issues/detail?id=941740#c6

The mute events on a remote track are only supposed to fire from negotiation changes (SRD/SLD) and real network issues (timeout or BYE), not content.

The MediaStreamTrack from canvas.captureStream() should not be muted at all once network connection is established.

Is this a case of WebRTC not being observable with regard to the exact timing of network connections, where the only observable behaviour from that connection is the inconsistency as to when the connection is established?

Could be 1 second this run, could be 45 seconds next run when unmute is dispatched - and not just from a canvas.captureStream()?

That is precisely what occurs at this code https://plnkr.co/edit/Axkb8s?preview (change codec at MediaRecorder for Nightly to run). Could take 2 seconds after first run (if cache is enabled), could take 30 seconds third run, just have to await unmute and resize events - because WebRTC is not guaranteed to transmit accurate width and height of videos for every frame either.

More importantly, since the function at the code returns a MediaStream from a canvas

Expected result every time:

checking
connected
HERE

For clarity, if gather the correctly, the MediaStreamTrack from canvas.captureStream() is expected to not be muted, irrespective whether or not any images have been drawn onto the canvas? Because muted is within the domain of only networking (WebRTC), and, in fact, the canvas itself is an image that should at least account for 1 image: meaning the MediaStreamTrack should be enabled === true and muted === false because we always begin the network connect with at least 1 image, whether "black frames" or transparent frames?

Hmm, looks like it's intermittent actually. I've filed a bug.

Not intermittent. At Nightly if an image is not drawn onto the canvas where the MediaStream and MediaStreamTrack therefrom is passed to addTrack() unmute will never fire


      const pc1 = new RTCPeerConnection(),
        pc2 = new RTCPeerConnection();

      (async () => {
        try {
          const {ctx, imageData, stream} = whiteNoise();
          console.log(stream.getTracks()[0].muted);
          stream.getTracks()[0].onmute = stream.getTracks()[0].onunmute = e => {
            console.log(e.type);
          }
          pc1.addTrack(stream.getTracks()[0], stream);
          const { track, streams } = await new Promise(r => (pc2.ontrack = r));
          // uncomment below for `unmute` to be dispatched
          // ctx.putImageData(imageData, 0, 0);
          // this has no effect, probably should be a Promise, not observable
          // stream.requestFrame();
          track.onmute = e => console.log(e);
          track.onunmute = () => {
            console.log('HERE');
            video.srcObject = stream;
          };
        } catch (e) {
          console.log(e);
        }
      })();

      pc1.onicecandidate = e => pc2.addIceCandidate(e.candidate);
      pc2.onicecandidate = e => pc1.addIceCandidate(e.candidate);
      pc1.oniceconnectionstatechange = e => console.log(pc1.iceConnectionState);

      pc1.onnegotiationneeded = async e => {
        try {
          await pc1.setLocalDescription(await pc1.createOffer());
          await pc2.setRemoteDescription(pc1.localDescription);
          await pc2.setLocalDescription(await pc2.createAnswer());
          await pc1.setRemoteDescription(pc2.localDescription);
        } catch (e) {
          console.log(e);
        }
      };

      function whiteNoise(width = 160, height = 120) {
        const canvas = Object.assign(document.createElement('canvas'), {
          width,
          height,
        });
        const ctx = canvas.getContext('2d');
        const data = new Uint8ClampedArray(
          Array(canvas.width ** 2)
            .fill([0, 0, 0, 255])
            .flat()
        );

        const imageData = new ImageData(data, canvas.width);

        /*
        requestAnimationFrame(function draw() {
          ctx.putImageData(imageData, 0, 0);
          requestAnimationFrame(draw);
        });
        */
        return {ctx, imageData, stream: canvas.captureStream(0)};
      }

To exclude for loop and getImageData() as a potential reason for not firing unmute

// uncomment below for `unmute` to be dispatched
ctx.drawImage(document.createElement('canvas'));

a canvas with no context set passed to drawImage() also results in unmute being fired.

A MediaStreamTrack from canvas.captureStream() could at least have one frame of content, if not a continuous stream of the same content resulting in not muted === true at all, equivalent to a <canvas> with no context, a transparent image. WebRTC should interpret that data as it is, a transparent image set to default canvas width and height, unless set otherwise https://github.com/web-platform-tests/wpt/pull/23091/commits/854e994232f0529d752e39859b2b4d361eb76968: capture the canvas at default frames per second or value passed to captureStream(), irrespective of whether or not requestFrame() (not defined as being observable) or other Canvas API is used to draw a new frame onto the canvas.

The original track is not muted.

const { track, streams } = await new Promise(r => (pc2.ontrack = r));
console.log(stream.getTracks()[0].muted, track.muted); // false true

A WebRTC issue, not a canvas.captureStream() issue; or are both implementations the cause of the input, output inconsistency?

At Firefox, if the context is local testing suspend and emptied events of HTMLMediaElement could be used to detect a remote MediaStreamTrack from RTCPeerConnection()

          video.addEventListener(
            'emptied',
            e => {
              alert(e.type);
              if (track.muted) {
                alert('emptied:'+track.muted);
                ctx.drawImage(new Image(), 0, 0);
              }
            },
            { once: true }
          );
          video.addEventListener(
            'suspend',
            e => {
              alert(e.type);
              if (track.muted) {
                alert('suspend:'+ track.muted);
                ctx.drawImage(new Image(), 0, 0);
              }
            },
            { once: true }
          );
          video.srcObject = streams[0];
          track.onmute = e => alert(e.type);
          track.onunmute = e => {
            if (video.srcObject.getVideoTracks()[0].id === track.id) {
              alert(video.srcObject.getVideoTracks()[0].muted)
            }
          };

which does not output the same behaviour at Chromium, where mute and unmute states are toggled, am not certain if a timer or other means is used to determine when the states are changed and when the events are fired; out of the control of the application, save for drawing an image onto the the known canvas source of the stream, which might present challenges to coordinate if the stream is in fact remote. Firefox does not fire mute event using the same code.

Priority: -- → P2

(In reply to guest271314 from comment #4)

Per https://bugs.chromium.org/p/chromium/issues/detail?id=941740#c6

The mute events on a remote track are only supposed to fire from negotiation changes (SRD/SLD) and real network issues (timeout or BYE), not content.

The MediaStreamTrack from canvas.captureStream() should not be muted at all once network connection is established.

Sadly, no. webrtc-pc says that we unmute when we actually receive media packets from the other side, not when a connection is established. Therefore, the problem here is that sometimes no media packets are being transmitted when the media source is a capture stream.

Does not appear to be a (recent) regression.

Therefore, the problem here is that sometimes no media packets are being transmitted when the media source is a capture stream.

The user called captureStream() on a canvas. canvas.captureStream() captures the canvas, which is media, an image:

  • When a user right-clicks on a canvas with no context they can still download the image.
  • document.createElement('canvas').toDataURL() returns a data URI of a valid image.
  • document.createElement('canvas').toBlob(blob => console.log(blob)) logs Blob {size: 1179, type: "image/png"}, a valid image.

The first frame from of the MediaStreamTrack should be the canvas itself, whether the implementation set the image to transparent or "black" frame.

Further, canvas.captureStream() could stream transparent or "black frames" at FPS.

Is the specification clear on when exactly "black frames" are streamed?

Executing clearRect() or fillRect() does not necessarily "change" the image appreciably from before either call.

At least the first frame should be transferred over the network, to conform with the fact that canvas already is an image: media.

canvas.captureStream() or WebRTC or both specifications probably need to address the issue for clarity.

Consider language in https://w3c.github.io/mediacapture-fromelement/#html-canvas-element-media-capture-extensions

In order to support manual control of frame capture with the requestFrame() method, browsers MUST support a value of 0 for frameRequestRate. However, a captured stream MUST request capture of a frame when created, even if frameRequestRate is zero.

emphasis added, implying that "when created" a "a captured stream MUST request capture of a frame".

After describing what should occur when the stream is created, capture of a frame, without stating any paint must occur during or after that creation process, "paint" is mentioned

A new frame is requested from the canvas when frameCaptureRequested is true and the canvas is painted.

The on creation language states what MUST occur: the only frame that can be captured at that point is the canvas itself, which is by default, an image, media, the same initial format of media which comprises the stream after a paint, the only difference is a paint in this case can constitute a change of a pixel from value 0 to value 0 with clearRect() or putImageData() with the same initial image data, thus nothing has actually been painted, and the "frame" at that point is no different from the initial "frame", the default transparent or "black frame" canvas. Therefore, WebRTC connection should not be muted, because we have at least that first frame, and, technically, transparent frames could stream at FPS even if no "change" occurs at the canvas, due to the fact canvas is already an image, without a need to paint anything.If no "paint" is made after first frame, then, perhaps mute could be fired, set, not before, because we already have media to signal. In this case we disregard what canvas is in every other case though should not.

Right, the problem lies somewhere in the interface between the webrtc code and the image capture code on the sender side. It is probably some subtle timing flaw or race.

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #2)

Looks like this might be racing with getImageData https://jsfiddle.net/jib1/kzsh9wgx/32/ because without that call (34) there's no frame.

By spec there should be no frame since nothing has been drawn to the canvas.

 A new frame is requested from the canvas when [[frameCaptureRequested]] is true and the canvas is painted. Each time that the captured canvas is painted, the following steps are executed:

1. For each track capturing from the canvas execute the following steps:
    1. If new content has been drawn to the canvas since it was last painted, and if the [[frameCaptureRequested]] internal slot of track is set, add a new frame to track containing what was painted to the canvas. 

The text is a bit confusing because a canvas is painted, we don't paint to a canvas. We draw to a canvas. And there's no good definition for these terms either, see #29, but the gist of them is clear IMO.

This -> is -> the -> reason for the black frame, in that order on the stack. getImageData() makes the canvas think it's been drawn to, so a fully transparent frame is captured.

Assignee: nobody → apehrson
Status: NEW → ASSIGNED
Component: WebRTC → Canvas: 2D
Summary: canvas.captureStream over RTCPeerConnection intermittently fails to unmute remote track. → CanvasRenderingContext2D with live captureStream captures a frame on first EnsureTarget
OS: macOS → Unspecified
Pushed by pehrsons@gmail.com:
https://hg.mozilla.org/integration/autoland/rev/ca9c321d4225
Add WPT checking that HTMLCanvasElement doesn't capture a frame on getImageData. r=jib
https://hg.mozilla.org/integration/autoland/rev/22d52c012fd0
Don't let CanvasRenderingContext2D::EnsureTarget lead to frame capture. r=nical

Backed out for causing reftest failures.

Backout link: https://hg.mozilla.org/integration/autoland/rev/3361bb25ac75b33bbe9d7baece2ef3f7d61b1d2e

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&searchStr=crash&revision=22d52c012fd00c639c0b8df4441d29ab5f05c070

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=319001794&repo=autoland&lineNumber=7797

REFTEST TEST-START | dom/media/test/crashtests/1560215.html
[task 2020-10-19T11:26:31.538Z] 11:26:31 INFO - REFTEST TEST-LOAD | file:///builds/worker/workspace/build/tests/reftest/tests/dom/media/test/crashtests/1560215.html | 690 / 3855 (17%)
[task 2020-10-19T11:26:31.595Z] 11:26:31 INFO - [Child 1755, Main Thread] WARNING: DispatchEvent called on non-current inner window, dropping. Please check the window in the caller instead.: file /builds/worker/checkouts/gecko/dom/base/nsGlobalWindowInner.cpp:4038
[task 2020-10-19T11:26:31.596Z] 11:26:31 INFO - JavaScript error: , line 0: InvalidStateError: Navigated away from page
[task 2020-10-19T11:26:32.623Z] 11:26:32 INFO - [Child 1755, Main Thread] WARNING: Wrong inner/outer window combination!: file /builds/worker/checkouts/gecko/dom/base/Document.cpp:7134
[task 2020-10-19T11:26:32.624Z] 11:26:32 INFO - [Child 1755, Main Thread] WARNING: Wrong inner/outer window combination!: file /builds/worker/checkouts/gecko/dom/base/Document.cpp:7134
[task 2020-10-19T11:26:32.625Z] 11:26:32 INFO - [Child 1755, Main Thread] WARNING: Wrong inner/outer window combination!: file /builds/worker/checkouts/gecko/dom/base/Document.cpp:7134
[task 2020-10-19T11:31:31.558Z] 11:31:31 INFO - REFTEST TEST-UNEXPECTED-FAIL | dom/media/test/crashtests/1560215.html | load failed: timed out waiting for reftest-wait to be removed
[task 2020-10-19T11:31:31.559Z] 11:31:31 INFO - REFTEST INFO | Saved log: START file:///builds/worker/workspace/build/tests/reftest/tests/dom/media/test/crashtests/1560215.html
[task 2020-10-19T11:31:31.559Z] 11:31:31 INFO - REFTEST INFO | Saved log: [CONTENT] OnDocumentLoad triggering WaitForTestEnd
[task 2020-10-19T11:31:31.559Z] 11:31:31 INFO - REFTEST INFO | Saved log: [CONTENT] WaitForTestEnd: Adding listeners
[task 2020-10-19T11:31:31.560Z] 11:31:31 INFO - REFTEST INFO | Saved log: Initializing canvas snapshot
[task 2020-10-19T11:31:31.560Z] 11:31:31 INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress
[task 2020-10-19T11:31:31.560Z] 11:31:31 INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress: STATE_WAITING_TO_FIRE_INVALIDATE_EVENT
[task 2020-10-19T11:31:31.560Z] 11:31:31 INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress: dispatching MozReftestInvalidate
[task 2020-10-19T11:31:31.560Z] 11:31:31 INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress
[task 2020-10-19T11:31:31.560Z] 11:31:31 INFO - REFTEST INFO | Saved log: [CONTENT] AfterPaintListener in file:///builds/worker/workspace/build/tests/reftest/tests/dom/media/test/crashtests/1560215.html
[task 2020-10-19T11:31:31.560Z] 11:31:31 INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress: STATE_WAITING_FOR_REFTEST_WAIT_REMOVAL
[task 2020-10-19T11:31:31.560Z] 11:31:31 INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress: waiting for reftest-wait to be removed
[task 2020-10-19T11:31:31.560Z] 11:31:31 INFO - REFTEST INFO | Saved log: [CONTENT] HandlePendingTasksAfterMakeProgress updating canvas
[task 2020-10-19T11:31:31.560Z] 11:31:31 INFO - REFTEST INFO | Saved log: [CONTENT] SendUpdateCanvasForEvent with 1 rects
[task 2020-10-19T11:31:31.561Z] 11:31:31 INFO - REFTEST INFO | Saved log: [CONTENT] Rect: 0 0 800 1000
[task 2020-10-19T11:31:31.561Z] 11:31:31 INFO - REFTEST INFO | Saved log: Updating canvas for invalidation
[task 2020-10-19T11:31:31.561Z] 11:31:31 INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress
[task 2020-10-19T11:31:31.561Z] 11:31:31 INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress: STATE_WAITING_FOR_REFTEST_WAIT_REMOVAL
[task 2020-10-19T11:31:31.561Z] 11:31:31 INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress: waiting for reftest-wait to be removed
[task 2020-10-19T11:31:31.561Z] 11:31:31 INFO - REFTEST TEST-END | dom/media/test/crashtests/1560215.html
[task 2020-10-19T11:31:31.599Z] 11:31:31 INFO - REFTEST TEST-START | dom/media/test/crashtests/1547784.html

Flags: needinfo?(apehrson)

Thanks! I'll update the test that times out.

Flags: needinfo?(apehrson)

With the fix for this bug context.scale(64, 64) no longer results in a frame
capture. I have tested 19cf79b6f07d as reported in bug 1560215 and it reproduces
the original crash with this test change.

Pushed by pehrsons@gmail.com:
https://hg.mozilla.org/integration/autoland/rev/ac4b468736bb
Add WPT checking that HTMLCanvasElement doesn't capture a frame on getImageData. r=jib
https://hg.mozilla.org/integration/autoland/rev/883757387f9d
Don't let CanvasRenderingContext2D::EnsureTarget lead to frame capture. r=nical
https://hg.mozilla.org/integration/autoland/rev/080bbf659bc5
Update crashtest to use canvas method that actually draws something. r=jib
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/26311 for changes under testing/web-platform/tests
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → 84 Branch
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: