1598237 - mute and unmute events of MediaStreamTrack from canvas.captureStream() do not fire under any circumstances

Status: RESOLVED WONTFIX

(Core :: WebRTC: Audio/Video, defect)

Description

[guest271314]

Attachment: canvasMediaStreamTrackMuteUnmuteEventsNeverFire.html

User Agent: Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/80.0.3962.2 Chrome/80.0.3962.2 Safari/537.36

Steps to reproduce:

Try using various code which should trigger mute (and unmute) events of MediaStreamTrack from canvas.captureStream().

Actual results:

mute and unmute events of MediaStreamTrack from canvas.captureStream() do not fire under any circumstances.

Expected results:

At least mute event should be fired per

A mute[1] event is raised on the MediaStreamTrack when content availability changes.

Chromium 80 fires

1. mute
2. unmute
3. mute

events using the same code.

[1] https://w3c.github.io/mediacapture-main/#event-mediastreamtrack-mute

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Comment 2

[Sotaro Ikeda [:sotaro]]

To run Attachment 9110463 [details], pref gfx.offscreencanvas.enabled:true is necessary, since offscreen canvas is not enabled by default. By enabling the offscreencanvas, unmute event was raised once.

Comment 3

[Sotaro Ikeda [:sotaro]]

guest271314, did you enable offscreencanvas with pref gfx.offscreencanvas.enabled:true in about:config?

Comment 4

[guest271314]

Attachment: noMediaTransferredFromCanvasToMediaStreamTrackFirefox.html

(In reply to Sotaro Ikeda [:sotaro] from comment #3)

guest271314, did you enable offscreencanvas with pref gfx.offscreencanvas.enabled:true in about:config?

Yes.

When ImageBitmapRenderingContext and transferFromImageBitmap() are used no media are transmitted from the <canvas> to the MediaStreamTrack (https://bugzilla.mozilla.org/show_bug.cgi?id=1590030, https://bugzilla.mozilla.org/show_bug.cgi?id=1557980)

At Firefox 70 the image is displayed at the <canvas>.

At Nightly 72 the image is not displayed at the <canvas>. To view the image it is necessary to either draw the <canvas> onto another <canvas> or right-click and select "View image" which navigates to a Blob URL where the image is displayed.

To reproduce run attached file 9110463: canvasMediaStreamTrackMuteUnmuteEventsNeverFire.html

Comment 5

[Sotaro Ikeda [:sotaro]]

Thanks for the detail! From comment 4, I noticed that WebRender does not handle ImageBitmap. Then created Bug 1599653.

Comment 6

[Sotaro Ikeda [:sotaro]]

From Attachment 9111627 [details], I also noticed a problem of video tag rendering of canvas.captureStream(). Then created Bug 1599696.

Comment 7

[guest271314]

Once https://phabricator.services.mozilla.com/D54890 and https://phabricator.services.mozilla.com/D54893 are reviewed and merged the changes made will be available at Nightly, correct?

Comment 8

[Sotaro Ikeda [:sotaro]]

(In reply to guest271314 from comment #7)

Once https://phabricator.services.mozilla.com/D54890 and https://phabricator.services.mozilla.com/D54893 are reviewed and merged the changes made will be available at Nightly, correct?

Yes, correct. m-c is soft freeze this week, then I am going to try to land them next week.

Comment 9

[guest271314]

Why does Firefox 70 display the image on canvas when ctx.transferFromImageBitmap(bitmap) is executed, where Nightly 72 does not?

Comment 10

[Sotaro Ikeda [:sotaro]]

(In reply to guest271314 from comment #9)

Why does Firefox 70 display the image on canvas when ctx.transferFromImageBitmap(bitmap) is executed, where Nightly 72 does not?

Can you attach about:support info to the bug? Nightly 72 might use WebRender as compositor. In this case, WebRender does not support canvas ImageBitmap rendering. It is Bug 1599653.

Comment 11

[guest271314]

Attachment: about_support_nightly_72.txt

Comment 12

[guest271314]

Attachment: about_support_firefox_70.txt

Comment 13

[guest271314]

According to about:support at Nightly WebRender is

opt-in by default: WebRender is an opt-in feature
available by user: Qualified enabled by pref

How to set "Compositing" to "Basic" (Firefox default setting) at Nightly?

Comment 14

[guest271314]

According to https://www.askvg.com/tip-how-to-enable-or-disable-new-webrender-feature-in-mozilla-firefox/ WebRender can be disabled by setting gfx.webrender.all to false, however that does not appear to have any affect where the same values are printed at about:support.

Comment 15

[Sotaro Ikeda [:sotaro]]

Can you try to set pref gfx.webrender.force-disabled:true ?

Comment 16

[guest271314]

Same result.

Comment 17

[guest271314]

(In reply to guest271314 from comment #16)

<s>Same result.</s>

Had to restart Nightly after setting gfx.webrender.force-disabled to true. The the image does display.

Comment 18

[guest271314]

Still does not display image when transferControlToOffscreen() is used in a Worker.

Comment 19

[Sotaro Ikeda [:sotaro]]

(In reply to guest271314 from comment #18)

Still does not display image when transferControlToOffscreen() is used in a Worker.

guest271314, is there a sample of doing it? With Attachment 9110463 [details], image was not rendered because canvas is not appended in document.

Comment 20

[guest271314]

Attachment: offscreenCanvasInWorkerDoesNotDisplayImageOnCanvas.html

Comment 21

[guest271314]

Attachment: offscreenCanvasInWorkerDoesNotDisplayImageOnCanvasWorker.js

Comment 22

[guest271314]

(In reply to Sotaro Ikeda [:sotaro] from comment #19)

(In reply to guest271314 from comment #18)

Still does not display image when transferControlToOffscreen() is used in a Worker.

guest271314, is there a sample of doing it?

See attachments https://bugzilla.mozilla.org/attachment.cgi?id=9113391, https://bugzilla.mozilla.org/attachment.cgi?id=9113392, plnkr https://plnkr.co/edit/eXESuVqOp5J9AAAft07E?p=preview which demonstrate the transferControlToOffscreen(), "bitmaprenderer", transferFromimageBitmap() does not display the image onto the <canvas>. There are previously filed bugs re that issue https://bugzilla.mozilla.org/show_bug.cgi?id=1590030 which are not fixed as yet, and related to this bug, that is why mentioned those cases.

(As an aside if the OffscreenCanvas is transferred back to the main thread from a Worker none of the methods are executable, .toBlobb() will throw not a function)

With Attachment 9110463 [details], image was not rendered because canvas is not appended in document.

At Nightly 73 build https://hg.mozilla.org/mozilla-central/rev/3c08edf74d039af79f9daad8ff5b57ffb64fdab6 the <canvas> does not need to be appended to the document to be displayed at <video>. That attachment is to demostrate that mute, unmute, ended events of MediaStreamTrack do not fire, which is still the case at the above linked build. This bug https://bugzilla.mozilla.org/show_bug.cgi?id=1557980 is similar to the present issue. Have not yet tested MediaRecorder after the relevant fixes, though should record, as long as the canvas is not offscreen in a Worker.

Comment 23

[guest271314]

MediaRecorder records the image when transferFromImageBitmap() is used now.

Comment 24

[guest271314]

Re

There are previously filed bugs re that issue https://bugzilla.mozilla.org/show_bug.cgi?id=1590030 which are not fixed as yet

Correction: That bug appears to be fixed at Nightly 73 with the changes made to canvas.

Comment 25

[guest271314]

This bug was filed because https://www.w3.org/TR/mediacapture-fromelement/

A mute event is raised on the MediaStreamTrack when content availability changes.

does not occur at Firefox or Nightly under any cases that have that far tried.

Is that by design or a bug?

Comment 26

[guest271314]

If we adjust the code at https://bugzilla.mozilla.org/attachment.cgi?id=9113391 and https://bugzilla.mozilla.org/attachment.cgi?id=9113392 to use transferToImageBitmap() we can observe the bitmap is transferred to the OffscreenCanvas within the Worker, though to display the image requires creating a new <canvas>. This should be possible using the single original <canvas>.

const bmp = offscreen.transferToImageBitmap();
postMessage({bmp}, [bmp]);

worker.onmessage = e => {
  const c = document.createElement('canvas')
  const x = c.getContext('bitmaprenderer')
  document.body.appendChild(c)
  c.width = e.data.bmp.width;
  c.height = e.data.bmp.height;
  x.transferFromImageBitmap(e.data.bmp)
}

Comment 27

[Sotaro Ikeda [:sotaro]]

(In reply to guest271314 from comment #25)

Is that by design or a bug?

HTMLMediaElement seems to handle mute event correctly by using HTMLMediaElement::MediaElementTrackSource. But CanvasCaptureMediaStream seems not handle mute event. I wonder if this is a bug.

:pehrsons, can you comment to Comment 25?

Comment 28

[Andreas Pehrson [:pehrsons]]

(In reply to guest271314 from comment #25)

This bug was filed because https://www.w3.org/TR/mediacapture-fromelement/

A mute event is raised on the MediaStreamTrack when content availability changes.

does not occur at Firefox or Nightly under any cases that have that far tried.

Is that by design or a bug?

That spec language is for HTMLMediaElement capture, not HTMLCanvasElement capture. Thus, by design we are spec compliant for mute/unmute events for HTMLCanvasElement capture.

What chromium does is up to them. But in my opinion, making interoperable implementations requires a spec that defines the behavior to follow.

Closing this as WONTFIX as it needs to be resolved in the spec first.

Comment 29

[guest271314]

(In reply to Andreas Pehrson [:pehrsons] from comment #28)

(In reply to guest271314 from comment #25)

This bug was filed because https://www.w3.org/TR/mediacapture-fromelement/

A mute event is raised on the MediaStreamTrack when content availability changes.

does not occur at Firefox or Nightly under any cases that have that far tried.

Is that by design or a bug?

That spec language is for HTMLMediaElement capture, not HTMLCanvasElement capture. Thus, by design we are spec compliant for mute/unmute events for HTMLCanvasElement capture.

What chromium does is up to them. But in my opinion, making interoperable implementations requires a spec that defines the behavior to follow.

Closing this as WONTFIX as it needs to be resolved in the spec first.

https://github.com/w3c/mediacapture-fromelement/issues/82

Comment 30

[guest271314]

(In reply to Sotaro Ikeda [:sotaro] from comment #19)

(In reply to guest271314 from comment #18)

Still does not display image when transferControlToOffscreen() is used in a Worker.

guest271314, is there a sample of doing it? With Attachment 9110463 [details], image was not rendered because canvas is not appended in document.

Can you kindly take a look at https://bugzilla.mozilla.org/show_bug.cgi?id=1598998, which describes the bug with OffscreenCanvas, transferControlToOffscreen() and transferFromImageBitmap() in a Worker?

Comment 31

[guest271314]

Attachment: firefoxDoesNotSetCanvasCaptureMediaStreamTrackMutedTrueUnderAnyCircumstances.html

(In reply to Andreas Pehrson [:pehrsons] from comment #28)

(In reply to guest271314 from comment #25)

This bug was filed because https://www.w3.org/TR/mediacapture-fromelement/

A mute event is raised on the MediaStreamTrack when content availability changes.

does not occur at Firefox or Nightly under any cases that have that far tried.

Is that by design or a bug?

That spec language is for HTMLMediaElement capture, not HTMLCanvasElement capture. Thus, by design we are spec compliant for mute/unmute events for HTMLCanvasElement capture.

What chromium does is up to them. But in my opinion, making interoperable implementations requires a spec that defines the behavior to follow.

Closing this as WONTFIX as it needs to be resolved in the spec first.

Re specification issue https://github.com/w3c/mediacapture-fromelement/issues/82 if

A captured stream MUST immediately cease to capture content if the origin-clean flag of the source canvas becomes false after the stream is created by captureStream(). The captured MediaStreamTrack MUST become muted, producing no new content while the canvas remains in this state.

is the only case where CanvasCaptureMediaStreamTrack muted property MUST be set to true Firefox 70 and Night 73 fail.

Comment 32

[Andreas Pehrson [:pehrsons]]

Bug 1231131 should fix that.