mute and unmute events of MediaStreamTrack from canvas.captureStream() do not fire under any circumstances
Categories
(Core :: WebRTC: Audio/Video, defect)
Tracking
()
People
(Reporter: guest271314, Unassigned)
References
Details
Attachments
(7 files)
User Agent: Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/80.0.3962.2 Chrome/80.0.3962.2 Safari/537.36
Steps to reproduce:
Try using various code which should trigger mute (and unmute) events of MediaStreamTrack from canvas.captureStream().
Actual results:
mute and unmute events of MediaStreamTrack from canvas.captureStream() do not fire under any circumstances.
Expected results:
At least mute event should be fired per
A mute[1] event is raised on the MediaStreamTrack when content availability changes.
Chromium 80 fires
- mute
- unmute
- mute
events using the same code.
[1] https://w3c.github.io/mediacapture-main/#event-mediastreamtrack-mute
Comment 1•4 years ago
|
||
Bugbug thinks this bug should belong to this component, but please revert this change in case of error.
Comment 2•4 years ago
•
|
||
To run Attachment 9110463 [details], pref gfx.offscreencanvas.enabled:true is necessary, since offscreen canvas is not enabled by default. By enabling the offscreencanvas, unmute event was raised once.
Updated•4 years ago
|
Comment 3•4 years ago
|
||
guest271314, did you enable offscreencanvas with pref gfx.offscreencanvas.enabled:true in about:config?
Reporter | ||
Comment 4•4 years ago
|
||
(In reply to Sotaro Ikeda [:sotaro] from comment #3)
guest271314, did you enable offscreencanvas with pref gfx.offscreencanvas.enabled:true in about:config?
Yes.
When ImageBitmapRenderingContext
and transferFromImageBitmap()
are used no media are transmitted from the <canvas>
to the MediaStreamTrack
(https://bugzilla.mozilla.org/show_bug.cgi?id=1590030, https://bugzilla.mozilla.org/show_bug.cgi?id=1557980)
At Firefox 70 the image is displayed at the <canvas>
.
At Nightly 72 the image is not displayed at the <canvas>
. To view the image it is necessary to either draw the <canvas>
onto another <canvas>
or right-click and select "View image" which navigates to a Blob URL where the image is displayed.
To reproduce run attached file 9110463: canvasMediaStreamTrackMuteUnmuteEventsNeverFire.html
Comment 5•4 years ago
|
||
Thanks for the detail! From comment 4, I noticed that WebRender does not handle ImageBitmap. Then created Bug 1599653.
Updated•4 years ago
|
Updated•4 years ago
|
Comment 6•4 years ago
|
||
From Attachment 9111627 [details], I also noticed a problem of video tag rendering of canvas.captureStream(). Then created Bug 1599696.
Updated•4 years ago
|
Reporter | ||
Comment 7•4 years ago
|
||
Once https://phabricator.services.mozilla.com/D54890 and https://phabricator.services.mozilla.com/D54893 are reviewed and merged the changes made will be available at Nightly, correct?
Comment 8•4 years ago
|
||
(In reply to guest271314 from comment #7)
Once https://phabricator.services.mozilla.com/D54890 and https://phabricator.services.mozilla.com/D54893 are reviewed and merged the changes made will be available at Nightly, correct?
Yes, correct. m-c is soft freeze this week, then I am going to try to land them next week.
Reporter | ||
Comment 9•4 years ago
|
||
Why does Firefox 70 display the image on canvas
when ctx.transferFromImageBitmap(bitmap)
is executed, where Nightly 72 does not?
Comment 10•4 years ago
•
|
||
(In reply to guest271314 from comment #9)
Why does Firefox 70 display the image on
canvas
whenctx.transferFromImageBitmap(bitmap)
is executed, where Nightly 72 does not?
Can you attach about:support info to the bug? Nightly 72 might use WebRender as compositor. In this case, WebRender does not support canvas ImageBitmap rendering. It is Bug 1599653.
Reporter | ||
Comment 11•4 years ago
|
||
Reporter | ||
Comment 12•4 years ago
|
||
Reporter | ||
Comment 13•4 years ago
|
||
According to about:support
at Nightly WebRender is
opt-in by default: WebRender is an opt-in feature
available by user: Qualified enabled by pref
How to set "Compositing" to "Basic" (Firefox default setting) at Nightly?
Reporter | ||
Comment 14•4 years ago
|
||
According to https://www.askvg.com/tip-how-to-enable-or-disable-new-webrender-feature-in-mozilla-firefox/ WebRender can be disabled by setting gfx.webrender.all
to false
, however that does not appear to have any affect where the same values are printed at about:support.
Comment 15•4 years ago
|
||
Can you try to set pref gfx.webrender.force-disabled:true ?
Reporter | ||
Comment 17•4 years ago
|
||
(In reply to guest271314 from comment #16)
<s>Same result.</s>
Had to restart Nightly after setting gfx.webrender.force-disabled
to true
. The the image does display.
Reporter | ||
Comment 18•4 years ago
|
||
Still does not display image when transferControlToOffscreen()
is used in a Worker
.
Updated•4 years ago
|
Comment 19•4 years ago
|
||
(In reply to guest271314 from comment #18)
Still does not display image when
transferControlToOffscreen()
is used in aWorker
.
guest271314, is there a sample of doing it? With Attachment 9110463 [details], image was not rendered because canvas is not appended in document.
Reporter | ||
Comment 20•4 years ago
|
||
Reporter | ||
Comment 21•4 years ago
|
||
Reporter | ||
Comment 22•4 years ago
|
||
(In reply to Sotaro Ikeda [:sotaro] from comment #19)
(In reply to guest271314 from comment #18)
Still does not display image when
transferControlToOffscreen()
is used in aWorker
.guest271314, is there a sample of doing it?
See attachments https://bugzilla.mozilla.org/attachment.cgi?id=9113391, https://bugzilla.mozilla.org/attachment.cgi?id=9113392, plnkr https://plnkr.co/edit/eXESuVqOp5J9AAAft07E?p=preview which demonstrate the transferControlToOffscreen()
, "bitmaprenderer"
, transferFromimageBitmap()
does not display the image onto the <canvas>
. There are previously filed bugs re that issue https://bugzilla.mozilla.org/show_bug.cgi?id=1590030 which are not fixed as yet, and related to this bug, that is why mentioned those cases.
(As an aside if the OffscreenCanvas
is transferred back to the main thread from a Worker
none of the methods are executable, .toBlobb()
will throw not a function)
With Attachment 9110463 [details], image was not rendered because canvas is not appended in document.
At Nightly 73 build https://hg.mozilla.org/mozilla-central/rev/3c08edf74d039af79f9daad8ff5b57ffb64fdab6 the <canvas>
does not need to be appended to the document
to be displayed at <video>
. That attachment is to demostrate that mute
, unmute
, ended
events of MediaStreamTrack
do not fire, which is still the case at the above linked build. This bug https://bugzilla.mozilla.org/show_bug.cgi?id=1557980 is similar to the present issue. Have not yet tested MediaRecorder
after the relevant fixes, though should record, as long as the canvas
is not offscreen in a Worker
.
Reporter | ||
Comment 23•4 years ago
|
||
MediaRecorder
records the image when transferFromImageBitmap()
is used now.
Reporter | ||
Comment 24•4 years ago
|
||
Re
There are previously filed bugs re that issue https://bugzilla.mozilla.org/show_bug.cgi?id=1590030 which are not fixed as yet
Correction: That bug appears to be fixed at Nightly 73 with the changes made to canvas
.
Reporter | ||
Comment 25•4 years ago
|
||
This bug was filed because https://www.w3.org/TR/mediacapture-fromelement/
A mute event is raised on the MediaStreamTrack when content availability changes.
does not occur at Firefox or Nightly under any cases that have that far tried.
Is that by design or a bug?
Reporter | ||
Comment 26•4 years ago
|
||
If we adjust the code at https://bugzilla.mozilla.org/attachment.cgi?id=9113391 and https://bugzilla.mozilla.org/attachment.cgi?id=9113392 to use transferToImageBitmap()
we can observe the bitmap is transferred to the OffscreenCanvas
within the Worker
, though to display the image requires creating a new <canvas>
. This should be possible using the single original <canvas>
.
const bmp = offscreen.transferToImageBitmap();
postMessage({bmp}, [bmp]);
worker.onmessage = e => {
const c = document.createElement('canvas')
const x = c.getContext('bitmaprenderer')
document.body.appendChild(c)
c.width = e.data.bmp.width;
c.height = e.data.bmp.height;
x.transferFromImageBitmap(e.data.bmp)
}
Comment 27•4 years ago
•
|
||
(In reply to guest271314 from comment #25)
Is that by design or a bug?
HTMLMediaElement seems to handle mute event correctly by using HTMLMediaElement::MediaElementTrackSource. But CanvasCaptureMediaStream seems not handle mute event. I wonder if this is a bug.
:pehrsons, can you comment to Comment 25?
Comment 28•4 years ago
|
||
(In reply to guest271314 from comment #25)
This bug was filed because https://www.w3.org/TR/mediacapture-fromelement/
A mute event is raised on the MediaStreamTrack when content availability changes.
does not occur at Firefox or Nightly under any cases that have that far tried.
Is that by design or a bug?
That spec language is for HTMLMediaElement capture, not HTMLCanvasElement capture. Thus, by design we are spec compliant for mute/unmute events for HTMLCanvasElement capture.
What chromium does is up to them. But in my opinion, making interoperable implementations requires a spec that defines the behavior to follow.
Closing this as WONTFIX as it needs to be resolved in the spec first.
Reporter | ||
Comment 29•4 years ago
|
||
(In reply to Andreas Pehrson [:pehrsons] from comment #28)
(In reply to guest271314 from comment #25)
This bug was filed because https://www.w3.org/TR/mediacapture-fromelement/
A mute event is raised on the MediaStreamTrack when content availability changes.
does not occur at Firefox or Nightly under any cases that have that far tried.
Is that by design or a bug?
That spec language is for HTMLMediaElement capture, not HTMLCanvasElement capture. Thus, by design we are spec compliant for mute/unmute events for HTMLCanvasElement capture.
What chromium does is up to them. But in my opinion, making interoperable implementations requires a spec that defines the behavior to follow.
Closing this as WONTFIX as it needs to be resolved in the spec first.
Reporter | ||
Comment 30•4 years ago
|
||
(In reply to Sotaro Ikeda [:sotaro] from comment #19)
(In reply to guest271314 from comment #18)
Still does not display image when
transferControlToOffscreen()
is used in aWorker
.guest271314, is there a sample of doing it? With Attachment 9110463 [details], image was not rendered because canvas is not appended in document.
Can you kindly take a look at https://bugzilla.mozilla.org/show_bug.cgi?id=1598998, which describes the bug with OffscreenCanvas
, transferControlToOffscreen()
and transferFromImageBitmap()
in a Worker
?
Reporter | ||
Comment 31•4 years ago
|
||
(In reply to Andreas Pehrson [:pehrsons] from comment #28)
(In reply to guest271314 from comment #25)
This bug was filed because https://www.w3.org/TR/mediacapture-fromelement/
A mute event is raised on the MediaStreamTrack when content availability changes.
does not occur at Firefox or Nightly under any cases that have that far tried.
Is that by design or a bug?
That spec language is for HTMLMediaElement capture, not HTMLCanvasElement capture. Thus, by design we are spec compliant for mute/unmute events for HTMLCanvasElement capture.
What chromium does is up to them. But in my opinion, making interoperable implementations requires a spec that defines the behavior to follow.
Closing this as WONTFIX as it needs to be resolved in the spec first.
Re specification issue https://github.com/w3c/mediacapture-fromelement/issues/82 if
A captured stream MUST immediately cease to capture content if the origin-clean flag of the source canvas becomes false after the stream is created by captureStream(). The captured MediaStreamTrack MUST become muted, producing no new content while the canvas remains in this state.
is the only case where CanvasCaptureMediaStreamTrack
muted
property MUST be set to true
Firefox 70 and Night 73 fail.
Description
•