Closed Bug 1598237 Opened 4 years ago Closed 4 years ago

mute and unmute events of MediaStreamTrack from canvas.captureStream() do not fire under any circumstances

Categories

(Core :: WebRTC: Audio/Video, defect)

Product:
Core
Component:
WebRTC: Audio/Video
Version:
72 Branch
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED WONTFIX

People

(Reporter: guest271314, Unassigned)

References

Details

Attachments

(7 files)

canvasMediaStreamTrackMuteUnmuteEventsNeverFire.html
65.18 KB, text/html
Details
noMediaTransferredFromCanvasToMediaStreamTrackFirefox.html
25.25 KB, text/html
Details
about_support_nightly_72.txt
19.46 KB, text/plain
Details
about_support_firefox_70.txt
19.32 KB, text/plain
Details
offscreenCanvasInWorkerDoesNotDisplayImageOnCanvas.html
595 bytes, text/html
Details
offscreenCanvasInWorkerDoesNotDisplayImageOnCanvasWorker.js
24.86 KB, text/javascript
Details
firefoxDoesNotSetCanvasCaptureMediaStreamTrackMutedTrueUnderAnyCircumstances.html
65.52 KB, text/html
Details

User Agent: Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/80.0.3962.2 Chrome/80.0.3962.2 Safari/537.36

Steps to reproduce:

Try using various code which should trigger mute (and unmute) events of MediaStreamTrack from canvas.captureStream().

Actual results:

mute and unmute events of MediaStreamTrack from canvas.captureStream() do not fire under any circumstances.

Expected results:

At least mute event should be fired per

A mute[1] event is raised on the MediaStreamTrack when content availability changes.

Chromium 80 fires

  1. mute
  2. unmute
  3. mute

events using the same code.

[1] https://w3c.github.io/mediacapture-main/#event-mediastreamtrack-mute

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Component: Untriaged → Canvas: 2D
Product: Firefox → Core

To run Attachment 9110463 [details], pref gfx.offscreencanvas.enabled:true is necessary, since offscreen canvas is not enabled by default. By enabling the offscreencanvas, unmute event was raised once.

See Also: → offscreen-canvas

guest271314, did you enable offscreencanvas with pref gfx.offscreencanvas.enabled:true in about:config?

Flags: needinfo?(guest271314)

(In reply to Sotaro Ikeda [:sotaro] from comment #3)

guest271314, did you enable offscreencanvas with pref gfx.offscreencanvas.enabled:true in about:config?

Yes.

When ImageBitmapRenderingContext and transferFromImageBitmap() are used no media are transmitted from the <canvas> to the MediaStreamTrack (https://bugzilla.mozilla.org/show_bug.cgi?id=1590030, https://bugzilla.mozilla.org/show_bug.cgi?id=1557980)

At Firefox 70 the image is displayed at the <canvas>.

At Nightly 72 the image is not displayed at the <canvas>. To view the image it is necessary to either draw the <canvas> onto another <canvas> or right-click and select "View image" which navigates to a Blob URL where the image is displayed.

To reproduce run attached file 9110463: canvasMediaStreamTrackMuteUnmuteEventsNeverFire.html

Flags: needinfo?(guest271314)

Thanks for the detail! From comment 4, I noticed that WebRender does not handle ImageBitmap. Then created Bug 1599653.

See Also: → 1599653
No longer blocks: offscreen-canvas
See Also: → offscreen-canvas

From Attachment 9111627 [details], I also noticed a problem of video tag rendering of canvas.captureStream(). Then created Bug 1599696.

Depends on: 1599696, 1599653
See Also: 1599653 →

Once https://phabricator.services.mozilla.com/D54890 and https://phabricator.services.mozilla.com/D54893 are reviewed and merged the changes made will be available at Nightly, correct?

(In reply to guest271314 from comment #7)

Once https://phabricator.services.mozilla.com/D54890 and https://phabricator.services.mozilla.com/D54893 are reviewed and merged the changes made will be available at Nightly, correct?

Yes, correct. m-c is soft freeze this week, then I am going to try to land them next week.

Why does Firefox 70 display the image on canvas when ctx.transferFromImageBitmap(bitmap) is executed, where Nightly 72 does not?

(In reply to guest271314 from comment #9)

Why does Firefox 70 display the image on canvas when ctx.transferFromImageBitmap(bitmap) is executed, where Nightly 72 does not?

Can you attach about:support info to the bug? Nightly 72 might use WebRender as compositor. In this case, WebRender does not support canvas ImageBitmap rendering. It is Bug 1599653.

Flags: needinfo?(guest271314)
Flags: needinfo?(guest271314)

According to about:support at Nightly WebRender is

opt-in by default: WebRender is an opt-in feature
available by user: Qualified enabled by pref

How to set "Compositing" to "Basic" (Firefox default setting) at Nightly?

According to https://www.askvg.com/tip-how-to-enable-or-disable-new-webrender-feature-in-mozilla-firefox/ WebRender can be disabled by setting gfx.webrender.all to false, however that does not appear to have any affect where the same values are printed at about:support.

Can you try to set pref gfx.webrender.force-disabled:true ?

Flags: needinfo?(guest271314)

Same result.

Flags: needinfo?(guest271314)

(In reply to guest271314 from comment #16)

<s>Same result.</s>

Had to restart Nightly after setting gfx.webrender.force-disabled to true. The the image does display.

Still does not display image when transferControlToOffscreen() is used in a Worker.

(In reply to guest271314 from comment #18)

Still does not display image when transferControlToOffscreen() is used in a Worker.

guest271314, is there a sample of doing it? With Attachment 9110463 [details], image was not rendered because canvas is not appended in document.

Flags: needinfo?(guest271314)
Flags: needinfo?(guest271314)

(In reply to Sotaro Ikeda [:sotaro] from comment #19)

(In reply to guest271314 from comment #18)

Still does not display image when transferControlToOffscreen() is used in a Worker.

guest271314, is there a sample of doing it?

See attachments https://bugzilla.mozilla.org/attachment.cgi?id=9113391, https://bugzilla.mozilla.org/attachment.cgi?id=9113392, plnkr https://plnkr.co/edit/eXESuVqOp5J9AAAft07E?p=preview which demonstrate the transferControlToOffscreen(), "bitmaprenderer", transferFromimageBitmap() does not display the image onto the <canvas>. There are previously filed bugs re that issue https://bugzilla.mozilla.org/show_bug.cgi?id=1590030 which are not fixed as yet, and related to this bug, that is why mentioned those cases.

(As an aside if the OffscreenCanvas is transferred back to the main thread from a Worker none of the methods are executable, .toBlobb() will throw not a function)

With Attachment 9110463 [details], image was not rendered because canvas is not appended in document.

At Nightly 73 build https://hg.mozilla.org/mozilla-central/rev/3c08edf74d039af79f9daad8ff5b57ffb64fdab6 the <canvas> does not need to be appended to the document to be displayed at <video>. That attachment is to demostrate that mute, unmute, ended events of MediaStreamTrack do not fire, which is still the case at the above linked build. This bug https://bugzilla.mozilla.org/show_bug.cgi?id=1557980 is similar to the present issue. Have not yet tested MediaRecorder after the relevant fixes, though should record, as long as the canvas is not offscreen in a Worker.

MediaRecorder records the image when transferFromImageBitmap() is used now.

Re

There are previously filed bugs re that issue https://bugzilla.mozilla.org/show_bug.cgi?id=1590030 which are not fixed as yet

Correction: That bug appears to be fixed at Nightly 73 with the changes made to canvas.

This bug was filed because https://www.w3.org/TR/mediacapture-fromelement/

A mute event is raised on the MediaStreamTrack when content availability changes.

does not occur at Firefox or Nightly under any cases that have that far tried.

Is that by design or a bug?

If we adjust the code at https://bugzilla.mozilla.org/attachment.cgi?id=9113391 and https://bugzilla.mozilla.org/attachment.cgi?id=9113392 to use transferToImageBitmap() we can observe the bitmap is transferred to the OffscreenCanvas within the Worker, though to display the image requires creating a new <canvas>. This should be possible using the single original <canvas>.

const bmp = offscreen.transferToImageBitmap();
postMessage({bmp}, [bmp]);

worker.onmessage = e => {
  const c = document.createElement('canvas')
  const x = c.getContext('bitmaprenderer')
  document.body.appendChild(c)
  c.width = e.data.bmp.width;
  c.height = e.data.bmp.height;
  x.transferFromImageBitmap(e.data.bmp)
}

(In reply to guest271314 from comment #25)

Is that by design or a bug?

HTMLMediaElement seems to handle mute event correctly by using HTMLMediaElement::MediaElementTrackSource. But CanvasCaptureMediaStream seems not handle mute event. I wonder if this is a bug.

:pehrsons, can you comment to Comment 25?

Flags: needinfo?(apehrson)

(In reply to guest271314 from comment #25)

This bug was filed because https://www.w3.org/TR/mediacapture-fromelement/

A mute event is raised on the MediaStreamTrack when content availability changes.

does not occur at Firefox or Nightly under any cases that have that far tried.

Is that by design or a bug?

That spec language is for HTMLMediaElement capture, not HTMLCanvasElement capture. Thus, by design we are spec compliant for mute/unmute events for HTMLCanvasElement capture.

What chromium does is up to them. But in my opinion, making interoperable implementations requires a spec that defines the behavior to follow.

Closing this as WONTFIX as it needs to be resolved in the spec first.

Status: UNCONFIRMED → RESOLVED
Closed: 4 years ago
Component: Canvas: 2D → WebRTC: Audio/Video
Flags: needinfo?(apehrson)
Resolution: --- → WONTFIX

(In reply to Andreas Pehrson [:pehrsons] from comment #28)

(In reply to guest271314 from comment #25)

This bug was filed because https://www.w3.org/TR/mediacapture-fromelement/

A mute event is raised on the MediaStreamTrack when content availability changes.

does not occur at Firefox or Nightly under any cases that have that far tried.

Is that by design or a bug?

That spec language is for HTMLMediaElement capture, not HTMLCanvasElement capture. Thus, by design we are spec compliant for mute/unmute events for HTMLCanvasElement capture.

What chromium does is up to them. But in my opinion, making interoperable implementations requires a spec that defines the behavior to follow.

Closing this as WONTFIX as it needs to be resolved in the spec first.

https://github.com/w3c/mediacapture-fromelement/issues/82

(In reply to Sotaro Ikeda [:sotaro] from comment #19)

(In reply to guest271314 from comment #18)

Still does not display image when transferControlToOffscreen() is used in a Worker.

guest271314, is there a sample of doing it? With Attachment 9110463 [details], image was not rendered because canvas is not appended in document.

Can you kindly take a look at https://bugzilla.mozilla.org/show_bug.cgi?id=1598998, which describes the bug with OffscreenCanvas, transferControlToOffscreen() and transferFromImageBitmap() in a Worker?

(In reply to Andreas Pehrson [:pehrsons] from comment #28)

(In reply to guest271314 from comment #25)

This bug was filed because https://www.w3.org/TR/mediacapture-fromelement/

A mute event is raised on the MediaStreamTrack when content availability changes.

does not occur at Firefox or Nightly under any cases that have that far tried.

Is that by design or a bug?

That spec language is for HTMLMediaElement capture, not HTMLCanvasElement capture. Thus, by design we are spec compliant for mute/unmute events for HTMLCanvasElement capture.

What chromium does is up to them. But in my opinion, making interoperable implementations requires a spec that defines the behavior to follow.

Closing this as WONTFIX as it needs to be resolved in the spec first.

Re specification issue https://github.com/w3c/mediacapture-fromelement/issues/82 if

A captured stream MUST immediately cease to capture content if the origin-clean flag of the source canvas becomes false after the stream is created by captureStream(). The captured MediaStreamTrack MUST become muted, producing no new content while the canvas remains in this state.

is the only case where CanvasCaptureMediaStreamTrack muted property MUST be set to true Firefox 70 and Night 73 fail.

Flags: needinfo?(apehrson)

Bug 1231131 should fix that.

Flags: needinfo?(apehrson)
You need to log in before you can comment on or make changes to this bug.