CanvasCaptureMediaStream from 2D Canvas not visible in local video element on Mac

RESOLVED FIXED in Firefox 41

Status

()

defect
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: pehrsons, Assigned: pehrsons)

Tracking

41 Branch
mozilla41
All
macOS
Points:
---
Dependency tree / graph
Bug Flags:
in-testsuite +

Firefox Tracking Flags

(firefox41 fixed)

Details

Attachments

(3 attachments)

On Nightly, with pref canvas.capturestream.enabled = true

STR:
1. Go to https://mozilla.github.io/webrtc-landing/canvas_swap.html
2. Click start

Expected:
The canvas, a local video element showing the canvas (small on the right), and a remote video element showing the canvas (large on the left) should be visible.

Actual:
On Mac 10.10, I don't see anything in the local video element. Time is progressing though.
I remember seeing something similar in my early rounds through developing the CanvasCaptureMediaStream. IIRC it was because the compositor couldn't handle a SourceSurface of type Cairo when it was in fact a Cairo-wrapped DataSourceSurface. Doing GetDataSurface() and handing it a SourceSurface with type Data worked then.

This might be the same issue. Might also be e10s-specific.
Nope, not specific to e10s.
Duplicate of this bug: 1170593
Assignee: nobody → pehrsons
Status: NEW → ASSIGNED
Did some initial testing.

When getting a snapshot from the canvas on Mac we get a Skia surface back. If I do GetDataSurface() and pass that on instead, the local video element plays the frames just fine.

I'll dig deeper to see where the root cause is (a switch-case somewhere I think) or if this will do as a fix.
So what's happening is this:

The 2D canvas gives us a Skia surface (Canvas Backend = Skia). We put it in a CairoImage (the only Image that can hold SourceSurfaces). An ImageClient wants a TextureClient from this CairoImage, and goes about to copy it using a DrawTargetCG (Content Backend = CoreGraphics). Stacktrace thus far:

>   * frame #0: 0x00000001028db9ce XUL`mozilla::gfx::DrawTargetCG::CopySurface(this=0x000000010fb86c40, aSurface=0x000000011e4741a0, aSourceRect=0x0000000115403a28, aDestination=0x0000000115403a20) + 30 at DrawTargetCG.cpp:1612
>     frame #1: 0x0000000102981650 XUL`mozilla::layers::CairoImage::GetTextureClient(this=0x000000011137e920, aClient=<unavailable>) + 432 at ImageContainer.cpp:561
>     frame #2: 0x00000001029f57b0 XUL`mozilla::layers::ImageClientSingle::UpdateImage(this=0x000000010fb0d420, aContainer=0x000000011e4cba80, aContentFlags=<unavailable>) + 96 at ImageClient.cpp:160
>     frame #3: 0x0000000102a382e1 XUL`mozilla::layers::UpdateImageClientNow(aClient=0x000000010fb0d420, aContainer=0x000000011e4cba80) + 65 at ImageBridgeChild.cpp:385
>     frame #4: 0x0000000102179c70 XUL`MessageLoop::RunTask(this=0x0000000115403d00, task=0x0000000112b95820) + 144 at message_loop.cc:361

This is the `if` that ignores the copy, I take it because of the Canvas/Content backend mismatch: http://hg.mozilla.org/mozilla-central/file/98820360ab66/gfx/2d/DrawTargetCG.cpp#l1618
Bug 1169126 - Always get a DataSourceSurface when capturing Canvas snapshot. r=gw280
Attachment #8615784 - Flags: review?(gwright)
George, is this what I have to do or are there any alternatives? CanvasClient manages things quite differently I believe - also doesn't have the requirement to provide data in software to its consumers.
It seems to me like the correct solution here is to fix DrawTargetCG so that it can draw SourceSurfaceSkia surfaces
(In reply to George Wright (:gw280) from comment #8)
> It seems to me like the correct solution here is to fix DrawTargetCG so that
> it can draw SourceSurfaceSkia surfaces

Then how can we make sure that it works on all platforms? How can we make sure it doesn't regress, for instance when the canvas backend changes somewhere sometime?

The video element is reporting frames as rendered and drawing the video element into a canvas works (it's a canvas backend image after all), so I don't know a way of detecting this in a test.
Flags: needinfo?(gwright)
(In reply to Andreas Pehrson [:pehrsons] (Telenor) from comment #9)
> (In reply to George Wright (:gw280) from comment #8)
> > It seems to me like the correct solution here is to fix DrawTargetCG so that
> > it can draw SourceSurfaceSkia surfaces

Yes, this is the expected behaviour for a DrawTarget implementation.

> 
> Then how can we make sure that it works on all platforms? How can we make
> sure it doesn't regress, for instance when the canvas backend changes
> somewhere sometime?
> 
> The video element is reporting frames as rendered and drawing the video
> element into a canvas works (it's a canvas backend image after all), so I
> don't know a way of detecting this in a test.

Reftests grab a snapshot of the rendered window, so should be able to detect that the video element is blank.
DrawTarget::OptimizeSourceSurface returns a SourceSurface object, and SourceSurface objects by design should be able to be drawn into any DrawTarget (in the same way as a DataSourceSurface object can be drawn into any DT). If a DrawTarget is failing to draw that SourceSurface for whatever reason, then there's a bug in Moz2D.

I think Matt's solution for a test should work.
I think we can just modify DrawTargetCG::CopySurface() to check if we're not of type SurfaceType::COREGRAPHICS_IMAGE or SurfaceType::COREGRAPHICS_CGCONTEXT, and if we're not then call GetDataSurface() there.
Flags: needinfo?(gwright)
Or better yet, skip the if entirely because GetRetainedImageFromSourceSurface() will do the right thing anyway.
(In reply to George Wright (:gw280) from comment #11)
> DrawTarget::OptimizeSourceSurface returns a SourceSurface object, and
> SourceSurface objects by design should be able to be drawn into any
> DrawTarget (in the same way as a DataSourceSurface object can be drawn into
> any DT). If a DrawTarget is failing to draw that SourceSurface for whatever
> reason, then there's a bug in Moz2D.

Ah, I wasn't aware of this. Excellent!


> I think Matt's solution for a test should work.

Reftest, of course. Thanks guys!
Comment on attachment 8615784 [details]
MozReview Request: Bug 1169126 - Part 1. Make DrawTargetCG::CopySurface able to handle all SourceSurface types. r=gw280

Bug 1169126 - Part 1. Make DrawTargetCG::CopySurface able to handle all SourceSurface types. r=gw280
Attachment #8615784 - Attachment description: MozReview Request: Bug 1169126 - Always get a DataSourceSurface when capturing Canvas snapshot. r=gw280 → MozReview Request: Bug 1169126 - Part 1. Make DrawTargetCG::CopySurface able to handle all SourceSurface types. r=gw280
Bug 1169126 - Part 2. Add webgl mochitest for captureStream(). r=jgilbert
Attachment #8620763 - Flags: review?(jgilbert)
Bug 1169126 - Part 3. Add canvas 2d mochitest for captureStream(). r=gw280
Attachment #8620765 - Flags: review?(gwright)
Fixed like suggested in comment 13. Are you OK to review DrawTargetCG George?

Also reftests working great on my Mac. Here's for other platforms: https://treeherder.mozilla.org/#/jobs?repo=try&revision=a246785a7661
Out of old habit I wrote mochitest in the commit messages. I'll fix that.
Comment on attachment 8620763 [details]
MozReview Request: Bug 1169126 - Part 2. Add webgl reftest for captureStream(). r=jgilbert

Bug 1169126 - Part 2. Add webgl reftest for captureStream(). r=jgilbert
Attachment #8620763 - Attachment description: MozReview Request: Bug 1169126 - Part 2. Add webgl mochitest for captureStream(). r=jgilbert → MozReview Request: Bug 1169126 - Part 2. Add webgl reftest for captureStream(). r=jgilbert
Attachment #8620765 - Attachment description: MozReview Request: Bug 1169126 - Part 3. Add canvas 2d mochitest for captureStream(). r=gw280 → MozReview Request: Bug 1169126 - Part 3. Add canvas 2d reftest for captureStream(). r=gw280
Comment on attachment 8620765 [details]
MozReview Request: Bug 1169126 - Part 3. Add canvas 2d reftest for captureStream(). r=gw280

Bug 1169126 - Part 3. Add canvas 2d reftest for captureStream(). r=gw280
There are errors in the log for that failed test on B2G:
> 22:28:59     INFO -  06-11 04:31:58.212 I/Gecko   (  714): REFTEST TEST-LOAD | http://10.0.2.2:8888/tests/dom/canvas/test/reftest/capturestream.html | 59 / 587 (10%)
> 22:28:59     INFO -  06-11 04:32:00.321 W/AudioFlinger(   40): Thread AudioOut_1 cannot connect to the power manager service
> 22:28:59     INFO -  06-11 04:32:00.612 D/        (  841): HostConnection::get() New Host Connection established 0x4566f460, tid 860
> 22:28:59     INFO -  06-11 04:32:00.650 E/EGL_emulation(  841): tid 860: eglMakeCurrent(947): error 0x3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.650 E/libEGL  (  841): eglMakeCurrent:674 error 3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.661 E/EGL_emulation(  841): tid 860: eglMakeCurrent(947): error 0x3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.661 E/libEGL  (  841): eglMakeCurrent:674 error 3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.661 E/EGL_emulation(  841): tid 860: eglMakeCurrent(947): error 0x3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.681 E/libEGL  (  841): eglMakeCurrent:674 error 3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.681 E/EGL_emulation(  841): tid 860: eglMakeCurrent(947): error 0x3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.681 E/libEGL  (  841): eglMakeCurrent:674 error 3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.681 E/EGL_emulation(  841): tid 860: eglMakeCurrent(947): error 0x3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.690 E/libEGL  (  841): eglMakeCurrent:674 error 3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.690 E/EGL_emulation(  841): tid 860: eglMakeCurrent(947): error 0x3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.690 E/libEGL  (  841): eglMakeCurrent:674 error 3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.700 E/EGL_emulation(  841): tid 860: eglMakeCurrent(947): error 0x3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.700 E/libEGL  (  841): eglMakeCurrent:674 error 3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.700 E/EGL_emulation(  841): tid 860: eglMakeCurrent(947): error 0x3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.710 E/libEGL  (  841): eglMakeCurrent:674 error 3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.710 E/EGL_emulation(  841): tid 860: eglMakeCurrent(947): error 0x3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.710 E/libEGL  (  841): eglMakeCurrent:674 error 3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.720 E/EGL_emulation(  841): tid 860: eglMakeCurrent(947): error 0x3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.720 E/libEGL  (  841): eglMakeCurrent:674 error 3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:01.221 W/Test Container(  841): [JavaScript Warning: "Use of Mutation Events is deprecated. Use MutationObserver instead." {file: "chrome://reftest/content/reftest-content.js" line: 593}]
> 22:28:59     INFO -  06-11 04:32:05.191 I/Gecko   (  714): RESTORE PREFERENCE pref(canvas.capturestream.enabled,false)

It appears that B2G uses SkiaGL for its canvas backend and Cairo for the content backend. From the log it looks like there's something failing on the GL side.

Unless anyone objects, I'll file a bug and disable the test on B2G for now.
Comment on attachment 8615784 [details]
MozReview Request: Bug 1169126 - Part 1. Make DrawTargetCG::CopySurface able to handle all SourceSurface types. r=gw280

https://reviewboard.mozilla.org/r/10359/#review9527

Ship It!
Attachment #8615784 - Flags: review?(gwright) → review+
Comment on attachment 8620765 [details]
MozReview Request: Bug 1169126 - Part 3. Add canvas 2d reftest for captureStream(). r=gw280

https://reviewboard.mozilla.org/r/10823/#review9529

Ship It!
Attachment #8620765 - Flags: review?(gwright) → review+
Comment on attachment 8620763 [details]
MozReview Request: Bug 1169126 - Part 2. Add webgl reftest for captureStream(). r=jgilbert

https://reviewboard.mozilla.org/r/10821/#review9571

Ship It!
Attachment #8620763 - Flags: review?(jgilbert) → review+
Comment on attachment 8620763 [details]
MozReview Request: Bug 1169126 - Part 2. Add webgl reftest for captureStream(). r=jgilbert

Bug 1169126 - Part 2. Add webgl reftest for captureStream(). r=jgilbert
Attachment #8620763 - Flags: review+
Comment on attachment 8620765 [details]
MozReview Request: Bug 1169126 - Part 3. Add canvas 2d reftest for captureStream(). r=gw280

Bug 1169126 - Part 3. Add canvas 2d reftest for captureStream(). r=gw280
Attachment #8620765 - Flags: review+
Comment on attachment 8620763 [details]
MozReview Request: Bug 1169126 - Part 2. Add webgl reftest for captureStream(). r=jgilbert

Disabled the test on Windows 7 and 8. Carries forward r=jgilbert.
Attachment #8620763 - Flags: review+
Comment on attachment 8620765 [details]
MozReview Request: Bug 1169126 - Part 3. Add canvas 2d reftest for captureStream(). r=gw280

Disabled the test on B2G and Windows 7 and Windows 8. Carries forward r=gw280.
Attachment #8620765 - Flags: review+
https://hg.mozilla.org/mozilla-central/rev/0e2935ed6dd2
https://hg.mozilla.org/mozilla-central/rev/b792def14028
https://hg.mozilla.org/mozilla-central/rev/d45f5cd7c687
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Flags: in-testsuite+
Resolution: --- → FIXED
Target Milestone: --- → mozilla41
Depends on: 1175257
Depends on: 1175590
 Firefox isn't capable to record "CanvasCaptureMediaStream" object. Why is it so? I could't use canvas.captureStream() which returns CanvasCaptureMediaStream object. I need to record the firefox browser. How could I do with MediaStream API, if this object is not supported in firefox?
 Firefox isn't capable to record "CanvasCaptureMediaStream" object. Why is it so? I could't use canvas.captureStream() which returns CanvasCaptureMediaStream object. I need to record the firefox browser. How could I do with MediaStream API, if this object is not supported in firefox?
This is not a support channel. Please file a new bug if you have found one.
You need to log in before you can comment on or make changes to this bug.