1169126 - CanvasCaptureMediaStream from 2D Canvas not visible in local video element on Mac

Status: RESOLVED FIXED

(Core :: Audio/Video, defect)

Description

[Andreas Pehrson [:pehrsons]]

On Nightly, with pref canvas.capturestream.enabled = true

STR:
1. Go to https://mozilla.github.io/webrtc-landing/canvas_swap.html
2. Click start

Expected:
The canvas, a local video element showing the canvas (small on the right), and a remote video element showing the canvas (large on the left) should be visible.

Actual:
On Mac 10.10, I don't see anything in the local video element. Time is progressing though.

Comment 1

[Andreas Pehrson [:pehrsons]]

I remember seeing something similar in my early rounds through developing the CanvasCaptureMediaStream. IIRC it was because the compositor couldn't handle a SourceSurface of type Cairo when it was in fact a Cairo-wrapped DataSourceSurface. Doing GetDataSurface() and handing it a SourceSurface with type Data worked then.

This might be the same issue. Might also be e10s-specific.

Comment 2

[Andreas Pehrson [:pehrsons]]

Nope, not specific to e10s.

Comment 4

[Andreas Pehrson [:pehrsons]]

Did some initial testing.

When getting a snapshot from the canvas on Mac we get a Skia surface back. If I do GetDataSurface() and pass that on instead, the local video element plays the frames just fine.

I'll dig deeper to see where the root cause is (a switch-case somewhere I think) or if this will do as a fix.

Comment 5

[Andreas Pehrson [:pehrsons]]

So what's happening is this:

The 2D canvas gives us a Skia surface (Canvas Backend = Skia). We put it in a CairoImage (the only Image that can hold SourceSurfaces). An ImageClient wants a TextureClient from this CairoImage, and goes about to copy it using a DrawTargetCG (Content Backend = CoreGraphics). Stacktrace thus far:

>   * frame #0: 0x00000001028db9ce XUL`mozilla::gfx::DrawTargetCG::CopySurface(this=0x000000010fb86c40, aSurface=0x000000011e4741a0, aSourceRect=0x0000000115403a28, aDestination=0x0000000115403a20) + 30 at DrawTargetCG.cpp:1612
>     frame #1: 0x0000000102981650 XUL`mozilla::layers::CairoImage::GetTextureClient(this=0x000000011137e920, aClient=<unavailable>) + 432 at ImageContainer.cpp:561
>     frame #2: 0x00000001029f57b0 XUL`mozilla::layers::ImageClientSingle::UpdateImage(this=0x000000010fb0d420, aContainer=0x000000011e4cba80, aContentFlags=<unavailable>) + 96 at ImageClient.cpp:160
>     frame #3: 0x0000000102a382e1 XUL`mozilla::layers::UpdateImageClientNow(aClient=0x000000010fb0d420, aContainer=0x000000011e4cba80) + 65 at ImageBridgeChild.cpp:385
>     frame #4: 0x0000000102179c70 XUL`MessageLoop::RunTask(this=0x0000000115403d00, task=0x0000000112b95820) + 144 at message_loop.cc:361

This is the `if` that ignores the copy, I take it because of the Canvas/Content backend mismatch: http://hg.mozilla.org/mozilla-central/file/98820360ab66/gfx/2d/DrawTargetCG.cpp#l1618

Comment 6

[Andreas Pehrson [:pehrsons]]

Attachment: MozReview Request: Bug 1169126 - Part 1. Make DrawTargetCG::CopySurface able to handle all SourceSurface types. r=gw280

Bug 1169126 - Always get a DataSourceSurface when capturing Canvas snapshot. r=gw280

Comment 7

[Andreas Pehrson [:pehrsons]]

George, is this what I have to do or are there any alternatives? CanvasClient manages things quite differently I believe - also doesn't have the requirement to provide data in software to its consumers.

Comment 8

[George Wright (:gw280) (needinfo me!)]

It seems to me like the correct solution here is to fix DrawTargetCG so that it can draw SourceSurfaceSkia surfaces

Comment 9

[Andreas Pehrson [:pehrsons]]

(In reply to George Wright (:gw280) from comment #8)
> It seems to me like the correct solution here is to fix DrawTargetCG so that
> it can draw SourceSurfaceSkia surfaces

Then how can we make sure that it works on all platforms? How can we make sure it doesn't regress, for instance when the canvas backend changes somewhere sometime?

The video element is reporting frames as rendered and drawing the video element into a canvas works (it's a canvas backend image after all), so I don't know a way of detecting this in a test.

Comment 10

[Matt Woodrow (:mattwoodrow)]

(In reply to Andreas Pehrson [:pehrsons] (Telenor) from comment #9)
> (In reply to George Wright (:gw280) from comment #8)
> > It seems to me like the correct solution here is to fix DrawTargetCG so that
> > it can draw SourceSurfaceSkia surfaces

Yes, this is the expected behaviour for a DrawTarget implementation.

> 
> Then how can we make sure that it works on all platforms? How can we make
> sure it doesn't regress, for instance when the canvas backend changes
> somewhere sometime?
> 
> The video element is reporting frames as rendered and drawing the video
> element into a canvas works (it's a canvas backend image after all), so I
> don't know a way of detecting this in a test.

Reftests grab a snapshot of the rendered window, so should be able to detect that the video element is blank.

Comment 11

[George Wright (:gw280) (needinfo me!)]

DrawTarget::OptimizeSourceSurface returns a SourceSurface object, and SourceSurface objects by design should be able to be drawn into any DrawTarget (in the same way as a DataSourceSurface object can be drawn into any DT). If a DrawTarget is failing to draw that SourceSurface for whatever reason, then there's a bug in Moz2D.

I think Matt's solution for a test should work.

Comment 12

[George Wright (:gw280) (needinfo me!)]

I think we can just modify DrawTargetCG::CopySurface() to check if we're not of type SurfaceType::COREGRAPHICS_IMAGE or SurfaceType::COREGRAPHICS_CGCONTEXT, and if we're not then call GetDataSurface() there.

Comment 13

[George Wright (:gw280) (needinfo me!)]

Or better yet, skip the if entirely because GetRetainedImageFromSourceSurface() will do the right thing anyway.

Comment 14

[Andreas Pehrson [:pehrsons]]

(In reply to George Wright (:gw280) from comment #11)
> DrawTarget::OptimizeSourceSurface returns a SourceSurface object, and
> SourceSurface objects by design should be able to be drawn into any
> DrawTarget (in the same way as a DataSourceSurface object can be drawn into
> any DT). If a DrawTarget is failing to draw that SourceSurface for whatever
> reason, then there's a bug in Moz2D.

Ah, I wasn't aware of this. Excellent!


> I think Matt's solution for a test should work.

Reftest, of course. Thanks guys!

Comment 15

[Andreas Pehrson [:pehrsons]]

Comment on attachment 8615784 [details]
MozReview Request: Bug 1169126 - Part 1. Make DrawTargetCG::CopySurface able to handle all SourceSurface types. r=gw280

Bug 1169126 - Part 1. Make DrawTargetCG::CopySurface able to handle all SourceSurface types. r=gw280

Comment 16

[Andreas Pehrson [:pehrsons]]

Attachment: MozReview Request: Bug 1169126 - Part 2. Add webgl reftest for captureStream(). r=jgilbert

Bug 1169126 - Part 2. Add webgl mochitest for captureStream(). r=jgilbert

Comment 17

[Andreas Pehrson [:pehrsons]]

Attachment: MozReview Request: Bug 1169126 - Part 3. Add canvas 2d reftest for captureStream(). r=gw280

Bug 1169126 - Part 3. Add canvas 2d mochitest for captureStream(). r=gw280

Comment 18

[Andreas Pehrson [:pehrsons]]

Fixed like suggested in comment 13. Are you OK to review DrawTargetCG George?

Also reftests working great on my Mac. Here's for other platforms: https://treeherder.mozilla.org/#/jobs?repo=try&revision=a246785a7661

Comment 19

[Andreas Pehrson [:pehrsons]]

Out of old habit I wrote mochitest in the commit messages. I'll fix that.

Comment 20

[Andreas Pehrson [:pehrsons]]

Comment on attachment 8620763 [details]
MozReview Request: Bug 1169126 - Part 2. Add webgl reftest for captureStream(). r=jgilbert

Bug 1169126 - Part 2. Add webgl reftest for captureStream(). r=jgilbert

Comment 21

[Andreas Pehrson [:pehrsons]]

Comment on attachment 8620765 [details]
MozReview Request: Bug 1169126 - Part 3. Add canvas 2d reftest for captureStream(). r=gw280

Bug 1169126 - Part 3. Add canvas 2d reftest for captureStream(). r=gw280

Comment 22

[Andreas Pehrson [:pehrsons]]

Looks like we're broken on B2G as well: http://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=http://ftp.mozilla.org/pub/mozilla.org/b2g/try-builds/[e-mail]-a246785a7661/try-emulator/try_ubuntu64_vm-b2g-emulator_test-reftest-8-bm115-tests1-linux64-build272.txt.gz&only_show_unexpected=1

Comment 23

[Andreas Pehrson [:pehrsons]]

There are errors in the log for that failed test on B2G:
> 22:28:59     INFO -  06-11 04:31:58.212 I/Gecko   (  714): REFTEST TEST-LOAD | http://10.0.2.2:8888/tests/dom/canvas/test/reftest/capturestream.html | 59 / 587 (10%)
> 22:28:59     INFO -  06-11 04:32:00.321 W/AudioFlinger(   40): Thread AudioOut_1 cannot connect to the power manager service
> 22:28:59     INFO -  06-11 04:32:00.612 D/        (  841): HostConnection::get() New Host Connection established 0x4566f460, tid 860
> 22:28:59     INFO -  06-11 04:32:00.650 E/EGL_emulation(  841): tid 860: eglMakeCurrent(947): error 0x3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.650 E/libEGL  (  841): eglMakeCurrent:674 error 3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.661 E/EGL_emulation(  841): tid 860: eglMakeCurrent(947): error 0x3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.661 E/libEGL  (  841): eglMakeCurrent:674 error 3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.661 E/EGL_emulation(  841): tid 860: eglMakeCurrent(947): error 0x3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.681 E/libEGL  (  841): eglMakeCurrent:674 error 3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.681 E/EGL_emulation(  841): tid 860: eglMakeCurrent(947): error 0x3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.681 E/libEGL  (  841): eglMakeCurrent:674 error 3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.681 E/EGL_emulation(  841): tid 860: eglMakeCurrent(947): error 0x3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.690 E/libEGL  (  841): eglMakeCurrent:674 error 3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.690 E/EGL_emulation(  841): tid 860: eglMakeCurrent(947): error 0x3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.690 E/libEGL  (  841): eglMakeCurrent:674 error 3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.700 E/EGL_emulation(  841): tid 860: eglMakeCurrent(947): error 0x3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.700 E/libEGL  (  841): eglMakeCurrent:674 error 3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.700 E/EGL_emulation(  841): tid 860: eglMakeCurrent(947): error 0x3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.710 E/libEGL  (  841): eglMakeCurrent:674 error 3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.710 E/EGL_emulation(  841): tid 860: eglMakeCurrent(947): error 0x3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.710 E/libEGL  (  841): eglMakeCurrent:674 error 3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.720 E/EGL_emulation(  841): tid 860: eglMakeCurrent(947): error 0x3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.720 E/libEGL  (  841): eglMakeCurrent:674 error 3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:01.221 W/Test Container(  841): [JavaScript Warning: "Use of Mutation Events is deprecated. Use MutationObserver instead." {file: "chrome://reftest/content/reftest-content.js" line: 593}]
> 22:28:59     INFO -  06-11 04:32:05.191 I/Gecko   (  714): RESTORE PREFERENCE pref(canvas.capturestream.enabled,false)

It appears that B2G uses SkiaGL for its canvas backend and Cairo for the content backend. From the log it looks like there's something failing on the GL side.

Unless anyone objects, I'll file a bug and disable the test on B2G for now.

Comment 24

[George Wright (:gw280) (needinfo me!)]

Comment on attachment 8615784 [details]
MozReview Request: Bug 1169126 - Part 1. Make DrawTargetCG::CopySurface able to handle all SourceSurface types. r=gw280

https://reviewboard.mozilla.org/r/10359/#review9527

Ship It!

Comment 25

[George Wright (:gw280) (needinfo me!)]

Comment on attachment 8620765 [details]
MozReview Request: Bug 1169126 - Part 3. Add canvas 2d reftest for captureStream(). r=gw280

https://reviewboard.mozilla.org/r/10823/#review9529

Ship It!

Comment 26

[Kelsey Gilbert [:jgilbert]]

Comment on attachment 8620763 [details]
MozReview Request: Bug 1169126 - Part 2. Add webgl reftest for captureStream(). r=jgilbert

https://reviewboard.mozilla.org/r/10821/#review9571

Ship It!

Comment 27

[Andreas Pehrson [:pehrsons]]

Comment on attachment 8620763 [details]
MozReview Request: Bug 1169126 - Part 2. Add webgl reftest for captureStream(). r=jgilbert

Bug 1169126 - Part 2. Add webgl reftest for captureStream(). r=jgilbert

Comment 28

[Andreas Pehrson [:pehrsons]]

Comment on attachment 8620765 [details]
MozReview Request: Bug 1169126 - Part 3. Add canvas 2d reftest for captureStream(). r=gw280

Bug 1169126 - Part 3. Add canvas 2d reftest for captureStream(). r=gw280

Comment 29

[Andreas Pehrson [:pehrsons]]

Comment on attachment 8620763 [details]
MozReview Request: Bug 1169126 - Part 2. Add webgl reftest for captureStream(). r=jgilbert

Disabled the test on Windows 7 and 8. Carries forward r=jgilbert.

Comment 30

[Andreas Pehrson [:pehrsons]]

Comment on attachment 8620765 [details]
MozReview Request: Bug 1169126 - Part 3. Add canvas 2d reftest for captureStream(). r=gw280

Disabled the test on B2G and Windows 7 and Windows 8. Carries forward r=gw280.

Comment 31

[Andreas Pehrson [:pehrsons]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=a246785a7661

and after the latest revision,
https://treeherder.mozilla.org/#/jobs?repo=try&revision=2ae5feeff77b

Comment 32

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/0e2935ed6dd2
https://hg.mozilla.org/integration/mozilla-inbound/rev/b792def14028
https://hg.mozilla.org/integration/mozilla-inbound/rev/d45f5cd7c687

Comment 33

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/0e2935ed6dd2
https://hg.mozilla.org/mozilla-central/rev/b792def14028
https://hg.mozilla.org/mozilla-central/rev/d45f5cd7c687

Comment 34

[muthusmkmtp]

 Firefox isn't capable to record "CanvasCaptureMediaStream" object. Why is it so? I could't use canvas.captureStream() which returns CanvasCaptureMediaStream object. I need to record the firefox browser. How could I do with MediaStream API, if this object is not supported in firefox?

Comment 35

[muthusmkmtp]

 Firefox isn't capable to record "CanvasCaptureMediaStream" object. Why is it so? I could't use canvas.captureStream() which returns CanvasCaptureMediaStream object. I need to record the firefox browser. How could I do with MediaStream API, if this object is not supported in firefox?

Comment 36

[Andreas Pehrson [:pehrsons]]

This is not a support channel. Please file a new bug if you have found one.