CanvasCaptureMediaStream from 2D Canvas not visible in local video element on Mac

RESOLVED FIXED in Firefox 41

Status

()

RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: pehrsons, Assigned: pehrsons)

Tracking

(Depends on: 1 bug)

41 Branch
mozilla41
All
Mac OS X
Points:
---
Dependency tree / graph
Bug Flags:
in-testsuite +

Firefox Tracking Flags

(firefox41 fixed)

Details

MozReview Requests

()

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(3 attachments)

(Assignee)

Description

3 years ago
On Nightly, with pref canvas.capturestream.enabled = true

STR:
1. Go to https://mozilla.github.io/webrtc-landing/canvas_swap.html
2. Click start

Expected:
The canvas, a local video element showing the canvas (small on the right), and a remote video element showing the canvas (large on the left) should be visible.

Actual:
On Mac 10.10, I don't see anything in the local video element. Time is progressing though.
(Assignee)

Comment 1

3 years ago
I remember seeing something similar in my early rounds through developing the CanvasCaptureMediaStream. IIRC it was because the compositor couldn't handle a SourceSurface of type Cairo when it was in fact a Cairo-wrapped DataSourceSurface. Doing GetDataSurface() and handing it a SourceSurface with type Data worked then.

This might be the same issue. Might also be e10s-specific.
(Assignee)

Comment 2

3 years ago
Nope, not specific to e10s.
(Assignee)

Updated

3 years ago
Duplicate of this bug: 1170593
(Assignee)

Updated

3 years ago
Assignee: nobody → pehrsons
Status: NEW → ASSIGNED
(Assignee)

Comment 4

3 years ago
Did some initial testing.

When getting a snapshot from the canvas on Mac we get a Skia surface back. If I do GetDataSurface() and pass that on instead, the local video element plays the frames just fine.

I'll dig deeper to see where the root cause is (a switch-case somewhere I think) or if this will do as a fix.
(Assignee)

Comment 5

3 years ago
So what's happening is this:

The 2D canvas gives us a Skia surface (Canvas Backend = Skia). We put it in a CairoImage (the only Image that can hold SourceSurfaces). An ImageClient wants a TextureClient from this CairoImage, and goes about to copy it using a DrawTargetCG (Content Backend = CoreGraphics). Stacktrace thus far:

>   * frame #0: 0x00000001028db9ce XUL`mozilla::gfx::DrawTargetCG::CopySurface(this=0x000000010fb86c40, aSurface=0x000000011e4741a0, aSourceRect=0x0000000115403a28, aDestination=0x0000000115403a20) + 30 at DrawTargetCG.cpp:1612
>     frame #1: 0x0000000102981650 XUL`mozilla::layers::CairoImage::GetTextureClient(this=0x000000011137e920, aClient=<unavailable>) + 432 at ImageContainer.cpp:561
>     frame #2: 0x00000001029f57b0 XUL`mozilla::layers::ImageClientSingle::UpdateImage(this=0x000000010fb0d420, aContainer=0x000000011e4cba80, aContentFlags=<unavailable>) + 96 at ImageClient.cpp:160
>     frame #3: 0x0000000102a382e1 XUL`mozilla::layers::UpdateImageClientNow(aClient=0x000000010fb0d420, aContainer=0x000000011e4cba80) + 65 at ImageBridgeChild.cpp:385
>     frame #4: 0x0000000102179c70 XUL`MessageLoop::RunTask(this=0x0000000115403d00, task=0x0000000112b95820) + 144 at message_loop.cc:361

This is the `if` that ignores the copy, I take it because of the Canvas/Content backend mismatch: http://hg.mozilla.org/mozilla-central/file/98820360ab66/gfx/2d/DrawTargetCG.cpp#l1618
(Assignee)

Comment 6

3 years ago
Created attachment 8615784 [details]
MozReview Request: Bug 1169126 - Part 1. Make DrawTargetCG::CopySurface able to handle all SourceSurface types. r=gw280

Bug 1169126 - Always get a DataSourceSurface when capturing Canvas snapshot. r=gw280
Attachment #8615784 - Flags: review?(gwright)
(Assignee)

Comment 7

3 years ago
George, is this what I have to do or are there any alternatives? CanvasClient manages things quite differently I believe - also doesn't have the requirement to provide data in software to its consumers.
It seems to me like the correct solution here is to fix DrawTargetCG so that it can draw SourceSurfaceSkia surfaces
(Assignee)

Comment 9

3 years ago
(In reply to George Wright (:gw280) from comment #8)
> It seems to me like the correct solution here is to fix DrawTargetCG so that
> it can draw SourceSurfaceSkia surfaces

Then how can we make sure that it works on all platforms? How can we make sure it doesn't regress, for instance when the canvas backend changes somewhere sometime?

The video element is reporting frames as rendered and drawing the video element into a canvas works (it's a canvas backend image after all), so I don't know a way of detecting this in a test.
Flags: needinfo?(gwright)
(In reply to Andreas Pehrson [:pehrsons] (Telenor) from comment #9)
> (In reply to George Wright (:gw280) from comment #8)
> > It seems to me like the correct solution here is to fix DrawTargetCG so that
> > it can draw SourceSurfaceSkia surfaces

Yes, this is the expected behaviour for a DrawTarget implementation.

> 
> Then how can we make sure that it works on all platforms? How can we make
> sure it doesn't regress, for instance when the canvas backend changes
> somewhere sometime?
> 
> The video element is reporting frames as rendered and drawing the video
> element into a canvas works (it's a canvas backend image after all), so I
> don't know a way of detecting this in a test.

Reftests grab a snapshot of the rendered window, so should be able to detect that the video element is blank.
DrawTarget::OptimizeSourceSurface returns a SourceSurface object, and SourceSurface objects by design should be able to be drawn into any DrawTarget (in the same way as a DataSourceSurface object can be drawn into any DT). If a DrawTarget is failing to draw that SourceSurface for whatever reason, then there's a bug in Moz2D.

I think Matt's solution for a test should work.
I think we can just modify DrawTargetCG::CopySurface() to check if we're not of type SurfaceType::COREGRAPHICS_IMAGE or SurfaceType::COREGRAPHICS_CGCONTEXT, and if we're not then call GetDataSurface() there.
Flags: needinfo?(gwright)
Or better yet, skip the if entirely because GetRetainedImageFromSourceSurface() will do the right thing anyway.
(Assignee)

Comment 14

3 years ago
(In reply to George Wright (:gw280) from comment #11)
> DrawTarget::OptimizeSourceSurface returns a SourceSurface object, and
> SourceSurface objects by design should be able to be drawn into any
> DrawTarget (in the same way as a DataSourceSurface object can be drawn into
> any DT). If a DrawTarget is failing to draw that SourceSurface for whatever
> reason, then there's a bug in Moz2D.

Ah, I wasn't aware of this. Excellent!


> I think Matt's solution for a test should work.

Reftest, of course. Thanks guys!
(Assignee)

Comment 15

3 years ago
Comment on attachment 8615784 [details]
MozReview Request: Bug 1169126 - Part 1. Make DrawTargetCG::CopySurface able to handle all SourceSurface types. r=gw280

Bug 1169126 - Part 1. Make DrawTargetCG::CopySurface able to handle all SourceSurface types. r=gw280
Attachment #8615784 - Attachment description: MozReview Request: Bug 1169126 - Always get a DataSourceSurface when capturing Canvas snapshot. r=gw280 → MozReview Request: Bug 1169126 - Part 1. Make DrawTargetCG::CopySurface able to handle all SourceSurface types. r=gw280
(Assignee)

Comment 16

3 years ago
Created attachment 8620763 [details]
MozReview Request: Bug 1169126 - Part 2. Add webgl reftest for captureStream(). r=jgilbert

Bug 1169126 - Part 2. Add webgl mochitest for captureStream(). r=jgilbert
Attachment #8620763 - Flags: review?(jgilbert)
(Assignee)

Comment 17

3 years ago
Created attachment 8620765 [details]
MozReview Request: Bug 1169126 - Part 3. Add canvas 2d reftest for captureStream(). r=gw280

Bug 1169126 - Part 3. Add canvas 2d mochitest for captureStream(). r=gw280
Attachment #8620765 - Flags: review?(gwright)
(Assignee)

Comment 18

3 years ago
Fixed like suggested in comment 13. Are you OK to review DrawTargetCG George?

Also reftests working great on my Mac. Here's for other platforms: https://treeherder.mozilla.org/#/jobs?repo=try&revision=a246785a7661
(Assignee)

Comment 19

3 years ago
Out of old habit I wrote mochitest in the commit messages. I'll fix that.
(Assignee)

Comment 20

3 years ago
Comment on attachment 8620763 [details]
MozReview Request: Bug 1169126 - Part 2. Add webgl reftest for captureStream(). r=jgilbert

Bug 1169126 - Part 2. Add webgl reftest for captureStream(). r=jgilbert
Attachment #8620763 - Attachment description: MozReview Request: Bug 1169126 - Part 2. Add webgl mochitest for captureStream(). r=jgilbert → MozReview Request: Bug 1169126 - Part 2. Add webgl reftest for captureStream(). r=jgilbert
(Assignee)

Updated

3 years ago
Attachment #8620765 - Attachment description: MozReview Request: Bug 1169126 - Part 3. Add canvas 2d mochitest for captureStream(). r=gw280 → MozReview Request: Bug 1169126 - Part 3. Add canvas 2d reftest for captureStream(). r=gw280
(Assignee)

Comment 21

3 years ago
Comment on attachment 8620765 [details]
MozReview Request: Bug 1169126 - Part 3. Add canvas 2d reftest for captureStream(). r=gw280

Bug 1169126 - Part 3. Add canvas 2d reftest for captureStream(). r=gw280
(Assignee)

Comment 23

3 years ago
There are errors in the log for that failed test on B2G:
> 22:28:59     INFO -  06-11 04:31:58.212 I/Gecko   (  714): REFTEST TEST-LOAD | http://10.0.2.2:8888/tests/dom/canvas/test/reftest/capturestream.html | 59 / 587 (10%)
> 22:28:59     INFO -  06-11 04:32:00.321 W/AudioFlinger(   40): Thread AudioOut_1 cannot connect to the power manager service
> 22:28:59     INFO -  06-11 04:32:00.612 D/        (  841): HostConnection::get() New Host Connection established 0x4566f460, tid 860
> 22:28:59     INFO -  06-11 04:32:00.650 E/EGL_emulation(  841): tid 860: eglMakeCurrent(947): error 0x3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.650 E/libEGL  (  841): eglMakeCurrent:674 error 3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.661 E/EGL_emulation(  841): tid 860: eglMakeCurrent(947): error 0x3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.661 E/libEGL  (  841): eglMakeCurrent:674 error 3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.661 E/EGL_emulation(  841): tid 860: eglMakeCurrent(947): error 0x3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.681 E/libEGL  (  841): eglMakeCurrent:674 error 3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.681 E/EGL_emulation(  841): tid 860: eglMakeCurrent(947): error 0x3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.681 E/libEGL  (  841): eglMakeCurrent:674 error 3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.681 E/EGL_emulation(  841): tid 860: eglMakeCurrent(947): error 0x3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.690 E/libEGL  (  841): eglMakeCurrent:674 error 3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.690 E/EGL_emulation(  841): tid 860: eglMakeCurrent(947): error 0x3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.690 E/libEGL  (  841): eglMakeCurrent:674 error 3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.700 E/EGL_emulation(  841): tid 860: eglMakeCurrent(947): error 0x3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.700 E/libEGL  (  841): eglMakeCurrent:674 error 3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.700 E/EGL_emulation(  841): tid 860: eglMakeCurrent(947): error 0x3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.710 E/libEGL  (  841): eglMakeCurrent:674 error 3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.710 E/EGL_emulation(  841): tid 860: eglMakeCurrent(947): error 0x3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.710 E/libEGL  (  841): eglMakeCurrent:674 error 3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.720 E/EGL_emulation(  841): tid 860: eglMakeCurrent(947): error 0x3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:00.720 E/libEGL  (  841): eglMakeCurrent:674 error 3002 (EGL_BAD_ACCESS)
> 22:28:59     INFO -  06-11 04:32:01.221 W/Test Container(  841): [JavaScript Warning: "Use of Mutation Events is deprecated. Use MutationObserver instead." {file: "chrome://reftest/content/reftest-content.js" line: 593}]
> 22:28:59     INFO -  06-11 04:32:05.191 I/Gecko   (  714): RESTORE PREFERENCE pref(canvas.capturestream.enabled,false)

It appears that B2G uses SkiaGL for its canvas backend and Cairo for the content backend. From the log it looks like there's something failing on the GL side.

Unless anyone objects, I'll file a bug and disable the test on B2G for now.
(Assignee)

Updated

3 years ago
See Also: → bug 1173686
Comment on attachment 8615784 [details]
MozReview Request: Bug 1169126 - Part 1. Make DrawTargetCG::CopySurface able to handle all SourceSurface types. r=gw280

https://reviewboard.mozilla.org/r/10359/#review9527

Ship It!
Attachment #8615784 - Flags: review?(gwright) → review+
Comment on attachment 8620765 [details]
MozReview Request: Bug 1169126 - Part 3. Add canvas 2d reftest for captureStream(). r=gw280

https://reviewboard.mozilla.org/r/10823/#review9529

Ship It!
Attachment #8620765 - Flags: review?(gwright) → review+
Comment on attachment 8620763 [details]
MozReview Request: Bug 1169126 - Part 2. Add webgl reftest for captureStream(). r=jgilbert

https://reviewboard.mozilla.org/r/10821/#review9571

Ship It!
Attachment #8620763 - Flags: review?(jgilbert) → review+
(Assignee)

Updated

3 years ago
See Also: → bug 1174056
(Assignee)

Comment 27

3 years ago
Comment on attachment 8620763 [details]
MozReview Request: Bug 1169126 - Part 2. Add webgl reftest for captureStream(). r=jgilbert

Bug 1169126 - Part 2. Add webgl reftest for captureStream(). r=jgilbert
Attachment #8620763 - Flags: review+
(Assignee)

Comment 28

3 years ago
Comment on attachment 8620765 [details]
MozReview Request: Bug 1169126 - Part 3. Add canvas 2d reftest for captureStream(). r=gw280

Bug 1169126 - Part 3. Add canvas 2d reftest for captureStream(). r=gw280
Attachment #8620765 - Flags: review+
(Assignee)

Comment 29

3 years ago
Comment on attachment 8620763 [details]
MozReview Request: Bug 1169126 - Part 2. Add webgl reftest for captureStream(). r=jgilbert

Disabled the test on Windows 7 and 8. Carries forward r=jgilbert.
Attachment #8620763 - Flags: review+
(Assignee)

Comment 30

3 years ago
Comment on attachment 8620765 [details]
MozReview Request: Bug 1169126 - Part 3. Add canvas 2d reftest for captureStream(). r=gw280

Disabled the test on B2G and Windows 7 and Windows 8. Carries forward r=gw280.
Attachment #8620765 - Flags: review+
https://hg.mozilla.org/mozilla-central/rev/0e2935ed6dd2
https://hg.mozilla.org/mozilla-central/rev/b792def14028
https://hg.mozilla.org/mozilla-central/rev/d45f5cd7c687
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago
status-firefox41: affected → fixed
Flags: in-testsuite+
Resolution: --- → FIXED
Target Milestone: --- → mozilla41
Depends on: 1175257
Depends on: 1175590

Comment 34

3 years ago
 Firefox isn't capable to record "CanvasCaptureMediaStream" object. Why is it so? I could't use canvas.captureStream() which returns CanvasCaptureMediaStream object. I need to record the firefox browser. How could I do with MediaStream API, if this object is not supported in firefox?

Comment 35

3 years ago
 Firefox isn't capable to record "CanvasCaptureMediaStream" object. Why is it so? I could't use canvas.captureStream() which returns CanvasCaptureMediaStream object. I need to record the firefox browser. How could I do with MediaStream API, if this object is not supported in firefox?
(Assignee)

Comment 36

3 years ago
This is not a support channel. Please file a new bug if you have found one.
You need to log in before you can comment on or make changes to this bug.