313462 - plugin content not drawn via drawWindow() on Windows

Status: RESOLVED FIXED

(Core :: Graphics: Canvas2D, defect, P2)

Description

[Peter van der Woude [:Peter6]]

Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8b5) Gecko/20051023 Firefox/1.5 ID:2005102302

<embed src="object.swf" type="application/x-shockwave-flash"></embed>
does not get painted in <canvas>

<embed src="object.swf" wmode="transparent" type="application/x-shockwave-flash"></embed>
does get painted in <canvas>

Unfortunately, setting wmode="transparent" causes a major slowdown for swf's

see http://www.macromedia.com/cfusion/knowledgebase/index.cfm?id=tn_14201

It would be nice if the <canvas> could paint the swf as if wmode="transparent" was set.

Comment 1

[Anne (:annevk)]

Example of your idea?

Comment 2

[Peter van der Woude [:Peter6]]

Attachment: swf for testcase

Comment 3

[Peter van der Woude [:Peter6]]

Attachment: tescase NO wmode

Comment 4

[Peter van der Woude [:Peter6]]

Attachment: testcase NO wmode

Comment 5

[Peter van der Woude [:Peter6]]

Attachment: testcase wmode="transparent"

Used extension:
http://users.blueprintit.co.uk/~dave/web/firefox/tabsidebar/index.html
 set Menu->View->Sidebar-> "tab sidebar" ON

Comment 6

[Anne (:annevk)]

It still is not clear to me what this has to do with html:canvas. (If the extension is required, please give an example where it is not.)

Comment 7

[Peter van der Woude [:Peter6]]

(In reply to comment #6)
> It still is not clear to me what this has to do with html:canvas. (If the
> extension is required, please give an example where it is not.)
> 

I don't have an example with html:canvas.

Comment 8

[Dave Townsend [:mossop]]

The extension uses the canvas method drawWindow to display its previews of the tab. This draws the entire contents of a given window onto the canvas. However in some circumstances the content on embeds is included and in others it is not.

I don't think its possible to come up with a testcase that isn't an extension since the drawWindow method is restricted to chrome code.

Comment 9

[Christian :Biesinger (don't email me, ping me on IRC)]

a testcase could use enablePrivilege...

Comment 10

[(not currently active) Ted Mielczarek]

See bug 313178 for an example of writing a testcase for drawWindow().

Comment 11

[Dave Townsend [:mossop]]

Attached is a testcase that includes both of the previous testcases in iframes then attempts to draw each to a canvas element. The second, that with wmode="transparent" gets drawn correctly while the first does not.

This also only works when the background colour is specified as opaque as in the testcase.

Comment 12

[Dave Townsend [:mossop]]

Attachment: testcase drawing iframes onto canvas

The mentioned testcase

Comment 13

[Dave Townsend [:mossop]]

Attachment: Replacement testcase

The previous version had a minor error and was dependant on which iframe loaded last.

I noticed with the last one that the privilege request didnt work from the attachment, so you may have to save this attachment to your disk and open it from there.

Comment 14

[Robert O'Callahan (:roc) (email my personal email if necessary)]

This is known ... it's hard to fix.

Comment 15

[Peter van der Woude [:Peter6]]

Attachment: screenshot previews

There are a few reasons why this would be good (if fixable):
1. MS is going to offer previews in IE7.
2. We need browser improvements that attract users (Eyecandy).
(Joe User doesn't care we fixed 1500 CSS bugs sice FF1.0)

Comment 16

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

That's true; however, not rendering flash content isn't all that much of a dealbreaker, IMO.

Comment 17

[Robert O'Callahan (:roc) (email my personal email if necessary)]

I wonder if we could use WM_PRINT here.

Comment 18

[Robert O'Callahan (:roc) (email my personal email if necessary)]

the patch in bug 317447 should fix this.

Comment 19

[Kathleen :Brade]

The testcase explained in comment 11 and attached in comment 13 (by Dave Townsend (Mossop)) doesn't seem valid to me.  Neither plugin is rendered on the canvas in 1.5.0.4 Firefox on my Intel-based Mac.  I see the same behavior with FF2beta1 (neither flash object renders on the canvas).  I also see the same behavior if I save the files locally.

I would like to obsolete the test case; objections?

Comment 20

[Dave Townsend [:mossop]]

(In reply to comment #19)
> I would like to obsolete the test case; objections?

If you have a better testcase then by all means. But the testcase in comment 13 is still valid for me.

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a1) Gecko/20060719 Minefield/3.0a1 ID:2006071913

Comment 21

[Mark Smith [:mcs]]

That test case works for me also (on Windows).  Apparently the Flash plugin acts different on the Mac.

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4

(In reply to comment #20)
> ... But the testcase in comment 13 is still valid for me.
> 
> Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a1) Gecko/20060719
> Minefield/3.0a1 ID:2006071913

Comment 22

[roxlu]

Hi  

What's the current status of this bug, is someone working on this right now? If so please contact me as I want to help to solve this bug.

Comment 24

[Joe Drew (not getting mail)]

This seems to work now; at least, attachment 200522 [details] renders exactly as I expect it to. RESOLVED FIXED?

Comment 25

[Dave Townsend [:mossop]]

This does indeed look right to me on OSX. What OS are you on Joe?

Comment 26

[Joe Drew (not getting mail)]

This works fine on OS X on Trunk, but (I imagine) isn't fixed in 1.8.

Comment 27

[Peter van der Woude [:Peter6]]

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b5pre) Gecko/2008030405 Minefield/3.0b5pre ID:2008030405
Hold on, this doesn't work on Windows

Comment 28

[Robert O'Callahan (:roc) (email my personal email if necessary)]

To get this working on Windows is going to require some use of PrintWindow (not available on Win2K). Fun little project for someone but not blocking.

Comment 29

[Henrik Andersson]

The testcase almost works, it's just that the top test does not work and that the bottom test does not update in realtime. This on windows.

Comment 30

[Dave Townsend [:mossop]]

(In reply to comment #29)
> The testcase almost works, it's just that the top test does not work and that
> the bottom test does not update in realtime. This on windows.

It isn't meant to update in realtime. The bottom test has always worked. So basically nothing has changed on Windows, but it has fixed on OSX.

Comment 31

[Peter van der Woude [:Peter6]]

(In reply to comment #30)

> It isn't meant to update in realtime. The bottom test has always worked. So
> basically nothing has changed on Windows, but it has fixed on OSX.
> 
Neither of the clocks show up in the TabSidebar, I don't think this ever worked after we switched to Cairo

Comment 32

[Rami Hänninen]

After some tests with some extensions that use 'drawWindow()', it seems that plugin content rendering has newer worked in Linux environment. And in current Cairo -based implementation, it still does not work. All other content seems to be rendered beautifully and fast, however, so we are moving into the right direction.

This is not an entirely minor issue as some earlier commentators have argued, because there are those nasty 'Flash only' sites out there that will show up as blank pages in any application that that tries to apply 'drawWindow' on them. A good example of (such a bad site design) would be 'http://www.acrobat.com/'.

As far as I understand these things, capturing "external" and "live" plugin generated visual content is much harder than capturing "internal" and "static" HTML DOM controlled content. The fix might event require some updates to Cairo, too, so if this is really the case, some collaboration with Cairo developers might be in order.

Finally, many plugins implement various "intro" -animations that are nothing but eye-candy. Such intro -animations are troublesome for applications that wish to capture a page as it appears after it is "fully armed" (after all intro-animations have finished). I known I'm drifting out of the scope of this bug, but I would still like to propose that it would be nice to have some plugin control API that 'drawWindow()' could use to ask the the plugins to skip over their intro animations so that 'drawWindow()' could capture the plugin visual content as it appears after all intro animations have finished.

Comment 33

[Karl Tomlinson (:karlt)]

(In reply to comment #0)
> Unfortunately, setting wmode="transparent" causes a major slowdown for swf's
> 
> see http://www.macromedia.com/cfusion/knowledgebase/index.cfm?id=tn_14201

wmode="opaque" would be more consistent with the default "window" setting, and
yet still render fine with drawWindow().

http://www.communitymx.com/content/source/E5141/

> It would be nice if the <canvas> could paint the swf as if wmode="transparent"
> was set.

The value of the wmode parameter can only be altered before being provided at
plugin instantiation.  I don't think we'd want to create a different plugin
instance for drawWindow because that would then be drawing different content.
There may be a case for providing a wmode="opaque" parameter to flash plugins
when the parameter is not set if performance with wmode="opaque" is
acceptable.

The best solution for windowed plugins on Linux would be to use the X11
Damage and Composite extensions, but that would be quite a lot of work
(bug 17447 comment 22).

(In reply to comment #11)
> This also only works when the background colour is specified as opaque as in
> the testcase.

Due to bug 453401, attachment 200522 [details] now only works when all html files are saved to disk (and the parent is editted appropriately).

But when I do that, semi-transparent background colours seem to work as
expected.

Comment 34

[Dão Gottwald [:dao]]

Did but 458928 entirely fix this?

Comment 35

[Robert O'Callahan (:roc) (email my personal email if necessary)]

No. Windowed plugins that are scrolled out of view will not be painted properly.

Comment 36

[/\/\arcio Galli]

Please consider this comment just a vote for the bug and also a thanks message since I have tested recently on Linux, and the PIP extension worked fine ( https://addons.mozilla.org/en-US/developers/add/7025 ) the extension recaptures canvas from a video being played, and places it over a floating pane. So far I have hacked with the wmode="transparent" workaround suggested by Robert.

Comment 37

[Robert O'Callahan (:roc) (email my personal email if necessary)]

It's not clear how to fix the rest of this bug.

Comment 38

[markkropf]

It has now been a few years since user roxlu offered to work on this. Is this still on anyone's radar?

Comment 41

[u279076]

I'm wondering if this should block 1.9.2?  This bug affects Aero Peek tab previews on Windows 7 when watching youtube videos.  When looking at the preview in Firefox, most of the time a black square is just shown, or the frame of the video when the window was taken out of focus is shown.  

If the goal of Aero Peek in 1.9.2 is IE8 parity, this should definitely block, IMO (IE8 previews play the video perfectly).

Just throwing this out there...

Comment 42

[Robert O'Callahan (:roc) (email my personal email if necessary)]

This can't block 1.9.2 because I don't even know if it's possible to fix it. IE uses an ActiveX version of Flash, and I don't think we want to support ActiveX just to fix this bug.

Comment 43

[Dão Gottwald [:dao]]

Aero Peek won't be in 3.6. See bug 525475.

Comment 44

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Let me expand comment #35 a little bit to make it clear what the status is.

We have code on trunk that calls the PrintWindow API on Windows when we're trying to paint a windowed plugin to a destination that's not the screen. Unfortunately it doesn't work in some cases because Windows computes a clip region for the plugin based on the HWND hierarchy, and clips the drawing to that region. So a plugin that is scrolled out of the window, or hidden, will not be drawn properly.

I don't know any way to work around that clipping problem, nor do I know any other approach to getting NPAPI plugins (especially Flash) to paint into an offscreen buffer. But some evil hack might exist, I don't know.

One thing that probably won't work, but I don't think has been tried, is firing the windowless paint message at a windowed plugin to get it to paint. That might work for Flash but it's unlikely to work for other plugins. Might be worth having a go, though, for plugins that advertise support for windowless mode.

Comment 45

[u279076]

I've found some inconsistency to how Flash previews are handled:

1. http://video.mpora.com/watch/JrCFkxtEh/ -> video plays in preview
2. http://www.youtube.com/watch?v=-uDixD4tXXU -> last frame displayed in preview

I'm not sure whether it's appropriate to carry this on in this bug or if a new bug should be filed specifically for Aero Peek and Flash Video.

Comment 47

[[not reading bugmail]]

(In reply to comment #45)
> I've found some inconsistency to how Flash previews are handled:
> 
> 1. http://video.mpora.com/watch/JrCFkxtEh/ -> video plays in preview
> 2. http://www.youtube.com/watch?v=-uDixD4tXXU -> last frame displayed in
> preview
> 
> I'm not sure whether it's appropriate to carry this on in this bug or if a new
> bug should be filed specifically for Aero Peek and Flash Video.

Aero Peek is bug 544924.

(In reply to comment #44)
> is firing the windowless paint message at a windowed plugin to get it to paint. That might work for Flash but it's unlikely to work for other plugins. Might be worth having a go, though, for plugins that advertise support for windowless mode.

I was thinking the same, but has it been tested yet?

Comment 48

[Robert O'Callahan (:roc) (email my personal email if necessary)]

I don't think so.

Comment 49

[Jim Mathies [:jimm]]

I've got a fix for bug 542656, and found this bug while testing. This is for oopp and non-oopp, windowed plugins. The print window code gets called once when the plugin loads, and then no more.

Roc, ideas where I might go searching for the problem? Maybe I can nail this bug while I'm at it.

Comment 50

[Robert O'Callahan (:roc) (email my personal email if necessary)]

No idea, sorry

Comment 51

[Jim Mathies [:jimm]]

I think I see the problem. When a tab paints, the taskbar preview script calls TaskbarPreview::Invalidate() so windows will request a new thumbnail. In the case of windowless, everything works as expected. With windowed controls, paints are handled internally by the plugin, so invalidate isn't called.

Since we have a sub class on these windows in the plugin code, it shouldn't be too hard to trap paints and call invalidate directly for the (tab?) top level window that contains the plugin. I'll see if I can work something up tomorrow.

Comment 52

[Jim Mathies [:jimm]]

(In reply to comment #51)
> I think I see the problem. When a tab paints, the taskbar preview script calls
> TaskbarPreview::Invalidate() so windows will request a new thumbnail. In the
> case of windowless, everything works as expected. With windowed controls,
> paints are handled internally by the plugin, so invalidate isn't called.
> 
> Since we have a sub class on these windows in the plugin code, it shouldn't be
> too hard to trap paints and call invalidate directly for the (tab?) top level
> window that contains the plugin. I'll see if I can work something up tomorrow.

Hmm, well, flash is doing non-standard painting to their window, paint events aren't being generated on a regular basis. We'll need some other solution here.

Comment 53

[Jim Mathies [:jimm]]

filed bug 557823 on plugin refresh related issues for tab previews.