Last Comment Bug 313462 - plugin content not drawn via drawWindow() on Windows
: plugin content not drawn via drawWindow() on Windows
Status: RESOLVED FIXED
:
Product: Core
Classification: Components
Component: Canvas: 2D (show other bugs)
: Trunk
: x86 Windows 2000
: P2 normal with 20 votes (vote)
: ---
Assigned To: Nobody; OK to take it and work on it
:
: Milan Sreckovic [:milan]
Mentors:
: 370292 445595 522969 550393 (view as bug list)
Depends on: 317447 453401 458928
Blocks: 329693 505404 442109 544924
  Show dependency treegraph
 
Reported: 2005-10-23 03:38 PDT by Peter van der Woude [:Peter6]
Modified: 2010-04-07 08:22 PDT (History)
48 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
swf for testcase (773 bytes, application/x-shockwave-flash)
2005-10-23 05:00 PDT, Peter van der Woude [:Peter6]
no flags Details
tescase NO wmode (424 bytes, text/plain)
2005-10-23 05:01 PDT, Peter van der Woude [:Peter6]
no flags Details
testcase NO wmode (468 bytes, text/html)
2005-10-23 05:02 PDT, Peter van der Woude [:Peter6]
no flags Details
testcase wmode="transparent" (488 bytes, text/html)
2005-10-23 05:05 PDT, Peter van der Woude [:Peter6]
no flags Details
testcase drawing iframes onto canvas (1.22 KB, text/html)
2005-10-23 08:00 PDT, Dave Townsend [:mossop]
no flags Details
Replacement testcase (1.05 KB, text/html)
2005-10-23 10:31 PDT, Dave Townsend [:mossop]
no flags Details
screenshot previews (103.95 KB, image/png)
2005-10-23 15:00 PDT, Peter van der Woude [:Peter6]
no flags Details

Description User image Peter van der Woude [:Peter6] 2005-10-23 03:38:01 PDT
Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8b5) Gecko/20051023 Firefox/1.5 ID:2005102302

<embed src="object.swf" type="application/x-shockwave-flash"></embed>
does not get painted in <canvas>

<embed src="object.swf" wmode="transparent" type="application/x-shockwave-flash"></embed>
does get painted in <canvas>

Unfortunately, setting wmode="transparent" causes a major slowdown for swf's

see http://www.macromedia.com/cfusion/knowledgebase/index.cfm?id=tn_14201

It would be nice if the <canvas> could paint the swf as if wmode="transparent" was set.
Comment 1 User image Anne (:annevk) 2005-10-23 04:52:23 PDT
Example of your idea?
Comment 2 User image Peter van der Woude [:Peter6] 2005-10-23 05:00:08 PDT
Created attachment 200496 [details]
swf for testcase
Comment 3 User image Peter van der Woude [:Peter6] 2005-10-23 05:01:31 PDT
Created attachment 200497 [details]
tescase NO wmode
Comment 4 User image Peter van der Woude [:Peter6] 2005-10-23 05:02:58 PDT
Created attachment 200498 [details]
testcase NO wmode
Comment 5 User image Peter van der Woude [:Peter6] 2005-10-23 05:05:26 PDT
Created attachment 200500 [details]
testcase wmode="transparent"

Used extension:
http://users.blueprintit.co.uk/~dave/web/firefox/tabsidebar/index.html
 set Menu->View->Sidebar-> "tab sidebar" ON
Comment 6 User image Anne (:annevk) 2005-10-23 05:10:18 PDT
It still is not clear to me what this has to do with html:canvas. (If the extension is required, please give an example where it is not.)
Comment 7 User image Peter van der Woude [:Peter6] 2005-10-23 05:15:48 PDT
(In reply to comment #6)
> It still is not clear to me what this has to do with html:canvas. (If the
> extension is required, please give an example where it is not.)
> 

I don't have an example with html:canvas.
Comment 8 User image Dave Townsend [:mossop] 2005-10-23 05:23:30 PDT
The extension uses the canvas method drawWindow to display its previews of the tab. This draws the entire contents of a given window onto the canvas. However in some circumstances the content on embeds is included and in others it is not.

I don't think its possible to come up with a testcase that isn't an extension since the drawWindow method is restricted to chrome code.
Comment 9 User image Christian :Biesinger (don't email me, ping me on IRC) 2005-10-23 06:14:16 PDT
a testcase could use enablePrivilege...

Comment 10 User image Ted Mielczarek [:ted.mielczarek] 2005-10-23 07:29:35 PDT
See bug 313178 for an example of writing a testcase for drawWindow().
Comment 11 User image Dave Townsend [:mossop] 2005-10-23 07:59:03 PDT
Attached is a testcase that includes both of the previous testcases in iframes then attempts to draw each to a canvas element. The second, that with wmode="transparent" gets drawn correctly while the first does not.

This also only works when the background colour is specified as opaque as in the testcase.
Comment 12 User image Dave Townsend [:mossop] 2005-10-23 08:00:21 PDT
Created attachment 200514 [details]
testcase drawing iframes onto canvas

The mentioned testcase
Comment 13 User image Dave Townsend [:mossop] 2005-10-23 10:31:16 PDT
Created attachment 200522 [details]
Replacement testcase

The previous version had a minor error and was dependant on which iframe loaded last.

I noticed with the last one that the privilege request didnt work from the attachment, so you may have to save this attachment to your disk and open it from there.
Comment 14 User image Robert O'Callahan (:roc) (email my personal email if necessary) 2005-10-23 14:19:11 PDT
This is known ... it's hard to fix.
Comment 15 User image Peter van der Woude [:Peter6] 2005-10-23 15:00:35 PDT
Created attachment 200553 [details]
screenshot previews

There are a few reasons why this would be good (if fixable):
1. MS is going to offer previews in IE7.
2. We need browser improvements that attract users (Eyecandy).
(Joe User doesn't care we fixed 1500 CSS bugs sice FF1.0)
Comment 16 User image Vladimir Vukicevic [:vlad] [:vladv] 2005-10-23 19:07:08 PDT
That's true; however, not rendering flash content isn't all that much of a dealbreaker, IMO.
Comment 17 User image Robert O'Callahan (:roc) (email my personal email if necessary) 2005-10-24 01:00:10 PDT
I wonder if we could use WM_PRINT here.
Comment 18 User image Robert O'Callahan (:roc) (email my personal email if necessary) 2005-11-22 16:41:08 PST
the patch in bug 317447 should fix this.
Comment 19 User image Kathleen Brade 2006-07-21 08:49:03 PDT
The testcase explained in comment 11 and attached in comment 13 (by Dave Townsend (Mossop)) doesn't seem valid to me.  Neither plugin is rendered on the canvas in 1.5.0.4 Firefox on my Intel-based Mac.  I see the same behavior with FF2beta1 (neither flash object renders on the canvas).  I also see the same behavior if I save the files locally.

I would like to obsolete the test case; objections?
Comment 20 User image Dave Townsend [:mossop] 2006-07-21 08:56:17 PDT
(In reply to comment #19)
> I would like to obsolete the test case; objections?

If you have a better testcase then by all means. But the testcase in comment 13 is still valid for me.

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a1) Gecko/20060719 Minefield/3.0a1 ID:2006071913

Comment 21 User image Mark Smith (:mcs) 2006-07-21 09:04:53 PDT
That test case works for me also (on Windows).  Apparently the Flash plugin acts different on the Mac.

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4

(In reply to comment #20)
> ... But the testcase in comment 13 is still valid for me.
> 
> Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a1) Gecko/20060719
> Minefield/3.0a1 ID:2006071913
Comment 22 User image roxlu 2006-10-27 12:44:00 PDT
Hi  

What's the current status of this bug, is someone working on this right now? If so please contact me as I want to help to solve this bug.
Comment 23 User image Chris Thomas (CTho) [formerly cst@andrew.cmu.edu cst@yecc.com] 2007-06-20 16:56:55 PDT
*** Bug 370292 has been marked as a duplicate of this bug. ***
Comment 24 User image Joe Drew (not getting mail) 2008-03-04 13:09:44 PST
This seems to work now; at least, attachment 200522 [details] renders exactly as I expect it to. RESOLVED FIXED?
Comment 25 User image Dave Townsend [:mossop] 2008-03-04 13:20:51 PST
This does indeed look right to me on OSX. What OS are you on Joe?
Comment 26 User image Joe Drew (not getting mail) 2008-03-04 13:22:32 PST
This works fine on OS X on Trunk, but (I imagine) isn't fixed in 1.8.
Comment 27 User image Peter van der Woude [:Peter6] 2008-03-04 13:32:28 PST
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b5pre) Gecko/2008030405 Minefield/3.0b5pre ID:2008030405
Hold on, this doesn't work on Windows
Comment 28 User image Robert O'Callahan (:roc) (email my personal email if necessary) 2008-03-04 13:42:09 PST
To get this working on Windows is going to require some use of PrintWindow (not available on Win2K). Fun little project for someone but not blocking.
Comment 29 User image Henrik Andersson 2008-03-04 13:45:52 PST
The testcase almost works, it's just that the top test does not work and that the bottom test does not update in realtime. This on windows.
Comment 30 User image Dave Townsend [:mossop] 2008-03-04 13:59:55 PST
(In reply to comment #29)
> The testcase almost works, it's just that the top test does not work and that
> the bottom test does not update in realtime. This on windows.

It isn't meant to update in realtime. The bottom test has always worked. So basically nothing has changed on Windows, but it has fixed on OSX.
Comment 31 User image Peter van der Woude [:Peter6] 2008-03-04 14:08:35 PST
(In reply to comment #30)

> It isn't meant to update in realtime. The bottom test has always worked. So
> basically nothing has changed on Windows, but it has fixed on OSX.
> 
Neither of the clocks show up in the TabSidebar, I don't think this ever worked after we switched to Cairo
Comment 32 User image Rami Hänninen 2008-07-01 00:48:39 PDT
After some tests with some extensions that use 'drawWindow()', it seems that plugin content rendering has newer worked in Linux environment. And in current Cairo -based implementation, it still does not work. All other content seems to be rendered beautifully and fast, however, so we are moving into the right direction.

This is not an entirely minor issue as some earlier commentators have argued, because there are those nasty 'Flash only' sites out there that will show up as blank pages in any application that that tries to apply 'drawWindow' on them. A good example of (such a bad site design) would be 'http://www.acrobat.com/'.

As far as I understand these things, capturing "external" and "live" plugin generated visual content is much harder than capturing "internal" and "static" HTML DOM controlled content. The fix might event require some updates to Cairo, too, so if this is really the case, some collaboration with Cairo developers might be in order.

Finally, many plugins implement various "intro" -animations that are nothing but eye-candy. Such intro -animations are troublesome for applications that wish to capture a page as it appears after it is "fully armed" (after all intro-animations have finished). I known I'm drifting out of the scope of this bug, but I would still like to propose that it would be nice to have some plugin control API that 'drawWindow()' could use to ask the the plugins to skip over their intro animations so that 'drawWindow()' could capture the plugin visual content as it appears after all intro animations have finished.
Comment 33 User image Karl Tomlinson (back Feb 1 :karlt) 2008-09-02 22:40:39 PDT
(In reply to comment #0)
> Unfortunately, setting wmode="transparent" causes a major slowdown for swf's
> 
> see http://www.macromedia.com/cfusion/knowledgebase/index.cfm?id=tn_14201

wmode="opaque" would be more consistent with the default "window" setting, and
yet still render fine with drawWindow().

http://www.communitymx.com/content/source/E5141/

> It would be nice if the <canvas> could paint the swf as if wmode="transparent"
> was set.

The value of the wmode parameter can only be altered before being provided at
plugin instantiation.  I don't think we'd want to create a different plugin
instance for drawWindow because that would then be drawing different content.
There may be a case for providing a wmode="opaque" parameter to flash plugins
when the parameter is not set if performance with wmode="opaque" is
acceptable.

The best solution for windowed plugins on Linux would be to use the X11
Damage and Composite extensions, but that would be quite a lot of work
(bug 17447 comment 22).

(In reply to comment #11)
> This also only works when the background colour is specified as opaque as in
> the testcase.

Due to bug 453401, attachment 200522 [details] now only works when all html files are saved to disk (and the parent is editted appropriately).

But when I do that, semi-transparent background colours seem to work as
expected.
Comment 34 User image Dão Gottwald [:dao] 2008-11-08 05:57:33 PST
Did but 458928 entirely fix this?
Comment 35 User image Robert O'Callahan (:roc) (email my personal email if necessary) 2008-11-09 13:06:57 PST
No. Windowed plugins that are scrolled out of view will not be painted properly.
Comment 36 User image /\/\arcio Galli 2008-11-17 08:33:04 PST
Please consider this comment just a vote for the bug and also a thanks message since I have tested recently on Linux, and the PIP extension worked fine ( https://addons.mozilla.org/en-US/developers/add/7025 ) the extension recaptures canvas from a video being played, and places it over a floating pane. So far I have hacked with the wmode="transparent" workaround suggested by Robert.
Comment 37 User image Robert O'Callahan (:roc) (email my personal email if necessary) 2008-11-17 12:52:24 PST
It's not clear how to fix the rest of this bug.
Comment 38 User image markkropf 2009-02-17 08:40:48 PST
It has now been a few years since user roxlu offered to work on this. Is this still on anyone's radar?
Comment 39 User image Dão Gottwald [:dao] 2009-10-19 08:58:48 PDT
*** Bug 445595 has been marked as a duplicate of this bug. ***
Comment 40 User image Dão Gottwald [:dao] 2009-10-19 08:59:18 PDT
*** Bug 522969 has been marked as a duplicate of this bug. ***
Comment 41 User image Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2009-11-09 12:00:26 PST
I'm wondering if this should block 1.9.2?  This bug affects Aero Peek tab previews on Windows 7 when watching youtube videos.  When looking at the preview in Firefox, most of the time a black square is just shown, or the frame of the video when the window was taken out of focus is shown.  

If the goal of Aero Peek in 1.9.2 is IE8 parity, this should definitely block, IMO (IE8 previews play the video perfectly).

Just throwing this out there...
Comment 42 User image Robert O'Callahan (:roc) (email my personal email if necessary) 2009-11-09 12:06:34 PST
This can't block 1.9.2 because I don't even know if it's possible to fix it. IE uses an ActiveX version of Flash, and I don't think we want to support ActiveX just to fix this bug.
Comment 43 User image Dão Gottwald [:dao] 2009-11-09 12:07:53 PST
Aero Peek won't be in 3.6. See bug 525475.
Comment 44 User image Robert O'Callahan (:roc) (email my personal email if necessary) 2009-11-09 12:18:54 PST
Let me expand comment #35 a little bit to make it clear what the status is.

We have code on trunk that calls the PrintWindow API on Windows when we're trying to paint a windowed plugin to a destination that's not the screen. Unfortunately it doesn't work in some cases because Windows computes a clip region for the plugin based on the HWND hierarchy, and clips the drawing to that region. So a plugin that is scrolled out of the window, or hidden, will not be drawn properly.

I don't know any way to work around that clipping problem, nor do I know any other approach to getting NPAPI plugins (especially Flash) to paint into an offscreen buffer. But some evil hack might exist, I don't know.

One thing that probably won't work, but I don't think has been tried, is firing the windowless paint message at a windowed plugin to get it to paint. That might work for Flash but it's unlikely to work for other plugins. Might be worth having a go, though, for plugins that advertise support for windowless mode.
Comment 45 User image Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2009-11-09 12:50:42 PST
I've found some inconsistency to how Flash previews are handled:

1. http://video.mpora.com/watch/JrCFkxtEh/ -> video plays in preview
2. http://www.youtube.com/watch?v=-uDixD4tXXU -> last frame displayed in preview

I'm not sure whether it's appropriate to carry this on in this bug or if a new bug should be filed specifically for Aero Peek and Flash Video.
Comment 46 User image Brian Polidoro 2010-03-05 15:01:06 PST
*** Bug 550393 has been marked as a duplicate of this bug. ***
Comment 47 User image [not reading bugmail] 2010-03-08 17:28:21 PST
(In reply to comment #45)
> I've found some inconsistency to how Flash previews are handled:
> 
> 1. http://video.mpora.com/watch/JrCFkxtEh/ -> video plays in preview
> 2. http://www.youtube.com/watch?v=-uDixD4tXXU -> last frame displayed in
> preview
> 
> I'm not sure whether it's appropriate to carry this on in this bug or if a new
> bug should be filed specifically for Aero Peek and Flash Video.

Aero Peek is bug 544924.

(In reply to comment #44)
> is firing the windowless paint message at a windowed plugin to get it to paint. That might work for Flash but it's unlikely to work for other plugins. Might be worth having a go, though, for plugins that advertise support for windowless mode.

I was thinking the same, but has it been tested yet?
Comment 48 User image Robert O'Callahan (:roc) (email my personal email if necessary) 2010-03-08 17:44:01 PST
I don't think so.
Comment 49 User image Jim Mathies [:jimm] 2010-04-05 14:46:10 PDT
I've got a fix for bug 542656, and found this bug while testing. This is for oopp and non-oopp, windowed plugins. The print window code gets called once when the plugin loads, and then no more.

Roc, ideas where I might go searching for the problem? Maybe I can nail this bug while I'm at it.
Comment 50 User image Robert O'Callahan (:roc) (email my personal email if necessary) 2010-04-05 16:15:49 PDT
No idea, sorry
Comment 51 User image Jim Mathies [:jimm] 2010-04-05 16:43:33 PDT
I think I see the problem. When a tab paints, the taskbar preview script calls TaskbarPreview::Invalidate() so windows will request a new thumbnail. In the case of windowless, everything works as expected. With windowed controls, paints are handled internally by the plugin, so invalidate isn't called.

Since we have a sub class on these windows in the plugin code, it shouldn't be too hard to trap paints and call invalidate directly for the (tab?) top level window that contains the plugin. I'll see if I can work something up tomorrow.
Comment 52 User image Jim Mathies [:jimm] 2010-04-06 07:10:43 PDT
(In reply to comment #51)
> I think I see the problem. When a tab paints, the taskbar preview script calls
> TaskbarPreview::Invalidate() so windows will request a new thumbnail. In the
> case of windowless, everything works as expected. With windowed controls,
> paints are handled internally by the plugin, so invalidate isn't called.
> 
> Since we have a sub class on these windows in the plugin code, it shouldn't be
> too hard to trap paints and call invalidate directly for the (tab?) top level
> window that contains the plugin. I'll see if I can work something up tomorrow.

Hmm, well, flash is doing non-standard painting to their window, paint events aren't being generated on a regular basis. We'll need some other solution here.
Comment 53 User image Jim Mathies [:jimm] 2010-04-07 08:22:04 PDT
filed bug 557823 on plugin refresh related issues for tab previews.

Note You need to log in before you can comment on or make changes to this bug.