plugin content not drawn via drawWindow() on Windows

RESOLVED FIXED

Status

()

Core
Canvas: 2D
P2
normal
RESOLVED FIXED
12 years ago
7 years ago

People

(Reporter: Peter6, Unassigned)

Tracking

(Depends on: 1 bug, Blocks: 2 bugs)

Trunk
x86
Windows 2000
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(5 attachments, 2 obsolete attachments)

(Reporter)

Description

12 years ago
Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8b5) Gecko/20051023 Firefox/1.5 ID:2005102302

<embed src="object.swf" type="application/x-shockwave-flash"></embed>
does not get painted in <canvas>

<embed src="object.swf" wmode="transparent" type="application/x-shockwave-flash"></embed>
does get painted in <canvas>

Unfortunately, setting wmode="transparent" causes a major slowdown for swf's

see http://www.macromedia.com/cfusion/knowledgebase/index.cfm?id=tn_14201

It would be nice if the <canvas> could paint the swf as if wmode="transparent" was set.

Comment 1

12 years ago
Example of your idea?
(Reporter)

Comment 2

12 years ago
Created attachment 200496 [details]
swf for testcase
(Reporter)

Comment 3

12 years ago
Created attachment 200497 [details]
tescase NO wmode
(Reporter)

Comment 4

12 years ago
Created attachment 200498 [details]
testcase NO wmode
Attachment #200497 - Attachment is obsolete: true
(Reporter)

Comment 5

12 years ago
Created attachment 200500 [details]
testcase wmode="transparent"

Used extension:
http://users.blueprintit.co.uk/~dave/web/firefox/tabsidebar/index.html
 set Menu->View->Sidebar-> "tab sidebar" ON

Comment 6

12 years ago
It still is not clear to me what this has to do with html:canvas. (If the extension is required, please give an example where it is not.)
(Reporter)

Comment 7

12 years ago
(In reply to comment #6)
> It still is not clear to me what this has to do with html:canvas. (If the
> extension is required, please give an example where it is not.)
> 

I don't have an example with html:canvas.
The extension uses the canvas method drawWindow to display its previews of the tab. This draws the entire contents of a given window onto the canvas. However in some circumstances the content on embeds is included and in others it is not.

I don't think its possible to come up with a testcase that isn't an extension since the drawWindow method is restricted to chrome code.
a testcase could use enablePrivilege...

See bug 313178 for an example of writing a testcase for drawWindow().
Attached is a testcase that includes both of the previous testcases in iframes then attempts to draw each to a canvas element. The second, that with wmode="transparent" gets drawn correctly while the first does not.

This also only works when the background colour is specified as opaque as in the testcase.
Created attachment 200514 [details]
testcase drawing iframes onto canvas

The mentioned testcase
Created attachment 200522 [details]
Replacement testcase

The previous version had a minor error and was dependant on which iframe loaded last.

I noticed with the last one that the privilege request didnt work from the attachment, so you may have to save this attachment to your disk and open it from there.
Attachment #200514 - Attachment is obsolete: true
This is known ... it's hard to fix.
Summary: make flash visable in canvas → make flash visible in canvas
Summary: make flash visible in canvas → flash content not drawn via drawWindow()
(Reporter)

Comment 15

12 years ago
Created attachment 200553 [details]
screenshot previews

There are a few reasons why this would be good (if fixable):
1. MS is going to offer previews in IE7.
2. We need browser improvements that attract users (Eyecandy).
(Joe User doesn't care we fixed 1500 CSS bugs sice FF1.0)
That's true; however, not rendering flash content isn't all that much of a dealbreaker, IMO.
I wonder if we could use WM_PRINT here.
Depends on: 317447
Summary: flash content not drawn via drawWindow() → plugin content not drawn via drawWindow()
the patch in bug 317447 should fix this.

Comment 19

11 years ago
The testcase explained in comment 11 and attached in comment 13 (by Dave Townsend (Mossop)) doesn't seem valid to me.  Neither plugin is rendered on the canvas in 1.5.0.4 Firefox on my Intel-based Mac.  I see the same behavior with FF2beta1 (neither flash object renders on the canvas).  I also see the same behavior if I save the files locally.

I would like to obsolete the test case; objections?
(In reply to comment #19)
> I would like to obsolete the test case; objections?

If you have a better testcase then by all means. But the testcase in comment 13 is still valid for me.

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a1) Gecko/20060719 Minefield/3.0a1 ID:2006071913

Comment 21

11 years ago
That test case works for me also (on Windows).  Apparently the Flash plugin acts different on the Mac.

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4

(In reply to comment #20)
> ... But the testcase in comment 13 is still valid for me.
> 
> Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a1) Gecko/20060719
> Minefield/3.0a1 ID:2006071913

Comment 22

11 years ago
Hi  

What's the current status of this bug, is someone working on this right now? If so please contact me as I want to help to solve this bug.
Duplicate of this bug: 370292
This seems to work now; at least, attachment 200522 [details] renders exactly as I expect it to. RESOLVED FIXED?
This does indeed look right to me on OSX. What OS are you on Joe?
This works fine on OS X on Trunk, but (I imagine) isn't fixed in 1.8.
Status: NEW → RESOLVED
Last Resolved: 9 years ago
Resolution: --- → WORKSFORME
Version: 1.8 Branch → Trunk
(Reporter)

Comment 27

9 years ago
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b5pre) Gecko/2008030405 Minefield/3.0b5pre ID:2008030405
Hold on, this doesn't work on Windows
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
Priority: -- → P2
To get this working on Windows is going to require some use of PrintWindow (not available on Win2K). Fun little project for someone but not blocking.

Comment 29

9 years ago
The testcase almost works, it's just that the top test does not work and that the bottom test does not update in realtime. This on windows.
(In reply to comment #29)
> The testcase almost works, it's just that the top test does not work and that
> the bottom test does not update in realtime. This on windows.

It isn't meant to update in realtime. The bottom test has always worked. So basically nothing has changed on Windows, but it has fixed on OSX.
(Reporter)

Comment 31

9 years ago
(In reply to comment #30)

> It isn't meant to update in realtime. The bottom test has always worked. So
> basically nothing has changed on Windows, but it has fixed on OSX.
> 
Neither of the clocks show up in the TabSidebar, I don't think this ever worked after we switched to Cairo

Comment 32

9 years ago
After some tests with some extensions that use 'drawWindow()', it seems that plugin content rendering has newer worked in Linux environment. And in current Cairo -based implementation, it still does not work. All other content seems to be rendered beautifully and fast, however, so we are moving into the right direction.

This is not an entirely minor issue as some earlier commentators have argued, because there are those nasty 'Flash only' sites out there that will show up as blank pages in any application that that tries to apply 'drawWindow' on them. A good example of (such a bad site design) would be 'http://www.acrobat.com/'.

As far as I understand these things, capturing "external" and "live" plugin generated visual content is much harder than capturing "internal" and "static" HTML DOM controlled content. The fix might event require some updates to Cairo, too, so if this is really the case, some collaboration with Cairo developers might be in order.

Finally, many plugins implement various "intro" -animations that are nothing but eye-candy. Such intro -animations are troublesome for applications that wish to capture a page as it appears after it is "fully armed" (after all intro-animations have finished). I known I'm drifting out of the scope of this bug, but I would still like to propose that it would be nice to have some plugin control API that 'drawWindow()' could use to ask the the plugins to skip over their intro animations so that 'drawWindow()' could capture the plugin visual content as it appears after all intro animations have finished.
Blocks: 443945

Updated

9 years ago
Blocks: 445474

Updated

9 years ago
Blocks: 445595

Updated

9 years ago
No longer blocks: 445474

Updated

9 years ago
Blocks: 329693
No longer blocks: 443945
Blocks: 442109
(In reply to comment #0)
> Unfortunately, setting wmode="transparent" causes a major slowdown for swf's
> 
> see http://www.macromedia.com/cfusion/knowledgebase/index.cfm?id=tn_14201

wmode="opaque" would be more consistent with the default "window" setting, and
yet still render fine with drawWindow().

http://www.communitymx.com/content/source/E5141/

> It would be nice if the <canvas> could paint the swf as if wmode="transparent"
> was set.

The value of the wmode parameter can only be altered before being provided at
plugin instantiation.  I don't think we'd want to create a different plugin
instance for drawWindow because that would then be drawing different content.
There may be a case for providing a wmode="opaque" parameter to flash plugins
when the parameter is not set if performance with wmode="opaque" is
acceptable.

The best solution for windowed plugins on Linux would be to use the X11
Damage and Composite extensions, but that would be quite a lot of work
(bug 17447 comment 22).

(In reply to comment #11)
> This also only works when the background colour is specified as opaque as in
> the testcase.

Due to bug 453401, attachment 200522 [details] now only works when all html files are saved to disk (and the parent is editted appropriately).

But when I do that, semi-transparent background colours seem to work as
expected.
Depends on: 453401

Updated

9 years ago
Depends on: 458928
Did but 458928 entirely fix this?
No. Windowed plugins that are scrolled out of view will not be painted properly.
Summary: plugin content not drawn via drawWindow() → plugin content not drawn via drawWindow() on Windows

Comment 36

9 years ago
Please consider this comment just a vote for the bug and also a thanks message since I have tested recently on Linux, and the PIP extension worked fine ( https://addons.mozilla.org/en-US/developers/add/7025 ) the extension recaptures canvas from a video being played, and places it over a floating pane. So far I have hacked with the wmode="transparent" workaround suggested by Robert.
It's not clear how to fix the rest of this bug.

Comment 38

9 years ago
It has now been a few years since user roxlu offered to work on this. Is this still on anyone's radar?

Updated

8 years ago
Blocks: 505404
No longer blocks: 445595

Updated

8 years ago
Duplicate of this bug: 445595

Updated

8 years ago
Duplicate of this bug: 522969
I'm wondering if this should block 1.9.2?  This bug affects Aero Peek tab previews on Windows 7 when watching youtube videos.  When looking at the preview in Firefox, most of the time a black square is just shown, or the frame of the video when the window was taken out of focus is shown.  

If the goal of Aero Peek in 1.9.2 is IE8 parity, this should definitely block, IMO (IE8 previews play the video perfectly).

Just throwing this out there...
Flags: blocking1.9.2?
This can't block 1.9.2 because I don't even know if it's possible to fix it. IE uses an ActiveX version of Flash, and I don't think we want to support ActiveX just to fix this bug.
Aero Peek won't be in 3.6. See bug 525475.
Flags: blocking1.9.2?
Let me expand comment #35 a little bit to make it clear what the status is.

We have code on trunk that calls the PrintWindow API on Windows when we're trying to paint a windowed plugin to a destination that's not the screen. Unfortunately it doesn't work in some cases because Windows computes a clip region for the plugin based on the HWND hierarchy, and clips the drawing to that region. So a plugin that is scrolled out of the window, or hidden, will not be drawn properly.

I don't know any way to work around that clipping problem, nor do I know any other approach to getting NPAPI plugins (especially Flash) to paint into an offscreen buffer. But some evil hack might exist, I don't know.

One thing that probably won't work, but I don't think has been tried, is firing the windowless paint message at a windowed plugin to get it to paint. That might work for Flash but it's unlikely to work for other plugins. Might be worth having a go, though, for plugins that advertise support for windowless mode.
I've found some inconsistency to how Flash previews are handled:

1. http://video.mpora.com/watch/JrCFkxtEh/ -> video plays in preview
2. http://www.youtube.com/watch?v=-uDixD4tXXU -> last frame displayed in preview

I'm not sure whether it's appropriate to carry this on in this bug or if a new bug should be filed specifically for Aero Peek and Flash Video.

Updated

7 years ago
Duplicate of this bug: 550393
Blocks: 544924
(In reply to comment #45)
> I've found some inconsistency to how Flash previews are handled:
> 
> 1. http://video.mpora.com/watch/JrCFkxtEh/ -> video plays in preview
> 2. http://www.youtube.com/watch?v=-uDixD4tXXU -> last frame displayed in
> preview
> 
> I'm not sure whether it's appropriate to carry this on in this bug or if a new
> bug should be filed specifically for Aero Peek and Flash Video.

Aero Peek is bug 544924.

(In reply to comment #44)
> is firing the windowless paint message at a windowed plugin to get it to paint. That might work for Flash but it's unlikely to work for other plugins. Might be worth having a go, though, for plugins that advertise support for windowless mode.

I was thinking the same, but has it been tested yet?
I don't think so.

Updated

7 years ago
Blocks: 542656

Comment 49

7 years ago
I've got a fix for bug 542656, and found this bug while testing. This is for oopp and non-oopp, windowed plugins. The print window code gets called once when the plugin loads, and then no more.

Roc, ideas where I might go searching for the problem? Maybe I can nail this bug while I'm at it.
No idea, sorry

Comment 51

7 years ago
I think I see the problem. When a tab paints, the taskbar preview script calls TaskbarPreview::Invalidate() so windows will request a new thumbnail. In the case of windowless, everything works as expected. With windowed controls, paints are handled internally by the plugin, so invalidate isn't called.

Since we have a sub class on these windows in the plugin code, it shouldn't be too hard to trap paints and call invalidate directly for the (tab?) top level window that contains the plugin. I'll see if I can work something up tomorrow.

Comment 52

7 years ago
(In reply to comment #51)
> I think I see the problem. When a tab paints, the taskbar preview script calls
> TaskbarPreview::Invalidate() so windows will request a new thumbnail. In the
> case of windowless, everything works as expected. With windowed controls,
> paints are handled internally by the plugin, so invalidate isn't called.
> 
> Since we have a sub class on these windows in the plugin code, it shouldn't be
> too hard to trap paints and call invalidate directly for the (tab?) top level
> window that contains the plugin. I'll see if I can work something up tomorrow.

Hmm, well, flash is doing non-standard painting to their window, paint events aren't being generated on a regular basis. We'll need some other solution here.

Updated

7 years ago
No longer blocks: 542656

Comment 53

7 years ago
filed bug 557823 on plugin refresh related issues for tab previews.
Status: REOPENED → RESOLVED
Last Resolved: 9 years ago7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.