492848 - Visibility notify event API for windowless plugins

Status: RESOLVED FIXED

(Core Graveyard :: Plug-ins, defect)

Description

[Oleg Romashin (:romaxa)]

For windowless plugins on mobile device it is important to know about their visibility state... 

For example if we have > 1 windows with various flash video or some other content, it would be nice to have some way to notify plugins are they visible or not. and plugins can decide to suspend some video decoding or switch games to pause state in case if they are on background (not visible window)...

Currently I cannot find any existing code or API to implement such notifications...

Can someone advice what is the best way to do that?

Comment 1

[Robert O'Callahan (:roc) (email my personal email if necessary)]

What about stuff like tab preview? Are you OK to display the old contents for hidden tabs?

Comment 2

[Josh Aas]

You might not need a new API. On Mac OS X we send NPP_SetWindow(NULL) when hiding a plugin behind another tab, you may be able to send NPP_SetWindow(NULL) on Windows/Linux mobile in a different case, for overlapping windows (not just tabs).

Comment 3

[Oleg Romashin (:romaxa)]

(In reply to comment #1)
> What about stuff like tab preview? Are you OK to display the old contents for
> hidden tabs?

I think it is ok, and it is more important to have better performance for current active page than update all the time all pages, and have whole device busy because all pages working, and rendering actively...

Comment 4

[Oleg Romashin (:romaxa)]

> You might not need a new API. On Mac OS X we send NPP_SetWindow(NULL) when
> on Windows/Linux mobile in a different case, for overlapping windows (not just
> tabs).

Yep, I know that I can send some event to plugin... but I was thinking what the best way to do that:
1) Implement separate plugin enumerator for DOMwindow, and send event for each plugin.
2) Or attach all plugin objects in nsObjectFrame to some parent dom window event, then send event to window and all objects will handle it automatically

Comment 5

[Oleg Romashin (:romaxa)]

Attachment: Proposed fix, used in microb-engine for N900

Comment 6

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Plugins in iframes won't hear about pluginsshow/pluginshide, right?

Another way the plugin can detect visibility is by tracking paints and invalidates. Suppose it periodically invalidates itself completely every few seconds. If it gets no paint event, then it must be completely invisible. If it gets a paint event, it is probably visible and should come out of its suspended state. Would that work? That wouldn't require any browser changes.

Comment 7

[Oleg Romashin (:romaxa)]

Yes, we were considering this option in the beginning... I need to check with flash team again and recall why it was rejected.

Comment 8

[Oleg Romashin (:romaxa)]

We had some problem with additional timeouts creation and internal flash implementation.

Theoretically it can be done:
NPP_InvalidateRect (create timer for 5 seconds)
..... (If timer created just continue.)
If timer finished and no Expose events came... then invoke suspend.

I will create new bug about this problem... and check how fast we can try this approach.

Comment 9

[Oleg Romashin (:romaxa)]

Also with this new approach we should document it and make sure  that all plugins have implemented this functionality...

But most of plugins already handling VisibilityNotify signals (since windowed mode)... and with my patch it is very easy to port those plugins.

Comment 10

[Oleg Romashin (:romaxa)]

Also question is why MacOSX port using this implementation and not rely all responsibility to plugins?

Comment 11

[Robert O'Callahan (:roc) (email my personal email if necessary)]

(In reply to comment #9)
> But most of plugins already handling VisibilityNotify signals (since windowed
> mode)... and with my patch it is very easy to port those plugins.

How many plugins really handle VisibilityNotify for this kind of optimization?

(In reply to comment #10)
> Also question is why MacOSX port using this implementation and not rely all
> responsibility to plugins?

I don't know, but we can't assume it's for this reason. They may need to know immediately when their window is hidden. Josh may know.

Comment 12

[Doug Turner (:dougt)]

> How many plugins really handle VisibilityNotify for this kind of optimization?

chicken and the egg.  right now, flash on the n900 device and diamondx.

Comment 13

[Robert O'Callahan (:roc) (email my personal email if necessary)]

In comment #9 Oleg said
> But most of plugins already handling VisibilityNotify signals (since windowed
> mode)... and with my patch it is very easy to port those plugins.

If "most plugins" just means Flash on the N900, then this isn't a very compelling argument.

Comment 14

[Doug Turner (:dougt)]

roc/josh, what is needed here; what are the next steps?

Comment 15

[Robert O'Callahan (:roc) (email my personal email if necessary)]

It seems to me nothing is needed here, Flash can do this optimization itself as per comments #6 and #8.

Comment 16

[Doug Turner (:dougt)]

wontfix.  per #6 #8.

Comment 17

[Doug Turner (:dougt)]

this is required to get invalidation events from the flash plugin on the maemo platform.  romaxa, can this be fixed on the plugin side before you ship?  If now, we will need something in the browser as right now we do not receive any invalidation events.

Comment 18

[Doug Turner (:dougt)]

Attachment: cleaned up patch -- minimum required.

assuming that flash can not be fixed, would this be acceptable?

Comment 19

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Comment on attachment 407933 [details] [diff] [review]
cleaned up patch -- minimum required.

This is pretty lame. It's not a very good visibility test --- it will report hidden tabs, scrolled out of view, etc, as visible --- but if it works for you, I can swallow it. Just promise me that Flash will get fixed and we can take this out.

Comment 20

[Doug Turner (:dougt)]

I am also unhappy with the change and hope that the plugin can be fixed before they ship.

Comment 21

[Doug Turner (:dougt)]

http://hg.mozilla.org/mozilla-central/rev/d7b2ca236644

Comment 22

[Mike Beltzner [:beltzner, not reading bugmail]]

Comment on attachment 407933 [details] [diff] [review]
cleaned up patch -- minimum required.

a192=beltzner

Comment 23

[Doug Turner (:dougt)]

http://hg.mozilla.org/releases/mozilla-1.9.2/rev/295544af3bda