348279 - Design spec/API for keyboard interaction with plugins

Status: RESOLVED FIXED

(Core :: DOM: UI Events & Focus Handling, defect)

Description

[Aaron Leventhal]

Spunoff from bug 93149 and bug 78414.

We need to start moving forward on fixing those bugs. A solid goal is to have an API that can be implemented both by plugins and Mozilla that will solve the problem.

Comment 1

[Oliver Yeoh]

Attachment: Documentation

Comment 2

[Oliver Yeoh]

Attachment: Patch v1

Comment 3

[Oliver Yeoh]

Attachment: Sample plugins

Sample plugins which should go into the plugin SDK

Comment 4

[Aaron Leventhal]

Is there anything we can do to let users get out of plugins which take focus but don't support tabbing integration or forwarding unused keystrokes?

For example, I go to nba.com and click on one of the tabs "Yesterday" or "Today".
I suppose the good thing is that keyboard users can't get in there in the first place, so can't really get in trouble. Or, can the plugin steal focus when in initializes itself. I guess that's not common.

Comment 5

[Oliver Yeoh]

(In reply to comment #4)
> Is there anything we can do to let users get out of plugins which take focus
> but don't support tabbing integration or forwarding unused keystrokes?
> 
> For example, I go to nba.com and click on one of the tabs "Yesterday" or
> "Today".
> I suppose the good thing is that keyboard users can't get in there in the first
> place, so can't really get in trouble. Or, can the plugin steal focus when in
> initializes itself. I guess that's not common.
> 

Yes, I think its possible. We are hooked or able to hook into the plugin's native window, so it is possible to look for a certain keystroke and take focus away from the plugin.

One possible issue is when the plugin contains native child windows. We would need to hook into all the plugin's child windows to filter the keypress events. I'm not sure if this is feasible or what the performance impact would be like.

Comment 6

[Deneb Meketa]

Oliver's proposal seems basically sane, but I have several questions:

- NPPVpluginSupportsFocusBool: the docs say "set to true if it can be focused by keyboard navigation".  This seems ambiguous.  A value of false could mean two things: (1) I am a new plugin, and I have nothing focusable in me; (2) I am an old plugin, and I don't understand this variable.  Is it acceptable for the browser to behave identically in these two situations?

- There are two ways in which a plugin can be notified that it is receiving focus: it can get a high-level focus event, or it can get a low-level mouse or keyboard event.  The spec should make it very clear, in every situation, which of these will be used (both if applicable), and in what order.  For example, when a user tabs into a plugin, does the plugin get only GetFocusFirstEvent, or does it also get the keyboard event afterwards (or beforehand)?  I'm in favor of delivering both, because this gives the plugin the most information; we could specify that GetFocusFirstEvent and GetFocusLastEvent is always immediately followed by the underlying event.  Also, what happens when a user clicks in a plugin with the mouse?  (This may not have to change from current behavior, but we should at least specify what the behavior is.)  Does the plugin get GetFocus followed by the mouse event?  Or just the mouse event?  Also, in all of these sequences, who has responsibility for actually setting OS focus to the plugin - does the browser do this, or does the plugin, and when?  The answer to this last question is likely to differ per platform.

- On the Mac (and in windowless mode), where plugins don't really have focus and all events come from the browser, how is tabbing out handled?  Do plugins just return false from NPP_HandleEvent for the tab keystroke event?  This is alluded to in the "Forward unused keystrokes" section, but it seems like it should also be mentioned in the "Tabbing out of the plugin" section.

- Regarding the discussion above... I'm not sure it seems like a good idea to hook window procs.  This only works on Windows, and it tends to create crashes and bug dependencies.  I would argue that accessible behavior can be expected only when the browser and a plugin are both coded to the new API.  Otherwise it seems like madness for the browser to try and work around a plugin that doesn't know how to behave.

Comment 7

[Oliver Yeoh]

(In reply to comment #6)
> - NPPVpluginSupportsFocusBool: the docs say "set to true if it can be focused
> by keyboard navigation".  This seems ambiguous.  A value of false could mean
> two things: (1) I am a new plugin, and I have nothing focusable in me; (2) I am
> an old plugin, and I don't understand this variable.  Is it acceptable for the
> browser to behave identically in these two situations?
> 
I think the browser should behave the same in the sense that neither of these plugins can be tabbed to. I could be wrong... Are there any use cases for differentiating these two types of plugin?

One thing to note though is that regardless of the NPPVpluginSupportsFocusBool value, the browser will still process any keystrokes forwarded from the plugin. 


> - There are two ways in which a plugin can be notified that it is receiving
> focus: it can get a high-level focus event, or it can get a low-level mouse or
> keyboard event.  The spec should make it very clear, in every situation, which
> of these will be used (both if applicable), and in what order.  
>
Agreed. I'll clarify this in the next revision of the specs.


> For example,
> when a user tabs into a plugin, does the plugin get only GetFocusFirstEvent, or
> does it also get the keyboard event afterwards (or beforehand)?  I'm in favor
> of delivering both, because this gives the plugin the most information; we
> could specify that GetFocusFirstEvent and GetFocusLastEvent is always
> immediately followed by the underlying event.
>
When tabbed to, the plugin will receive the native GetFocus event followed by the GetFocusFirst/LastEvent. 

The underlying keyboard event will be dropped... and I think it should be dropped because:
1) That key event took place outside of the plugin
2) Sending that key event might confuse the plugin into advancing its internal focus.

As you mentioned, the browser could send the key event earlier - before the GetFocus event. But I think that would add complexity to the plugin as it needs to know when a key event is bogus and when it should be processed. Any examples of how the underlying key event might be useful?


> Also, what happens when a user
> clicks in a plugin with the mouse?  (This may not have to change from current
> behavior, but we should at least specify what the behavior is.)  Does the
> plugin get GetFocus followed by the mouse event?  Or just the mouse event?
>
Indeed, this behavior is unchanged. We could certainly discuss it and explore if we could make mouse focus handling consistent with what we are doing here and vice versa. However, if there are non-trivial issues with the current mouse focus handling, I think separate bugs should to be filed :(

 
> Also, in all of these sequences, who has responsibility for actually setting OS
> focus to the plugin - does the browser do this, or does the plugin, and when? 
> The answer to this last question is likely to differ per platform.
> 
For keyboard:
a) tabbing into plugin - browser sets focus to the plugin then issues GetFocusFirst/Last
b) tabbing out of plugin - browser sets focus to itself
c) handling keystrokes from plugin - browser sets focus to itself if necessary 

For mouse-clicks on plugin:
a) mac and winless windows - browser sets focus to the plugin
b) linux, xembed and windows - plugin sets focus to itself


> - On the Mac (and in windowless mode), where plugins don't really have focus
> and all events come from the browser, how is tabbing out handled?  Do plugins
> just return false from NPP_HandleEvent for the tab keystroke event?  This is
> alluded to in the "Forward unused keystrokes" section, but it seems like it
> should also be mentioned in the "Tabbing out of the plugin" section.
> 
Yes, for Mac and windowless mode, the plugin simply returns false for the tab and shift-tab keystrokes. Will add this bit in the next specs update.


> - Regarding the discussion above... I'm not sure it seems like a good idea to
> hook window procs.  This only works on Windows, and it tends to create crashes
> and bug dependencies.  I would argue that accessible behavior can be expected
> only when the browser and a plugin are both coded to the new API.  Otherwise it
> seems like madness for the browser to try and work around a plugin that doesn't
> know how to behave.
> 
I see what you mean. I propose that we add NPN_HandleEvent. Plugins, except for Mac and windowless, can forward keys to the browser via this new API. 

One possible issue I see is that the NPEvent structure for windows is not great. Unlike XEvent and EventRecord, there are no bits to store the modifier keys.


These are great questions. Thanks for the feedback. With 5 different ways of creating plugins (and from what I heard one more in the form of Linux WMODE), I am sure there will be more.

I really want to get as much feedback as possible and revise the specs until we get it right. Nevertheless, I'm still hoping to get this done for Firefox 3 :)

Comment 8

[Oliver Yeoh]

Comment on attachment 257820 [details] [diff] [review]
Patch v1

Taking patch off review for now.

Comment 9

[Wayne Mery (:wsmwk)]

(In reply to comment #7)
> I see what you mean. I propose that we add NPN_HandleEvent. Plugins, except for
> Mac and windowless, can forward keys to the browser via this new API. 
> 
> One possible issue I see is that the NPEvent structure for windows is not
> great. Unlike XEvent and EventRecord, there are no bits to store the modifier
> keys.
> 
> These are great questions. Thanks for the feedback. With 5 different ways of
> creating plugins (and from what I heard one more in the form of Linux WMODE), I
> am sure there will be more.
> 
> I really want to get as much feedback as possible and revise the specs until we
> get it right. Nevertheless, I'm still hoping to get this done for Firefox 3 :)

if this is to get into FF3, is there an estimated deadline that must be met, eg will this need to be completed before 1.9 goes beta? 

Comment 10

[Aaron Leventhal]

Status report:
Work is stalled as the developer who was working on it no longer responds to pings. This is unfortunate but at this point we don't know of an engineer with available cycles. Anyone willing to try to take Oliver Yeoh's work in this bug to completion?

Comment 11

[Mats Ahlgren]

I would like to echo that the suggestion posted by Aaron Leventhal on 2002-09-09 17:08:40 PDT in bug 93149 seems like "the right way" -- namely, at the very least whatever you do, a certain set of prominent modifier+keys combos (e.g. tab-cycling, moving to the next form element, etc.) should be untrappable and unclaimable by plugins.

The rationale is thus: 1) You don't want to confuse users. 2) You don't want to trust 3rd-party code not to bug-up your core functionality. 3) You want to be accessible, which relies on #2. (For example, it's very annoying with alt-F to open the file menu stops working, because some webpage claimed an accessability key or trapped a keypress -- no offense, but letting untrusted webpage and plugins have full control of input events (without user consent) is a bad design model that must be fixed)

Comment 12

[Deneb Meketa]

I understand Mats' (and Aaron's) point about reserving certain key combinations for the browser, and, as a Flash Player developer, I wouldn't scream too loudly if this kind of restriction was put in place.

However, I'd like to point out the other side of the argument. If a plugin is well-written, and it has focus, then it's in a better position than the browser to assign key bindings, because its context is more specific than the browser's. For example, if on Windows Alt+F is being reserved for the File menu, and a Flash game has mapped Alt to "move left" and F to "fire", then a user pressing both at the same time would unintentionally trigger the File menu. That's a silly example, but there are millions of Flash movies out there, and surely some of them will conflict with the browser's key bindings. What makes this even harder for Flash developers is that the set of reserved bindings would probably be different on each platform - so Flash's premise of platform-independent execution would be compromised, and developers would have to test their key bindings on every platform in every browser. And what if the key bindings are user-configurable?

I am wondering if there is a middle ground, whereby plugins are free to do what they want, but there is a well-maintained list of default key bindings for each platform, which plugins are encouraged to avoid stepping on gratuitously.  If a plugin does step on one without good cause, well, it's a bad plugin, just like any other bad plugin, which could, after all, delete everything on your hard drive.

BTW, I hope it goes without saying that a plugin should never be able to capture any keystrokes if it doesn't have focus.

Comment 13

[Mike Schroepfer]

Mats can you take a look at this in the 1.9.1 cycle?

Comment 14

[Mats Ahlgren]

Hello Mike. I'd like to help, but I'm unfamiliar with Mozilla's design cycle paradigm. Please elaborate what you mean? (Are you asking me to temporarily downgrade and check in v1.9.1? -- I'm checking because such seems counterintuitive to the notion developing for the future, and perhaps a hassle without a virtual machine.)

Comment 15

[Ryan VanderMeulen [:RyanVM]]

He's asking if you can work on a fix for this bug in the Gecko 1.9.1 time frame. In other words, after the Gecko 1.9 cycle we're currently in (leading up to Firefox 3.0).

Comment 16

[:Gavin Sharp [email: [e-mail]]]

I'm pretty sure Schrep's comment was directed to the other Mats on the CC list - Mats Palmgren.

Comment 17

[danielle.pham]

Attachment: Modified Specs

I took the liberty to modified the original specs proposed by Oliver Yeoh. This modification took into consideration comments from various people and for my own needs (Java Plugin).
Please review and comment.

Comment 18

[danielle.pham]

Is the new owner for this been confirmed yet?
If needed, I'm interested to work with the owner to get this implemented for 1.9.1.

Comment 19

[Benjamin Hawkes-Lewis]

When Danielle Pham's draft says that on the Mac, "The plugin should simply return false when processing the keyDown event in the NPP_HandleEvent() function", does that mean that there would be no passthrough of key presses on the Mac? And if so, why is that a good idea?

Comment 20

[Jeremy Maitin-Shepard]

I think it is important that Mozilla always have the first crack at every key event, even when a plugin has focus.  This principle is important for ensuring that DHTML apps and plugins behave the same with respect to capturing and non-capturing keyboard event listeners.  As far as I understand, Firefox does not define many (any?) capturing keyboard event listeners, and consequently this proposal would not mean any real change in behavior compared to the existing proposal.  This proposal is, however, critically important for keyboard-driven front-ends like Conkeror and Vimperator that rely on very robust key bindings.  (In order to allow key bindings that would otherwise be used by the chrome UI to be passed through to a DHTML app, Vimperator provides a key binding to temporarily forward all keys (except e.g. escape which disables this mode) to the content page, and also provides a similar binding to forward only the next key event to the content page.)

The proposal is as follows.  I am presenting it in terms of the behavior, rather than in terms of specific implementation details like which piece of code is responsible for doing each action.  All keyboard events received while the plug-in is focused should still go through the normal propagation phase before the plugin does its normal processing of the key event.  If it reaches the plugin element itself without the propagation being stopped by a capturing event handler, then the event is forwarded to plugin code which then can perform its normal processing; after handling the event, the plugin returns some flag indicating whether it would like to disable bubbling of the event.  If bubbling is not disabled, the event then bubbles back up from the plugin element as normal.

I can see that this may present some implementation difficulties due to the fact that the DOM key event model (of keydown, keypress, and keyup) may not map all that exactly to the platform event model, but I think that those can be worked out.  I think, however, that it is certainly the right approach.

Note that presumably this approach requires more cooperation from the plugin in the case of windowed plugins on X11; windowless plugins should require considerably less cooperation from the plugin.

It is important, however, that however the API is defined, it be required, rather than "recommended", that plugins cooperate properly with mozilla with respect to key bindings, and that this cooperation not be specific to particular keys, like "Tab".  I can see that the current proposal's tab focus movement integration is useful, but to support chrome UIs that want to provide more substantial and robust keyboard navigation, it is important to allow capturing event handlers to override the plugin.

Comment 21

[Simon Bünzli]

For an alternative different proposal, see <https://bugzilla.mozilla.org/show_bug.cgi?id=78414#c236>.

Comment 22

[J]

add preference(s) to protect a (set of) hotkey(s) from being intercepted by plugins

true:  CTRL+T always functions as expected
false: enjoy another opportunity to practice patience

Comment 23

[Josh Aas]

There is a specification for this here:

https://wiki.mozilla.org/Plugins:AdvancedKeyHandling

Comment 24

[Dusan Maliarik]

There seems to be very little activity around this issue, but it's HUGE! It is a serious accessibility (and usability) issue. It completely kills (already poor) keyboard only navigation

Comment 25

[Dusan Maliarik]

IMHO, any spec / API is not necessary. We simply need to intercept some subset of keys, I would like to see comeback of functionality into ESC key, which should take back focus to browser, let Tab alone, we want to navigate inside plugin too, but there should be way to get back.

Comment 26

[Josh Aas]

Since this bug is about designing a spec, closing it out with the solution being:

https://wiki.mozilla.org/Plugins:AdvancedKeyHandling

Comment 27

[Firefox Bug Husbandry Bot]

https://bugzilla.mozilla.org/show_bug.cgi?id=1527381