1215061 - [META] Better keyboard shortcut support

Status: NEW

(WebExtensions :: General, defect, P3)

Description

[Gabor Krizsanits (INACTIVE)]

We'd like to support Vimperator-type functionality.

Comment 1

[Nick Alexander :nalexander [he/him]]

(In reply to Gabor Krizsanits [:krizsa :gabor] from comment #0)
> We'd like to support Vimperator-type functionality.

Is this equivalent to "Support chrome.commands"?  (See https://developer.chrome.com/extensions/commands.)  If not, can you explain what might be different?

Comment 2

[Gabor Krizsanits (INACTIVE)]

(In reply to Nick Alexander :nalexander from comment #1)
> (In reply to Gabor Krizsanits [:krizsa :gabor] from comment #0)
> Is this equivalent to "Support chrome.commands"?  (See
> https://developer.chrome.com/extensions/commands.)  If not, can you explain
> what might be different?

It is not. Part of this bug is to figure out what are the missing features from chrome.commands. Obviously one huge draw back of chrome.commands are all the limitations it comes with. But Vimperator comes with extensibility as well, and I'm not sure their plugins can work with the current API which is manifest driven. So probably we will need something that can work dynamically and programmatically, supports more keys and in some cases even allows the default keybindings to be overwritten. Note that Vimperator is just one example, there are other popular extensions with rich keyboard shortcut story. The goal is to extend the chrome API or to design an alternative API if needed that makes it possible to support our already popular add-ons.

Comment 3

[Simon Lydell]

Another Vimperator-like extension is VimFx[1]. (I'm the main developer of VimFx.) The idea of VimFx is to provide the vanilla Firefox experience, with a bunch of Vim-like keyboard shortcuts on top.

The biggest problems with chrome.commands for both extensions are:

1. chrome.commands only support keyboard shortcuts with modifiers (ctrl, alt, ctrl+alt), while Vimperator’s and VimFx’s shortcuts are mostly modifier-less. For example, pressing "t" could open a new tab.

2. Both add-ons have _tons_ of shortcuts and allows the user to add their own. There must be great default shortcuts, and all of them must be customizable.

The way VimFx works is basically like this (pseudo-JavaScript code):

> forEveryChromeWindow(window => {
>   window.addEventListener('keydown', event => {
>     const matchingCommand = getMatchingCommand(
>       previouslyEnteredKeys, event.key, event.code
>     )
> 
>     if (
>       (isUIEvent(event) && isTextInputFocused(event)) ||
>       state.webpageContentTextInputHasFocus === true
>     ) {
>       // Usually, only Escape for blurring the current text input
>       // is allowed inside text inputs.
>       if (!isAllowedInTextInputs(matchingCommand.shortcut)) {
>         return // Allow typing in text inputs without triggering commands.
>       }
>     }
> 
>     if (matchingCommand) {
>       matchingCommand.run()
>       event.preventDefault()
>       event.stopPropagation()
>     }
>   }, true)
> )

matchingCommand.run() might do stuff like:

- Focusing or blurring the location bar
- Opening or closing the find bar
- Opening or closing a tab
- Switch to another tab
- Sending a message to a frame script for interaction with web page content

Note that another nice thing about dealing with keydown events in the UI process (rather than in frame scripts) is that the keyboard shortcuts remain responsive even if the content process is slow. Yay multi-process!

Chrome has the Vimium[1] extension, which is similar to Vimperator and VimFx. Vimium listens for keydown events in content scripts (there's nothing else available). People keep requesting the same features from Vimium, but there's nothing they can do, because the Chrome extension API does not allow them:

- Vimium can't focus or blur the location bar. Instead, they've built their own location bar. There are no APIs for dealing with the location bar. Even if they were, Vimium couldn't bind the Escape key to do the blurring when inside the location bar. You can be similarly "trapped" inside the dev tools, for instance.
- Vimium can't interact with the find bar. Instead, they've built their own.
- Vimium can't do anything on chrome:// tabs, new-tab-page tabs, or any tabs where Chrome decided not to allow extensions. This is especially bad when using the shortcut to focus the next tab. If you come across a disallowed tab, you get stuck there and can't move on (with Vimium shortcuts).

I'm interested in making VimFx a WebExtension. I'm fully aware that we might not be able to port _everything,_ and that's totally fine as long we can just get the basic features going. But I can't even get that far.

Here's an idea:

1. Provide a customizable key for blurring the current UI control and return focus to the page. VimFx users want this key to be Escape.
2. Provide a WebExtension API that lets the extension register a keydown handler callback, in a background script (that is, in the UI process).
3. Provide WebExtension APIs for doing basic interactions with the most common UI elements, such as focusing the location bar. This is a totally separate issue, so I won't mention it again below.

Details about the keydown handler callback:

- It is called on keydown, but Firefox chooses _which_ keydown events to provide.
- It is _not_ called when a UI element that can be typed into is focused, such as the location bar, some <select>-style dropdown or menus (because of access keys).
- It _is_ called _before_ Firefox's own keyboard shortcuts are handled, so that the extension gets the chance to override the default shortcuts.
- It _is_ called regardless of what page is in the current tab. That includes about: pages.
- It is called with a regular event object, but with event.target and suchlike nulled out, as to not expose the UI.
- event.preventDefault should prevent default as expected.
- event.stopPropagation should stop the event from reaching the content page.

Note: VimFx currently has a keyup handler as well. All that is done there is:

- event.preventDefault() and event.stopPropagation() if those two methods were run in the corresponding keydown event.
- Keeping track of the currently held down modifier keys, for a UI thing. Nothing critical.

I'm sure that if the keydown proposal works out, we'll find a way to make keyup handling into it as well.

[1]: https://github.com/akhodakivskiy/VimFx/
[2]: https://github.com/philc/vimium/

Comment 4

[Nick Alexander :nalexander [he/him]]

Simon: thanks so much for the detailing your use case(s).  That kind of breakdown is really useful when choosing what Firefox should support and when designing APIs.

andym, others: I want to make sure Simon's effort is incorporated into this work, whatever that means.  NI just to make sure you process it.

Comment 5

[Andy McKay]

Processed, thanks Simon.

Comment 6

[Andy McKay]

Marking as approved because I don't think anyone has any problems about this at a high level and it meets all the requirements for a new API https://wiki.mozilla.org/WebExtensions/NewAPIs.

Comment 7

[Simon Lydell]

I’m so glad to hear that! Feel free send me all sorts of real use case questions you might get while designing the API :)

Comment 8

[Simon Lydell]

A quote from the Mozilla support page for Firefox keyboard shortcuts[1]:

"Note: Keyboard shortcuts can be customized using the Menu Wizard[2] extension."

Being able to customize the keyboard shortcuts is important for users who find it easier to use the keyboard than a mouse.

When Firefox switches to WebExtensions only, one of the following needs to be done (in addition to updating that link):

- Remove support for customizing Firefox's keyboard shortcuts. (That would be sad.)
- Bake keyboard shortcut customization into Firefox.
- Allow WebExtensions to customize Firefox keyboard shortcuts.

For the case of VimFx (mentioned in an above comment), it'd be nice if VimFx could trigger the default keyboard shortcut actions manually when a user has typed a VimFx shortcut.

[1]: https://support.mozilla.org/en-US/kb/keyboard-shortcuts-perform-firefox-tasks-quickly
[2]: https://addons.mozilla.org/firefox/addon/s3menu-wizard/

Comment 9

[Andy McKay]

Simon, would you be interested in trying to implement this?

Comment 10

[Simon Lydell]

Yes, I would. I've never worked on the Firefox code base before, and I have never written C++, but I could still try.

However, I'm not really sure what "this" means yet.

- What is acceptable in the WebExtensions philosophy? I wouldn't like this feature to be abused, and to become a common way that add-ons slow down Firefox at every key stroke.
- What are the needs from other add-ons than VimFx?
- Should we just start experimenting somewhere to make some sort of demo and see where things end up?

Comment 11

[Andy McKay]

Depending exactly what you want to do you shouldn't need to write C++. But you can write an experiment completely in JS without having to compile Firefox. Some documentation and examples are here:

http://webextensions-experiments.readthedocs.io/en/latest/

If that doesn't work, diving into Firefox might work, and again shouldn't be too bad depending upon what you want to do. If you do get to that, I recommend starting here:

http://areweeveryoneyet.org/onramp/desktop.html

Figuring out what "this" is the slightly harder part and for that, prototyping or throwing together some drafts in a wiki, gist or your favourite documentation tool might help.

The WebExtensions philosophy is focused on security, privacy, stability and performance. It's true that some APIs can be abused if a developer really wants to make a bad experience. But that's where we'll be able to help.

Two other add-ons that do commands:

https://addons.mozilla.org/en-US/firefox/addon/pentadactyl/
https://addons.mozilla.org/en-US/firefox/addon/vimperator/

Comment 12

[Simon Lydell]

Thanks! I'll try making an experiment in a few weeks.

Comment 13

[Simon Lydell]

Since my last comment, I’ve spent the most time thinking about API design and I have gone through several iterations. In the last few days I’ve arrived at something that I like and I think should work, and I have written it down. I then started trying to implement a WebExtension Experiment, but got stuck at even getting the Hello World example running.

Everything is available here: https://github.com/lydell/webextension-keyboard

- I’d very much appreciate feedback on what I’ve written about the proposed APIs!
- Help with getting the Hello World WebExtension Experiment running correctly would be awesome! https://github.com/lydell/webextension-keyboard/issues/1
- Feel free to open issues on the above mentioned GitHub repository.

Comment 14

[Croydon]

I also need dynamically and programmatically key registering. Thanks for your work Simon.

Comment 15

[Simon Lydell]

I’ve updated the proposal after some feedback. Check it out!

Croydon, would you mind telling what you need this feature for? I’d like to know that the proposed APIs actually do what you need.

Comment 16

[Croydon]

According to your readme onKey.addListener, onKey.removeListener, onKey.hasListener and keyboard.onKey would do everything I need already :)

Comment 17

[wsha.code]

That looks great, Simon. I think it has the core functionality needed to enable useful keyboard driven addons.

Here is a collection of thougts I had. I like the simplicity of the current proposal, so I am okay with these being disregarded but I just raise them for the sake of consideration:

1. The proposal allows for single key event listeners and suggests building up multi-key shortcuts by having the addon track the state of keyboard input. Are you concerned about a user running more than one addon that uses this API at once? If one addon supports a two key shortcut "g r" and another supports an "r" shortcut, then pressing "g r" will trigger both since the first addon has no way to tell the second not to fire after the user presses "g". I guess because WebExtensions are isolated you will never be able to check globally for conflicts between shortcuts the way KeyConfig does.

2. I recently switched from Pentadactyl to VimFx and one thing I miss is the ability to assign non-modifier-key shortcuts for other addons. What about an API to send key events to the browser? browser.keyboardShortcutActions is proposed to handle this for builtin shortcuts, but for example to open the NoScript UI one has to press "C-S-s". One can change the particular combination in preferences but it has to use a modifier key. With Pentadactyl, I had this mapped to "n s". WebExtensions can't talk to each other directly in general, but if I could have an addon that catches "n s" and sends "C-S-s" to the browser that would be good enough.

3. One thing VimFx made me aware of is the GCLI (because I can't help hitting ":" coming from Pentadactyl). The GCLI seems pretty powerful, and I think some of Pentadactyl's features not available in a WebExtension could be done through custom GCLI commands (and I think the GCLI will still work after XUL is deprecated?). Having an API to send commands to the GCLI from a WebExtension would help with providing current Vimperator/Pentadactyl functionality because an addon could then map keyboard shortcuts to GCLI commands. (Or maybe this could kind of be done through 2 by for example mapping "s s" to the browser key events ":screenshort<Enter>" though that seems inefficient.

Comment 18

[Simon Lydell]

Thank you for you feedback, wsha.code!

1. No, I'm not at all worried about extension conflicts. Today, VimFx uses a very similar approach to the proposed `browser.keyboard.onKey` API, and it is very much possible that several extensions use the same approach as well. However, we've had zero issues regarding conflicts with other extensions when it comes to keyboard handling. I think that the only way conflicts can arise is if the user installs VimFx and Vimperator at the same time and expects that to work. Of course it won't, and people don't expect that in reality either. The `browser.keyboard` API is not for "mainstream consumption" by every other WebExtension. It's for extensions specifically dealing with keyboard shortcuts. I don't think it makes sense to install more than one such extension at a time. Other extensions will do just fine with the `browser.commands` API.

2. I think that Cross-extension messaging is the solution here: https://developer.chrome.com/extensions/messaging#external (also, this is a bit out of scope for this ticket)

3. Let's not talk about GCLI in this ticket. It is out of scope, and makes it harder to follow the comments.

Comment 19

[Simon Lydell]

The proposals are progressing nicely. I've now managed to implement part of the API as a WebExtension Experiment. Follow along here: https://github.com/lydell/webextension-keyboard/issues/1

Comment 20

[wsha.code]

I think your stance on conflicts is reasonable. It might be worth warning the user that using more than one extension with the browser.keyboard preference can cause conflicts when the extension permissions are presented at install time.

Regarding cross-extension messaging, it's nice in theory, but I think in practice it won't be useful. For it to be useful, extension authors would have to go out of their way to create a public API for other extensions to use which I doubt they would do just so their shortcuts could be remapped.

Regarding the GCLI, I think it is out of scope of what you want to port VimFx to a WebExtension, and a more in depth discussion could take place on a separate issue, but I think it is worth mentioning in this issue since this issue was started as "support Vimperator working as a WebExtension".

Comment 21

[Andy McKay]

Kris, as the author of pentadactyl, you know this area well. Any comment so far for Simon?

Comment 22

[Kris Maglione [:kmag]]

Sorry, I'm busy trying to get things ready for the merge next week. I'll have more time to comment after Monday.

Comment 23

[Simon Lydell]

TL;DR: Turns out not much at all needs to be done – for VimFx's purposes (I'm dropping all other ambitions). What's needed is: One bug fix, and one new shortcut for the Commands API. No new APIs.

1. Fix bug 1338909.
2. Add "_focus_content_area" to the Commands API.

How did I come to the above conclusions? Read on.

I've made two important realizations:

- My proposed new APIs feel too complicated and too niche.
- Using only what WebExtensions already provide (regular old 'keydown' listeners in content scripts) works just fine 90% of the time.

The other 10% is not enough to motivate a new API. All that is needed is an escape hatch.

It turns out that the big part of the escape hatch already exists: Browser action popups. Such a popup:

- Can be opened in _any_ tab (unlike content scripts).
- Can capture 'keydown' events (like any old web page), and do whatever it wants with them.
- Can be opened with the Commands API. Shortcuts added via the Commands API can
  be executed in _any_ tab, and inside _any_ UI element (such as the location bar or the devtools).

I intend to use VimFx's browser action popup as a way to run VimFx shortcuts in tabs where content scripts aren't allowed, or if the web page makes VimFx's keyboard shortcuts unresponsive.

> "commands": {
>   "_execute_browser_action": {
>     "suggested_key": {
>       "default": "Ctrl+E"
>     }
>   }
> }

"Can't run a VimFx command, for some reason? Hit Ctrl+E first to "escape" into VimFx's popup."

To make this Ctrl+E idea work in practice, bug 1338909 needs to be resolved. That's a small bug, though.

There's one thing that cannot be done from a popup, though: Focusing web page elements. For example, VimFx's `gi` shortcut focuses the first text input it finds in a page. That can't be done from a popup, because there's no way to move focus from the popup back to the page. Right now, that either requires clicking inside the web page with the mouse, or pressing Tab or F6 a couple of times an hoping for the best. WebExtensions basically live and breathe inside the content area of the browser, but there's currently no sane way of getting back to that "world" using the keyboard.

That's why I propose this:

> "commands": {
>   "_focus_content_area": {
>     "suggested_key": {
>       "default": "Ctrl+Shift+E"
>     }
>   }
> }

- If the content area is already focused it is a no-op.
- Otherwise, the currently focused UI control is blurred and focus is moved to the content area. This means that you can use the arrow keys to scroll it, press `/` to quick find, and that content scripts can receive 'keydown' events etc.

(I'm not totally sold on Ctrl+Shift+E as the default VimFx shortcut, but that doesn't matter. People will want to be able to change it regardless.)

I'

In summary:

1. Fixing bug 1338909 should be a no-brainer, especially since Chrome already works that way.
2. Regarding adding "_focus_content_area" to the Commands API – do you want me to open a new ticket for that?

Comment 24

[Kumar McMillan [:kumar]]

The auto-focusing bug that needs fixing is tracked here now (I resolve the other as a dupe): bug 1324255

Comment 25

[Kumar McMillan [:kumar]]

> commands: _focus_content_area

Couldn't you also implement this by listening for a custom command in a background script? When received in the background, it could send a message to a content script that does querySelector(...).focus()

Comment 26

[Simon Lydell]

Thanks for the idea for _focus_content_area, Kumar!

I tried it out, and couldn’t get the focus to move.

Here’s my test extension:
https://gist.github.com/lydell/a730c852fdb37f6d8f1f30968f3b591b

Here’s exactly what I did:

1. Went to google.com.
2. Focused the location bar.
3. Pressed Ctrl+Shift+E.

Results: The `<body>` and the google search input got red borders, but neither of them was focused.

(I also tried your workaround for getting a browser action popup focused, but it made no difference when I tested. You can try it with the same test extension, with the Ctrl+E shortcut.)

Ubuntu 16.04
Firefox Nightly 54.0a1 (2017-02-05) (64-bit)

Comment 27

[Sergio]

Just a quick question cause i was hopping this new way to handle shortcuts to resolve my "issue" and i may need to fill another request now.

What about a way to add Commands outside the manifest.json? that way we can provide the users a way to customize the addon shortcuts in case of collision or just commodity/necessity.

Comment 28

[Simon Lydell]

Sergio, it seems like you are looking for bug 1303384.

Kumar, I now understand _why_ using `.focus()` in  a content script won't move the focus. Imagine a website doing `setInterval(() => { someInput.focus() }, 100)`. If that could "steal" the focus from the location bar, the website would effectively prevent the user from using the location bar at all (in that tab). The same restriction holds for content scripts (and malicious WebExtensions). The proposed _focus_content_area command, on the other hand, can only be invoked by the user explicitly pressing some keys on the keyboard.

Andy, what do you think about the _focus_content_area proposal?

Comment 29

[Simon Lydell]

I've opened bug 1341416 for _focus_content_area.

Comment 30

[Col]

The API suggestions that Simon made earlier are a good fit for Vimperator, but the compromise with the browser action page is less suitable for that purpose.

What can I do to push this bug further? Do you want a concrete implementation or is an API definition sufficient (and then someone from Mozilla works on it, presumably)?

Comment 31

[robehickman]

I do think this API is needed. My experience with Chrome extensions like cVim is that keyboard input using page scripts is unreliable. Key presses randomly don't work and I must either click on the page or reload it to get it to work again. By comparison Vimperator has been 100% reliable over the 6 or so years I've been using it.

Comment 32

[Simon Lydell]

Here's food for thought:

Vim-style extensions are all about modifierless shortcuts.

Firefox already has them. For example, `Space` to scroll down and `/` to open Quick Find. Neither of those conflict with typing spaces and slashes into text inputs – neither in UI elements nor in web page elements. They work in `about:` pages as well as in web pages.

So I guess the Commands API could be extended to tap into that system.

One notable difference, though, is that `Space` and `/` are `event.preventDefault()`:able by the web page, while Vim-style extension shortcuts typically aren't.

Also, some web sites don't bother calling `event.preventDefault()` for keys that don't usually do anything when pressed, such as A–Z.

I guess it'd be possible to let modifierless shortcuts decide if they are preventable or not. Just keep in mind that such shortcuts might conflict with "fancy text editors" on web pages. Determining if the currently focused element is "editable" is not easy: https://github.com/akhodakivskiy/VimFx/blob/85b5ab7794814636e09a2c5f6034c69c7257d2d8/extension/lib/utils.coffee#L123-L136

Just brainstorming and leaving a comment here in case anyone wants to pick up the project of new keyboard APIs. :)

Comment 33

[Col]

An extension of the commands API that allowed modifierless shortcuts, preventDefault, and the dynamic registration of those shortcuts (or just permitted us to hook every possible shortcut) seems OK for most commands, but the situation is a bit weird if you want to accept an arbitrary string as an argument to a command.

e.g. I want users to be able to type 'afoo' and to be redirected as if they'd clicked a 'href="foo"' link. If I want to capture 'foo' I either need to register commands for all the characters I think my user will press or I need to move the focus of the user to some element that I can addEventListener to.

That's possible, but seems a bit weird to me. It will also break on pages where we're not allowed to have content scripts unless we define some other UI element (a toolbar or a browser action, perhaps).

I think it's simpler and purer to just have an API that permits adding an eventListener to the root FF document.

Running `document.addEventListener("keydown", cb)` in the browser console basically gives me what I want (and what is described in comment #2). 

    browser.keyboard.addEventListener = function (cb) {
	document.addEventListener("keydown", Sanitiser(cb))
    }

I imagine `Sanitiser` receiving the real event, wrapping it up in an Event-like object that hides the real event as a private variable and controls access with 'privileged' functions[0].

Douglas Crockford suggests that this is possible[0], but I'm not a seasoned enough JS hacker to know if there's some way around his 'privileged' functions or private variables.

[0]: http://www.crockford.com/javascript/private.html

Comment 34

[Col]

Looks like `document` in the browser console refers to the first window only, so something a bit more sophisticated might be required in the implementation of the API, but hopefully not too hard.

Firefox developers: I'd really like this API to exist and I'm willing to work on it, but I don't know what help you need or want from the community. Please let me know how I can help :)

Comment 35

[Kumar McMillan [:kumar]]

If you'd like to build a prototype for a new API, it is a well documented and pretty straight forward process (you won't need to build your own Firefox): https://webextensions-experiments.readthedocs.io/en/latest/

Comment 36

[Kumar McMillan [:kumar]]

Oh, sorry for repeating what was already said. I wasn't caught up with the whole thread.

Comment 37

[Col]

I'd like to work on a keyboard webextension API, but this bugzilla thread has died out and I haven't had a lot of luck on IRC. Would a mozillan, or someone else who knows about Firefox's internals be willing to chat to me a bit?

I especially want to know:
	1. If mozilla are still willing to include this API
	2. What security or other constraints I should work within
	3. Some technical stuff about event handling in Firefox.

I'd really like for someone to commit to meeting for half an hour or so on IRC or whatever other IM service suits. Mail me and we can arrange a time or whatever. I live on GMT/UTC, but stay up late.

Comment 38

[Johann Hofmann [:johannh]]

Colin, can't you salvage parts of Simon's WE Experiment? https://github.com/lydell/webextension-keyboard

> I live on GMT/UTC, but stay up late.

You might need to contact folks in your evening hours, when it's daytime in the US.

Andy or Caitlin, since this API has been approved, can you please make sure Colin is receiving the required help to move a WE Experiment forward?

Comment 39

[Andy McKay]

Thanks for the ping, Colin and Kris have already chatted, but we are having a work week this week. So Kris and Colin are plan on talking about it next. But, its easy for us to forget things, so please ping us if the meeting doesn't happen.

Comment 40

[Caitlin Neiman [:caitmuenster]]

It looks like this is in progress, so I'm going to cancel the needinfo. If anything else is needed, please let me know!

Comment 41

[randomreporter]

Addons like Tree Style Tab or Tree Tabs that allow to collapse and expand tab trees need an ability to replace original shortcuts like Ctrl-Tab and Ctrl-Shift Tab. Why? Imagine that's a vertical tab bar:
[Parent tab in the collapsed tab tree]
   [child collapsed tab 1 (hidden)]
   [child collapsed tab 2 (hidden)]
[Standalone tab]

When [Parent tab in the collapsed tab tree] has focus and you press Ctrl-Tab, focus should move to the next visible tab on the tab bar, and that is [Standalone tab]. However, Ctrl-Tab by default will move focus to the next available tab on the tab bar, and that is [child collapsed tab 1 (hidden)].

Tree Tabs developer wrote a post about his problem here: https://forum.vivaldi.net/post/134420

Comment 42

[Col]

I've written a first version of the API. Requesting a review of my work so far in case I need to do any course corrections :)

github URL: https://github.com/cmcaine/keyboard-api

The readme contains an up to date list of questions for reviewers, which I've copied below.

Thanks,

## Questions for reviewers

Hi, and thanks for reviewing! Here are some particular questions I would like answered, please also give comments on other topics :)

 - What is best practice for sharing code between api.js and the frame script?
 - Events are sent to at least the web console of a frame/window before or at the same time as they get processed by my frame script. How do I stop that? (If I call preventDefault() and stopPropagation() in my frame script, the event is still logged by the web console)
 - Are my conditions in chromeListener (:122) sensible?
 - Why does TAB not emit a keydown event?
 - I'm using a frame script to listen for the events, that will mean duplicating the state of the suppression state machines, unless I make a synchronous call to the chrome process and let that keep the state. Which is preferred?
 - Why does using sendSyncMessage in frame.js:log() cause more frame scripts to appear when I broadcast the "suppress" message?

Comment 43

[Col]

kmag, we chatted a bit on IRC last week and you said that you'd have a look at my code. Could you clarify when you might be able to do that? If you're not able to do it, could you delegate it to someone else?

There's only ~200 lines of code (excluding comments) and I mostly just want answers to some of my questions above and a simple "go/no-go" on whether I should continue as I have been.

Cheers,

Comment 44

[Col]

I've been in contact with :kmag a few times on IRC now, but they haven't been able to find the time to review my work. I understand that you're all busy, but it would be really useful to me if someone could briefly review the work. :aswan has suggested :mixedpuppy or :zombie, so I'm tagging them.

My repo: https://github.com/cmcaine/keyboard-api

Answers to three of my questions would be very useful and do not require anyone to look at my code. I reiterate just those three in order of importance to me:

 1. Events are sent to at least the web console of a frame/window before or at the same time as they get processed by my frame script. How do I stop that? (If I call preventDefault() and stopPropagation() in my frame script, the event is still logged by the web console)
 2. I'm using a frame script to listen for the events, that will mean duplicating the state of the suppression state machines, unless I make a synchronous call to the chrome process and let that keep the state. Which is preferred?
 3. What is best practice for sharing code between api.js and the frame script?

The most important code-related question is:

 1. My frame scripts (loaded with the global message manager) no longer seem to be loaded into "regular" pages: they're only loaded on about:* and moz-extension://* pages. This is a regression since I last tested the code (~July 10th)

I also note that Koushien has started work on the other part of this API, the keyboard shortcut or keyboard navigation part: https://github.com/Koushien/keyboard-shortcut-api

Comment 45

[Col]

This is getting silly. Can someone please let me know when or if I will be able to get this work reviewed?

I understand you're busy, but some idea of how long I will have to wait would be most welcome.

Comment 46

[Shane Caraveo (:mixedpuppy)]

Hi Colin,
Basically we're on the run to 57 and have limited time to address all items that must be in 57.  This is a big bug with a lot to digest, and looking at the code I don't see a simple explanatory example use of the api.

Without having an opinion on the api itself, or fully grasping how its used, there are a couple small things I see.  Such as using lazy loading of modules (search for XPCOMUtils.defineLazy on searchfox), using windowTracker in place of window mediator.

Comment 47

[Simon Lydell]

A random opinion from the developer of the (now dead) VimFx[1]:

I think it makes sense not trying to rush this into Firefox 57. There are currently two WebExtensions that offer Vim-style shortcuts:

- Vimium-FF[2]
- Saka Key[3]

While they're not the same as Vimperator or VimFx, they at least go 50%/70%/90% of the way, which is better than nothing.
Let's try them out for a while to identify the most important shortcomings and then propose APIs based on that.

¯\_(ツ)_/¯

[1]: https://github.com/akhodakivskiy/VimFx/
[2]: https://addons.mozilla.org/firefox/addon/vimium-ff/
[3]: https://addons.mozilla.org/firefox/addon/saka-key/

Comment 48

[wsha.code]

Simon, I am confused by your suggestion that we need to try out WebExtensions to see what is missing. We already have a lot of experience with what is possible with WebExtensions from the history of Chrome extensions. Above you highlighted several features that have been requested for Vimium and that can not be implemented with the Chrome extensions API. The goal of Colin's current proposal is to allow keyboard input to be processed in contexts currently inaccessible to WebExtensions (basically the same as your last proposal) to allow these features to be implemented in WebExtensions.

I agree that there is no need to rush Colin's API. The features that it implements are not in the current WebExtensions API in part because of the potential security and performance issues they could cause. That said, he is just asking for feedback, not for something to be merged into Firefox.

Also, I have been testing Vimium-FF and Saka Key. One issue with using them to assess the shortcomings of WebExtensions is that they are both new and rough right now. For example, Vimium-FF can't open a link in a new tab or copy text to the clipboard. Saka Key has some performance issues and has trouble focusing input fields on some sites. They are both promising directions, but it might be better to look at the Crhome extensions like Vimium, cVim, Surfingkeys, etc. to assess what is possible.

Comment 49

[Col]

Thanks :mixedpuppy for having a look at my code. I'll try and make the changes you've suggested. :)

@Simon

We already know the limitations of the existing webext APIs in this space. From the readme on my project: "This API allows callers to listen to keydown events that occur anywhere in the browser, including in the two places that content scripts cannot be injected: the browser chrome and on restricted frames (about:*, addons.mozilla.org)."

Without this kind of API (combined with the keyboard-shortcut-api), these projects will not work in some tabs and in some GUI elements of Firefox, and that's an important usability regression.

We also get complaints that content scripts don't load fast enough and so keyboard shortcuts sometimes don't work as pages are changing.

Apologies for the brevity. Bit busy today.

Comment 50

[Andy McKay]

Sorry for the delays Colin, things are getting very hectic with the Firefox 57 merge looming. We've redirected this to Shane who's commented on this and hopefully has more opportunity to help.

Comment 51

[koushien]

Greetings everyone, as Colin mentioned above I've been working on the keyboard shortcut API found at https://github.com/Koushien/keyboard-shortcut-api and feel it is ready for review. 

The methods of this API perform actions possible with keyboard presses in Firefox. I leave comments on where I inferred that the calls used are actually the calls made by real keypress activation. The source is fast changing, but they should land you in the vicinity. I'm not sure what else to mention here, so hopefully the README covers most other things.

Thanks to anyone who can take a look and provide feedback.

Comment 52

[Col]

Thanks Andy :)

Shane, just adding you to needinfo. If you could review Koushien's keyboard-shortcut-api, that would be very useful to us.

It's the other half of lydell's original proposal and where my API handles receiving keyevents from places that are not currently permitted, the keyboard-shortcut-api performs tasks that are possible with Firefox's built-in keyboard shortcuts, but not possible with the webextension or web APIs at all or on all pages*.

Being able to call these functions programmatically is obviously very important if one wishes to implement an alternative UI for Firefox.

Finaly, we would like some function to focus the content area. I note that this has been requested before (Bug 1341416), and possibly discussed in the March 7th WebExt triage meeting, but no resolution was reported.

* The API does currently replicate some functionality that already exists in the tab, window, and history WebExt APIs. We're not sure if it should.

The API also provides some functionality (in navigation* page*, edit*) that is similar or identical to existing Web APIs. We provide that functionality here mainly because we can't load content scripts into all pages. But I also note that page scrolling by window.scrollBy doesn't use smooth scrolling and that navigation functions are usefully called before content scripts load.

Comment 53

[Col]

Correction: The shortcut API doesn't replicate any functionality in the history webext API. The point of similarity is that both allow you to clear recent history, but the function in this API triggers the firefox GUI for it, whereas the history API does not.

Comment 54

[Col]

@cnieman,

Could you add this bug to the September Triage meeting agenda?

Comment 55

[Caitlin Neiman [:caitmuenster]]

Awesome. This is added to the agenda for the September 26 meeting. Colin and koushien, would you be able to attend? 

Wiki: https://wiki.mozilla.org/Add-ons/Contribute/Triage

Agenda: https://docs.google.com/document/d/1pw5y-GHwDLPV9bYK4HWCiZtslqFtAeL3G9bC4ZDbdjs/edit#

Comment 56

[Col]

Thanks! I'll be able to. Can't speak for Koushien, but I'm sure they'll answer in a bit.

Comment 57

[koushien]

I should be able to, yes. Will joining the #addons channel on IRC be sufficient?

Comment 58

[Caitlin Neiman [:caitmuenster]]

Awesome! Yes, joining the #addons channel in IRC will be sufficient. :) See you on the 26th!

Comment 59

[eejdoowad]

Not sure if this is the right place, but I'm using standard event listeners in Saka Key, and it would be nice if there were a manifest permission you could declare to make event.preventDefault() work for all keyboard events. This is analogous to the "clipboardRead" permission you can declare to use document.execCommand("Paste").

Comment 60

[Col]

It's a meta bug now, so maybe?

Anyway, the issue as I understand it is that some of the keys that you want to preventDefault (e.g. <C-w>, <C-Tab>) are acted upon by the chrome window before they get to the tab window, so it's not really a case of flicking a switch and allowing preventDefault to work in content scripts.

Suppression (preventDefault and/or stopPropagation) turn out to be kind of tricky things to do in multiprocess firefox because chrome code cannot talk to webextension, frame or content code synchronously. So you have to tell the chrome code whether to suppress an event before the chrome code sees it.

See this issue for more information on my planned approach and please comment there if you have any problems/suggestions: https://github.com/lydell/webextension-keyboard/issues/1#issuecomment-292230434

Comment 61

[Tomislav Jovanovic :zombie]

I'm not familiar enough with these aspects to provide wide, architectural-level feedback here, but I'll be happy to review concrete patches after "the thing we want to do" is more concrete and agreed upon.

Comment 62

[Jens]

FWIW, this also seems to block Tree Style Tab from adding any kind of shortcuts support:

https://github.com/piroor/treestyletab/issues/669#issuecomment-343108872

user (me):
>> I am looking for a shortcut for "Close the current and all descendant tabs". 
>> ATM I have to move mouse to tab => right click on the tab => Tab Tree Style => "Close this tree"

author of TST:
> TST 2.0.x doesn't provide any shortcut except F1 to toggle the sidebar itself.
> Sorry but I have no plan to do around more keyboard shortcuts until better
> WebExtensions APIs are landed. See also: https://bugzilla.mozilla.org/show_bug.cgi?id=1215061

So I hope with FF 57 being out in a week (big congrats!!!) maybe this can rise in term to P2 from P3 ;)

Comment 63

[robehickman]

From my perspective some of the 'for security' justifications for lack of API's have no grounding. For example not allowing extensions to load on "https://addons.mozilla.org/". The only thing I can see an addon being able to do on this site is possibly install other addons without the users permission. However a script would trigger this by initiating a click event as if the user had done so themselves. Surely this would create a 'do you want to install' notification? Thus there is actually no security benefit to doing this. Frankly I would be far more concerned about having add-ons loaded on Internet banking sites as a compromised addon could do vastly more damage.

Under the circumstances I'm inclined to blacklist "https://addons.mozilla.org/" in my hosts file so it cannot load and thus cannot break by keyboard-centric user interface. This inconsistent behavior is extremely annoying and a awful user experience.

I don't use chrome because I strongly dislike the lack of addon flexibility it has. Firefox in copying this has seriously degraded it's viability. Currently we seem to be in a market of lousy web browsers and no alternatives. Things like qutebrowser don't count as they lack addons like lastpass. If you want to stand out in a saturated market the key is doing something different, providing features that users are dissatisfied with from the competition. The key to firefox at least for me is just how customizable it was.

Comment 64

[Sam Bull]

Also as a side note for the commands API, the manifest uses "suggested_key", which implies that this shortcut could be changed. It would be great if Firefox added a feature to allow users to customise these shortcuts, e.g. as an additional section of extension's preferences.

Comment 65

[Stephan Sokolow]

(In reply to Sam Bull from comment #64)
> Also as a side note for the commands API, the manifest uses "suggested_key",
> which implies that this shortcut could be changed. It would be great if
> Firefox added a feature to allow users to customise these shortcuts, e.g. as
> an additional section of extension's preferences.

I'd prefer the approach taken by KDE's global hotkey system where all of the shortcuts are displayed in a single, unified interface so that it's easy to see how they fit together into a consistent scheme.

(It would also have the advantage that it'd be possible to treat browser built-in shortcuts as "just another extension" so they could be customized too.)

Comment 66

[Mark Striemer [:mstriemer]]

There is work being done to allow an extension to change the shortcut for its own command in bug 1421811. This is the first bug that will work towards better support for shortcuts and commands.

After that bug has landed and the UX has been finalized we will add support for a user to change any extension's command shortcuts in bug 1303384.

We are currently discussing if we will be able to provide a UI to list and remap all of Firefox's shortcuts. This seems like something that could be very beneficial but we will need to loop in some other teams since it is out of scope for add-ons only work.

Comment 67

[Marc Auslander]

As a user who is keyboard-centric the loss of the ability to set global shortcuts is a major reason I'm still running ESR - and that won't last.  Granted Chrome is not much better - but firefox should lead in customizability!

One example.  Tab close is ctrl-w - but esc is IMHO a much better alternative.  I can no longer make esc a useful tab close in the current firefox.

Please please please.

Comment 68

[sgl4kn]

Have the same issue with ctrl+w/alt+4. I used to close tabs via the thumb button of my mouse (which actually sends alt+f4). But I can no longer assign "close tab" to alt+f4. And I understand that this is also not fixed by https://bugzilla.mozilla.org/show_bug.cgi?id=1421811, so https://addons.mozilla.org/de/firefox/addon/shortkeys/?src=search is not able to change it. 

So either https://bugzilla.mozilla.org/show_bug.cgi?id=1421811 should be adjusted to be able to change ctrl+w/alt+f4 or https://bugzilla.mozilla.org/show_bug.cgi?id=1303384 should be prioritized.

Comment 69

[Akkana Peck]

I have a different issue with ctrl-W: I want to disable closing tabs with it because it does Delete Word in every other app I use, and even in Firefox in the urlbar. So I'm typing along, entering my bug information or whatever, hit a ctrl-w to delete the last word and the whole tab disappears along with everything I've typed so far. I haven't found any reliable way to fix this in recent firefox versions except for compiling Firefox from source. Modifying browser/omni.ja works sometimes but not reliably. I don't need all the complicated vimperator stuff, just a way to override one compiled-in key binding.

Comment 70

[Stephan Sokolow]

As a workaround, have you considered a simple userscript that does something like this for every page the browser loads:

1. On first keypress or change event in an <input> or <textarea>, set a beforeunload handler and unregister the event handler that triggered.

2. On the clicking of any link or the submitting of any form, unset the beforeunload handler.

That'd give you an "Are you sure?" popup... at least until the Firefox developers get around to implementing their plans to forbid content-side "Are you sure?" popups in bug 578828.

Comment 71

[Hades]

Depends on: https://bugzilla.mozilla.org/show_bug.cgi?id=691127

Comment 73

[Sergio]

As it seems some bugs are being closed cause "there has been very little interest" on them i want to confirm that i'm still interested on this bug.

Comment 74

[Pranav bhattarai]

Why not let user control & change entire browser shortcuts?
Why not user vimify the entire browser experience?

That would have been really efficient way to use the browser.
Full control in user hand.

This is will certainly increase the speed of how the user interract with the browser.

Comment 77

[Kevin Cox [@kevincox:matrix.org]]

Another major issue I have is that Firefox allows websites to prevent some shortcuts. For example on some websites I can't open the history pane! We should be able to decide which shortcuts are preventable and which always work.

Comment 78

[Yuri Khan]

(In reply to Kevin Cox [:kevincox] from comment #77)

Another major issue I have is that Firefox allows websites to prevent some shortcuts. For example on some websites I can't open the history pane! We should be able to decide which shortcuts are preventable and which always work.

+1 to the above. One major offender is a code editing component, Ace Editor, that binds Ctrl+L to its “go to line number”. Ctrl+L is and should always be “focus the address bar”.

Comment 79

[erosman [:erosman]]

+1 to comment #77)

I use Firefox default keyboard shortcuts such as F12, Ctrl+Shift+h , Ctrl+Shift+j etc often and nowadays there are far too many sites that interfere with the normal browser experience.