1902858 - Add new APIs to felicitate autocorrection, autocompletion and other assistive features

Status: RESOLVED WONTFIX

(WebExtensions :: Frontend, enhancement)

Description

[Teal Dulcet [:tdulcet] (TB Council)]

Please consider adding one or more new privileged WebExtension APIs to felicitate implementing autocorrection, autocompletion and other related assistive features on all websites. Currently, this is not possible to implement these reliably from a content script without conflicting with the JS on many pages and thus breaking those pages for the user. The available Web APIs are also very limited in this regard. For example, it is not currently possible to get the caret position in contentEditable elements with respect to the innerText (see Mozilla Discourse), which is needed to robustly implement autocorrect. It also requires using the document.execCommand API, which is deprecated.

These new WebExtension APIs should provide full access to the any editable elements on the page, similar to how the spellchecker works. The proposed APIs could potentially also be used by grammar and style checker add-ons. Extensions should be able to react to any JS events before the page does and be able to easily manipulate the text both before and after the caret as the user is typing, without any interference from the page or performance degradation. Direct access to this interface in the Mozilla editor would be helpful. For reference, Microsoft previously proposed an API that could also be useful if made available to extensions.

For an example of the limitations of the existing APIs, try our Unicodify add-on, enable the experimental "Unicode autocorrect" feature on the options page and then try typing on various websites.

Comment 1

[Rob Wu [:robwu]]

The request here seems motivated by the shortcomings of the web platform APIs, along with the remark that there are efforts to improve the web platform APIs.

I don't expect us to have time to design let along implement extension APIs to support the use cases that you're asking for. Improving the web platform APIs seems like a more reasonable path forwards.

Comment 2

[Teal Dulcet [:tdulcet] (TB Council)]

I do not believe these use cases could ever be served solely by the Web APIs, as it would need to be privileged in order to avoid conflicting/interfering with any JS on the page. Also note that to my knowledge, there are currently no efforts to improve the relevant Web APIs. That proposed API I mentioned above was withdrawn by Microsoft in 2023.

Please consider reopening this bug so we can find a path forward for these important WebExtension use cases. I would also be happy to discuss this with you at MozWeek or at part of the WECG.

Comment 3

[Rob Wu [:robwu]]

I wonder what API you have in mind that would not degrade performance. Input is very sensitive to performance degradation when there are tasks blocking the processing of input events. Even the internal interface that you are referencing will impact responsiveness, e.g. as seen at https://searchfox.org/mozilla-central/rev/a26e972e97fbec860400d80df625193f4c88d64e/editor/libeditor/EditorBase.cpp#2733

Extensions are hosted in a separate process, and requiring process hops and waiting for the extension to process any events is going to result in an input delay due to the required asynchronicity. In theory, a content script can process input before the page is notified, but that is already possible today and I wonder what kind of APIs you are looking for that are necessary for your use case.

By "privileged", do you mean the privileges of regular extensions, or privileges of privileged (e.g. built-in) extensions?

Comment 4

[Teal Dulcet [:tdulcet] (TB Council)]

Thanks Rob for your response. I do not believe this would degrade the performance of inputs any more than the blocking webRequest API degrades the performance of network requests for example. Extensions can already significantly degrade the responsiveness of pages when typing by using any of the available JS input events (such as beforeinput) in a content script, which of course block until the provided callback function returns.

By "privileged", I just meant given a higher priority than the webpage's own scripts. Content scripts are isolated from webpages (with exceptions), but they are not currently given a higher priority. The JS events are registered on a first come first serve basis, which introduces a race condition and makes it impossible to insure that the WebExtension's code will always run before a page's scripts. The pages can also see all of the modification made by extensions and then attempt to make their own conflicting changes or otherwise interfere with the functionality of the add-on. Ideally, these new WebExtension APIs would allow making changes to editable elements that are invisible (or largely invisible) to the page's scripts, similar to the built in spellchecker or the new built in Firefox Translate feature.