1835790 - Native messaging regression in 114 (beta/rc)?

Status: VERIFIED FIXED

(WebExtensions :: General, defect, P1)

Description

[KeeForm]

Steps to reproduce:

My extension uses native messaging. After installing the extension and native messaging application on Windows 10 (https://keeform.org/keepass2/keeform-for-firefox), the configuration looks like this in file org.keeform.host.firefox.json:

{
"allowed_extensions": [
"[e-mail]"
],
"description": "KeeForm native messaging host",
"name": "org.keeform.host",
"path": "C:\Users\keeform\AppData\Local\KeeForm\keeform_host.exe",
"type": "stdio"
}

with registry key

[HKEY_CURRENT_USER\SOFTWARE\Mozilla\NativeMessagingHosts\org.keeform.host]
@="C:\Users\keeform\AppData\Local\KeeForm\org.keeform.host.firefox.json"

This works fine on Firefox 113 (or lower). It also works fine on Chromium based browsers.

Actual results:

But after installing Firefox 114 (beta/rc), native messaging does not work anymore. It is not clear why. This seems to be a regression in 114. I also tested a different path

"path": keeform_host.exe",

But that did not work either.

Expected results:

Native messaging should have worked fine in 114 as well.

Did native messaging change in the Firefox 114? But I could not find any info about that. Or is this just a regression?

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'WebExtensions::Untriaged' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 2

[Alex Cornestean]

Hello,

I tried reproducing the issue but I got stuck at the installation of the native messaging app on Windows 10.
As per the installation tutorial on https://keeform.org/keepass2/keeform-for-firefox, I first clicked the first button and the “KeeForm Installer v4.3.0.exe” installer is downloaded.
I run it and when I get to the “Select KeePass folder” step, the path is blank (it does not behave as in the tutorial video).
I tried adding the path manually, creating the necessary folder, to be as in the video tutorial (C:\Program Files (x86)\KeePass Password Safe 2), but I get an error and cannot install the native messaging app. I’ll attach a screenshot with the error I get.

Any advice on what to do to be able to install the app?

Comment 3

[Alex Cornestean]

Attachment: 2023-06-02_10h05_51.png

Comment 4

[KeeForm]

Hi,

Thank you. You could download the file

https://sourceforge.net/projects/keepass/files/KeePass%202.x/2.53.1/KeePass-2.53.1.zip/download

Unzip the file and use the folder that contains KeePass.exe during the installation. There should be no need to open KeePass.exe during your test.

Comment 5

[KeeForm]

Here is some additional information. When the extension is installed, the extension icon should be visible. The icon has an empty badge. And the badge is usually green after a successful installation. It is red in case of a connection issue (for example, when the user forgot to install the native app).

Comment 6

[Alex Cornestean]

Thank you for the file and additional info !

I reproduced the issue on Beta 114, Nightly 115 and the latest Nightly (116.0a1/20230605210706) under Windows 10 x64. Release 113 is unaffected.

I also ran a mozregression to search for the regressing bug and found this:

2023-06-06T12:58:06.546000: DEBUG : Found commit message:
Bug 1772932 - Port osfile.jsm usage to IOUtils in toolkit/components/extensions r=mixedpuppy

Differential Revision: https://phabricator.services.mozilla.com/D159955

2023-06-06T12:58:06.546000: DEBUG : Did not find a branch, checking all integration branches
2023-06-06T12:58:06.548000: INFO : The bisection is done.
2023-06-06T12:58:06.550000: INFO : Stopped

Corresponding pushlog: https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=6d838112fbd5b0dbc4b260e63abea99748e91587&tochange=cff507e2b180811754fda1c13649195b6d438f82

Comment 7

[BugBot [:suhaib / :marco/ :calixte]]

:barret, since you are the author of the regressor, bug 1772932, could you take a look? Also, could you set the severity field?

For more information, please visit BugBot documentation.

Comment 8

[Rob Wu [:robwu]]

I'll take a quick look.

Bug 1772932 caused a regression before that I fixed in bug 1830605. I'm wondering if there may be a different regression.

Comment 9

[Rob Wu [:robwu]]

KeeForm: to fix the issue in your extension, write the JSON file without a UTF-8 BOM at the front of it. If you have never heard of it, see byte order mark on Wikipedia.

In bug 1772932, OS.File.read was replaced with IOUtils.readUTF8. The former stripped UTF-8 BOMs, the latter does not. The implementation is at https://searchfox.org/mozilla-central/rev/27e4816536c891d85d63695025f2549fd7976392/dom/system/IOUtils.cpp#1277-1282

I'm going to patch the extension implementation to continue stripping the BOM (which Chrome does as well), but I would encourage IOUtils.readUTF8 and IOUtils.readJSON to take an option (or maybe do so by default?) to drop the UTF-8 BOM.

Comment 10

[KeeForm]

Thank you. I made a change in the installer to work around the bug. That fixes the issue for new users. But most of the unsuspecting existing users will still run into this bug soon.

Comment 11

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1772932

Comment 12

[Rob Wu [:robwu]]

Attachment: Bug 1835790 - Drop UTF-8 BOM when reading native manifests

Comment 13

[Rob Wu [:robwu]]

Comment on attachment 9337703 [details]
Bug 1835790 - Drop UTF-8 BOM when reading native manifests

Beta/Release Uplift Approval Request

• User impact if declined: Extensions relying on external JSON files may be broken if that external JSON file starts with a BOM. This affects the following APIs that use native manifests: nativeMessaging, storage (managed storage) and pkcs11
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Very targeted fix, fully covered by unit tests.
• String changes made/needed: none
• Is Android affected?: No

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3fe83edc0cd8
Drop UTF-8 BOM when reading native manifests r=extension-reviewers,rpl

Comment 15

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/3fe83edc0cd8

Comment 16

[Alex Cornestean]

Verified as Fixed on the latest Nightly (116.0a1/20230607033033). Tested under Windows 10 x64.

There is no longer a connection issue between the Firefox extension and the native messaging app. The extension icon shows a green badge as per Comment 5.

Comment 17

[Donal Meehan [:dmeehan]]

Comment on attachment 9337703 [details]
Bug 1835790 - Drop UTF-8 BOM when reading native manifests

Approved for 115.0b3.

Comment 18

[Donal Meehan [:dmeehan]]

https://hg.mozilla.org/releases/mozilla-beta/rev/8d11ff3c9d8a

Comment 19

[Alex Cornestean]

Verified as Fixed on the latest Beta (115.0b3/20230608214116). Tested under Windows 10 x64.

There is no longer a connection issue between the Firefox extension and the native messaging app. The extension icon shows a green badge.

Comment 22

[Pascal Chevrel:pascalc]

Comment on attachment 9337703 [details]
Bug 1835790 - Drop UTF-8 BOM when reading native manifests

Approved for 114.0.2 next week, thanks.

Comment 23

[Pascal Chevrel:pascalc]

https://hg.mozilla.org/releases/mozilla-release/rev/7780429a13e0

Comment 24

[Alex Cornestean]

Verified as Fixed on the latest Release (114.0.2/20230619081400). Tested under Windows 10 x64.

There is no longer a connection issue between the Firefox extension and the native messaging app. The extension icon shows a green badge as per Comment 5, thus confirming the fix.