1550793 - publish armagadd-on 2.0 hotfixes on AMO for self-install

Status: RESOLVED FIXED

(Firefox :: General, task)

Description

[Robert Helmer [:rhelmer]]

We have published a hotfix via Normandy for bug 1548973 and are preparing a legacy hotfix for older releases (52+) in bug 1549604.

However, some users can't/won't install app updates or receive automatic hotfixes, so let's publish these on AMO and direct users who want to self-install there.

Firefox 52 through 60 inclusive will need a legacy hotfix
Firefox 61 through 65 inclusive will get the WebExtension version

Comment 1

[Robert Helmer [:rhelmer]]

Attachment: Legacy hotfix for Firefox 52 through 56 (signed)

Hotfix for 52 through 60

Comment 2

[Robert Helmer [:rhelmer]]

Attachment: Hotfix for Firefox 61 through 65

Hotfix for 61 through 65

Comment 3

[Robert Helmer [:rhelmer]]

Comment on attachment 9064080 [details]
Hotfix for Firefox 61 through 65

Could you please sign this as "Mozilla Components"? This is for Firefox 61+ so you shouldn't need to do all the work from bug 1550793 I hope.

Comment 4

[Greg G]

Attachment: [e-mail]

untested used CN=[e-mail] from the manifest id

Comment 5

[jscher2000]

Comment on attachment 9064080 [details]
Hotfix for Firefox 61 through 65

In the somewhat likely event that users discover this bug, can the unsigned xpi file be hidden so it doesn't look like something people should try to install?

Comment 6

[Robert Helmer [:rhelmer]]

(In reply to jscher2000 from comment #5)

Comment on attachment 9064080 [details]
Hotfix for Firefox 61 through 65

In the somewhat likely event that users discover this bug, can the unsigned xpi file be hidden so it doesn't look like something people should try to install?

Sure, would probably help us keep track too :) I am actually about to submit a second one, to get rid of some misleading console warnings.

Comment 7

[Robert Helmer [:rhelmer]]

Attachment: Hotfix for Firefox 61 through 65 (unsigned)

The previous version was logging a misleading warning, now fixed and reviewed - could you please sign this one? Thanks!

Comment 8

[Greg G]

Attachment: Hotfix for Firefox 61 through 65 (signed)

Comment 9

[Robert Helmer [:rhelmer]]

(In reply to jscher2000 from comment #5)

Comment on attachment 9064080 [details]
Hotfix for Firefox 61 through 65

In the somewhat likely event that users discover this bug, can the unsigned xpi file be hidden so it doesn't look like something people should try to install?

Done.

Comment 10

[Graham Perrin]

https://discourse.mozilla.org/t/-/39845/33 raises public awareness of this bug so I'm taking the opportunity to comment.

(In reply to Robert Helmer [:rhelmer] from comment #1)

Legacy hotfix for Firefox 52 through 60 (signed)

From my Waterfox perspective:

• pass; success.

56.2.9 on (Tier-3) FreeBSD-CURRENT.

(I silently tested some of the earlier pre-release .xpi files. Whilst all those that I tested were ultimately successful, some were quirky in terms of workflow – the post-installation point at which the fix became effective. No such quirks with what I see in this bug 1550793.)

NB to avoid off-topic comments, I assume that feedback from other users of Waterfox – particularly those on Android – should be directed to Waterfox support areas.

---

From a Firefox perspective, I'm aware of one case that might be perceived (by some other users of Firefox) as the legacy hot-fix being not entirely successful. It's an edge case so I'll refrain from posting details here, unless prompted to do so by someone from Mozilla.

Comment 11

[Robert Helmer [:rhelmer]]

(In reply to Graham Perrin from comment #10)

https://discourse.mozilla.org/t/-/39845/33 raises public awareness of this bug so I'm taking the opportunity to comment.

I am glad you found it helpful, and I agree that another forum would be better for non-Firefox discussion in general.

From a Firefox perspective, I'm aware of one case that might be perceived (by some other users of Firefox) as the legacy hot-fix being not entirely successful. It's an edge case so I'll refrain from posting details here, unless prompted to do so by someone from Mozilla.

Any thoughts you have on potential problems/edge cases for the current hotfixes for Firefox users is certainly welcome. Thanks!

Comment 12

[Graham Perrin]

Thanks, (in reply to Robert Helmer [:rhelmer] from comment #11)

… potential problems/edge cases …

Consider the Wood Time theme at https://addons.mozilla.org/addon/wood-time/

• addition to Firefox 66.0.4 is apparently error-free (no suggestion of corruption at time of addition)
• users of post-fix Firefox 56.0.2 might find it impossible to add the theme; might encounter the red alerts that are recently associated with armagadd-on 2.0 i.e.

Download failed. Please check your connection.

– or (if attempting to add the .xpi without using the blue + Install Theme button):

The add-on downloaded from this site could not be installed because it appears to be corrupt.

tl;dr

• from what I can tell, this particular add-on may be exemplary of a type of issue that preceded armagadd-on 2.0
• I plan to re-raise this issue in a GitHub area for AMO, maybe a week after conclusion of a post-mortem.

If the Wood Time theme is corrupt, in a way that is not detectable whilst adding to (release) Firefox 66.0.4: maybe leave it untouched i.e. have a readily available example of something that's 'broken' but not broken in an armagadd-on sense.

FWIW I might have seen random/occasional evidence of corruption with this particular theme, but here's probably not the place to go into detail.

HTH

Comment 13

[cmn3-fox]

I think it was requested here:

https://bugzilla.mozilla.org/show_bug.cgi?id=1549604

that I in fact reply in this bug, so...

Hi, Lina. Yes, it installed an "add-on," which Firefox 52 tried to block, so I overrode that, and now the three extensions do work. Is this the final fix or a temporary one? If this is temporary, will this be removed when the final fix is pushed? Interestingly, when I opened Firefox 52 on that old computer to do this, HTTPS Everywhere had mysteriously re-appeared and was working even before I installed this "add-on" hotfix. But uBlock Origin and Privacy Badger did not re-appear until after the hotfix was applied. Cheers

Comment 14

[Graham Perrin]

(In reply to cmn3-fox from comment #13 and cmn3-fox from https://bugzilla.mozilla.org/show_bug.cgi?id=1549604#c90)

… HTTPS Everywhere had mysteriously re-appeared and was working even before I installed this "add-on" hotfix. …

https://addons.mozilla.org/addon/https-everywhere/versions/ shows that the two most recent versions were 2019.5.2.1 (on Thursday 2nd May) then 2019.5.6.1 (Tuesday 7th May).

Amongst users of Firefox whose profiles are bitten by armagadd-on 2.0 (and not yet fixed), first time users of HTTPS Everywhere may find it:

• impossible to install outdated version 2019.5.2.1 (the red alert, Download failed. Please check your connection.)
• possible to install more recent version 2019.5.6.1 (no red alert).

Firefox preferences are normally set to allow automatic updates to add-ons. Imagine the update from version 2019.5.2.1 to 2019.5.6.1 occurring silently, in the normal way.

cmn3-fox, I guess that your bugged Firefox 52 profile benefited from an automated update to the installable version of HTTPS Everywhere prior to you manually installing Mozilla's hot-fix extension. If you would like to know more, feel free to join the discussion under https://discourse.mozilla.org/t/-/39845/33?u=grahamperrin. Thanks.

---

uBlock Origin and Privacy Badger …

https://addons.mozilla.org/addon/privacy-badger17/versions/ the two most recent versions were 2019.1.30 then 2019.2.19

https://addons.mozilla.org/addon/ublock-origin/versions/ the two most recent were 1.18.16 (2019-04-03) then 1.19.0 (a few hours ago).

Comment 15

[cmn3-fox]

1. Is this the final fix for 52-60 or a temporary one?

2. Once you've deployed the hotfix by installing the extension, can you delete the extension, or no?

Comment 16

[Graham Perrin]

(In reply to cmn3-fox from comment #15)

1. Is this the final fix for 52-60 or a temporary one?

I assume that Mozilla will treat this hot-fix as final – and then make it available (maybe featured) in the extensions area of AMO https://addons.mozilla.org/extensions/ – if end users find no significant problem with the fix.

At a glance, I see no such problem in key areas such as these:

• this bug 1550793
• commentary under https://discourse.mozilla.org/t/-/39845/33
• https://support.mozilla.org/products/firefox

– so at this time, I'm hopeful.

---

2. Once you've deployed the hotfix by installing the extension, can you delete the extension, or no?

As far as I can tell:

• if Mozilla's certificate authority (CA) is successfully 'injected' then yes – you can, and should, delete the extension.

Also AFAICT:

• disabling the extension will have no effect.

For an ordinary extension, it would be unusual to find a non-effective Disable button.

This is an extraordinary extension – after we gain Mozilla's fix (its CA), we want nothing to delete or distrust the fix – so in this case it's proper for the Disable button to have no effect.

---

Under https://discourse.mozilla.org/t/-/39845/33 I'll add a couple of screenshots that may help users to tell whether the hot-fix is successful.

Comment 17

[Robert Helmer [:rhelmer]]

(In reply to Graham Perrin from comment #16)

(In reply to cmn3-fox from comment #15)

1. Is this the final fix for 52-60 or a temporary one?

I assume that Mozilla will treat this hot-fix as final – and then make it available (maybe featured) in the extensions area of AMO https://addons.mozilla.org/extensions/ – if end users find no significant problem with the fix.

QA is still ongoing, we hope this is the final fix for users who choose to self-install rather than upgrade to the latest Firefox release at this time. Upgrading to the latest Firefox is best, if at all possible.

2. Once you've deployed the hotfix by installing the extension, can you delete the extension, or no?

Yes, the new intermediate certificate persists in your profile so you can delete or disable the extension after installing. You can confirm the extension worked by looking in the Browser Console.

The output should look something like:

WebExtensions: new intermediate certificate added       api.js:32
WebExtensions: signatures re-verified                   api.js:42

Thanks for helping test this!

Comment 18

[cmn3-fox]

Some of us, of course, are running the latest Firefox on our computers (as I am on this Mac) but also have some old XP/Vista boxes sitting around for certain reasons and, on those, we are limited to using the final 52. On occasion, whilst doing something on one of those XP boxes, we have occasion to briefly visit the web and, when doing so, we use a variety of FF extensions to make that a safer journey, in addition to things like having set IE security to high to block, systemwide, almost everything that might want to install, being NATted and firewalled and etc, running antivirus and whatever. Thanks to FF engineers for fixing this "unsupported" software relatively quickly. I seem to remember Microsoft issuing some emergency XP patch not a whole long time ago, long after they'd stopped supporting XP, so such actions are not unheard-of. Cheers

Comment 19

[cmn3-fox]

Hi rhelmer: I have nothing like that in any tab of the Browser Console. But, I'm gonna go ahead and remove the extension on one XP box and see what happens. ;) If you want me to paste in all the output of all the tabs in the Browser Console, I could do that, as it doesn't look like anything sensitive.

Comment 20

[cmn3-fox]

Hi rhelmer: Removing the hotfix extension did not break the other extensions.

Browser Console contains many things, including:

addons.xpi WARN Failed to call uninstall for [e-mail]: Error: Unknown add-on ID [e-mail] (resource://gre/modules/addons/XPIProvider.jsm:8303:11) JS Stack trace: [e-mail]:8303:11 < [e-mail]:3409:31 < [e-mail]:3745:19 < [e-mail]:2830:25 < [e-mail]:237:12 < [e-mail]:790:5 < [e-mail]:976:9 < [e-mail]:3033:5 < [e-mail]:65:9

addons.xpi WARN Attempted to remove [e-mail] from app-profile but it was already gone

There are also a bunch of entries along these lines:

[e-mail] WARN Loading extension '[e-mail]': Reading manifest: Error processing devtools_page: An unexpected property was found in the WebExtension manifest.

addons.webextension.jid1-MnnxcxisBPnSXQ@jetpack WARN Loading extension 'jid1-MnnxcxisBPnSXQ@jetpack': Reading manifest: Error processing permissions.9: Unknown permission "privacy"

addons.webextension.jid1-MnnxcxisBPnSXQ@jetpack WARN Loading extension 'jid1-MnnxcxisBPnSXQ@jetpack': Reading manifest: Error processing storage: An unexpected property was found in the WebExtension manifest.

Cheers

Comment 21

[Graham Perrin]

(In reply to Graham Perrin from comment #16)

For now, please IGNORE my previously given advice:

2. Once you've deployed the hotfix by installing the extension, can you delete the extension, or no?

As far as I can tell:

• if Mozilla's certificate authority (CA) is successfully 'injected' then yes – you can, and should, delete the extension.

I'm no longer certain that deletion is appropriate.

---

Can someone who does NOT use containers please test the following?

1. add the hot-fix (the extension at https://bugzilla.mozilla.org/show_bug.cgi?id=1550793#module-attachments-title above)
2. remove the extension from Firefox
3. about:support
4. refresh Firefox
5. after the refresh, visit about:preferences
6. review the list of certificates.

Did the refresh, after removal of the extension, lead to loss of Mozilla's certificate authority (CA)?

If in doubt: backup before testing.

Back up and restore information in Firefox profiles | Firefox Help

Thanks

Comment 22

[Masatoshi Kimura [:emk]]

(In reply to Graham Perrin from comment #21)

IIRC the refresh will remove all add-ons (including the hot-fix). So I don't think it helps to keep the hot-fix. (But didn't test.)

Comment 23

[cmn3-fox]

Perhaps this well help. As I relayed above, I removed the hotfix extension. My extensions remained functional. Looking now, in Firefox 52.9.0 on Windows XP about:preferences#advanced/Certificates/View Certificates, I have one called "Mozilla Corporation signingca1.addons.mozilla.org". It says, "Could not verify this certificate because the issuer is unknown." The serial number is 10:00:08. The common name is root-ca-production-amo. The organizational unit is Mozilla AMO Production Signing Service. It began on April 3, 2015 and expires on April 3, 2025. It has an SHA-256 and SHA1 fingerprint. It's Version is 3. Is this helpful?

Comment 24

[Graham Perrin]

(In reply to Masatoshi Kimura [:emk] from comment #22)

refresh will remove all add-ons (including the hot-fix).

Of course! Thank you for the reminder. It was careless of me to not mention the knowledge base article:

Refresh Firefox - reset add-ons and settings | Firefox Help
http://mzl.la/15sSfJR

– in particular:

Note: Your old Firefox profile will be placed on your
          desktop in a folder named "Old Firefox Data".

The purpose of the six steps at comment 21 is:

• to test whether a post-fix refresh will remove the certificate
    (in cases where Firefox gains the certificate solely from a
    self-install extension from this bug or (eventually) from AMO).

If so:

• I suggest a brief explanation in a knowledge base article
• I do have a thought on how removal might be avoided, but I'll take that thought elsewhere …

---

Readers please note, this forward-looking stuff is not to detract from the more immediate need to test the effectiveness of the hot-fix at time of installation. Thanks.

Comment 25

[Robert Helmer [:rhelmer]]

Comment on attachment 9064110 [details]
Hotfix for Firefox 61 through 65 (signed)

Please sign this one with the "Mozilla Extensions" OU as well, as we discussed there is a bug in at least 60.6.1 that prevents the Components OU from working (https://bugzilla.mozilla.org/show_bug.cgi?id=1454820)

Comment 26

[Greg G]

Attachment: Hotfix for Firefox 57 through 61 (signed)

Comment 27

[cmn3-fox]

(In reply to Greg Guthe [:g-k] [:gguthe] from comment #26)

Is this something new that those of us who already applied the 52-26 extension hotfix need to also apply now?

Comment 28

[Robert Helmer [:rhelmer]]

Comment on attachment 9064486 [details]
Hotfix for Firefox 57 through 61 (signed)

This is signed with the "Mozilla Extensions" OU rather than "Components", due to bug 1454820.

Comment 29

[Robert Helmer [:rhelmer]]

Comment on attachment 9064073 [details]
Legacy hotfix for Firefox 52 through 56 (signed)

Please sign this one with the "Mozilla Extensions" OU as well (this is the legacy add-on), none of the combinations we have are actually working for 57->61 but I am hopeful this will.

Comment 30

[Greg G]

Attachment: Hotfix for Firefox 57 through 60 (signed)

Comment 31

[SeplPeda]

(In reply to Robert Helmer [:rhelmer] from comment #1)

Created attachment 9064073 [details]
Legacy hotfix for Firefox 52 through 56 (signed)

Hotfix for 52 through 60

Hi,
after installing the hotfix for my FF 56.0.2 everything working again. The addons are activated again and they are also updating ...

*** Except I can't install any new addon so far! ***

When I visit mozilla's addon site trying to install for example "enhancer for youtube" I am told that (I translate from german to english):

"This add-on needs a newer version of FF (minimum 55.0). You are using FF 50.0"

But of course this is BS, I am using version 56.0.2. And "enhancer for youtube" was perfectly running up to the moment when I (in desparation) uninstalled it because of the CA-desaster 8 days ago, trying to reinstall it ... which of course did not work out.

The point is that this behavior on mozilla's addon-site is now pervasive. No matter what I am trying to "add-on" I am told that my browser version is 50.0 .... Even for addons that are currently installed and running.

Comment 32

[jscher2000]

(In reply to SeplPeda from comment #31)

When I visit mozilla's addon site trying to install for example "enhancer for youtube" I am told that (I translate from german to english):

"This add-on needs a newer version of FF (minimum 55.0). You are using FF 50.0"

But of course this is BS, I am using version 56.0.2. And "enhancer for youtube" was perfectly running up to the moment when I (in desparation) uninstalled it because of the CA-desaster 8 days ago, trying to reinstall it ... which of course did not work out.

The point is that this behavior on mozilla's addon-site is now pervasive. No matter what I am trying to "add-on" I am told that my browser version is 50.0 .... Even for addons that are currently installed and running.

The Add-ons site gets your version from your user agent string. That commonly look something like this:

Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0

This can be overridden by a preference. To check on that, see: https://support.mozilla.org/kb/how-reset-default-user-agent-firefox

Also, if you have an extension similar to a "user agent switcher" please disable it or update your selection.

If that does not help, please test in Firefox's Safe Mode, and post a new question on Mozilla Support:

• https://support.mozilla.org/kb/troubleshoot-firefox-issues-using-safe-mode
• https://support.mozilla.org/questions/new/desktop/fix-problems

Scroll down past article suggestions to continue with the form. To avoid losing your place, you may want to open suggested articles/threads in new tabs.

Comment 33

[Robert Helmer [:rhelmer]]

(In reply to SeplPeda from comment #31)

(In reply to Robert Helmer [:rhelmer] from comment #1)

"This add-on needs a newer version of FF (minimum 55.0). You are using FF 50.0"

Do you have privacy.resistFingerprinting set in about:config?

Comment 34

[Robert Helmer [:rhelmer]]

Attachment: Hotfix for Firefox 57 through 60 (unsigned)

Please sign with "Mozilla Extensions" OU - the only expected change from previous version is:

1. the add-on ID, since the AMO folks want to host these as three separate add-ons and not different versions of the same one.
2. version bump
3. tightened up minVersion

Comment 35

[Greg G]

Attachment: Hotfix for Firefox 57 through 60 (signed)

Comment 36

[SeplPeda]

(In reply to Robert Helmer [:rhelmer] from comment #33)

(In reply to SeplPeda from comment #31)

(In reply to Robert Helmer [:rhelmer] from comment #1)

"This add-on needs a newer version of FF (minimum 55.0). You are using FF 50.0"

Do you have privacy.resistFingerprinting set in about:config?

Yep, resetting to "false" did the job. Thank You! Also thank you to Jscher2000 for providing help!

I have no idea how that got set to "true" ...

Comment 37

[:shell escalante]

Because the browser evolves over time - there were 3 separate extensions needed to deliver the fixes needed for different Firefox versions.

These extension will install a new security certificate and re-enable extensions and themes for:

1. Firefox 52-56 https://addons.mozilla.org/en-US/firefox/addon/disabled-add-on-fix-52-56/ bug 1551289
2. Firefox 57-60 https://addons.mozilla.org/en-US/firefox/addon/disabled-add-on-fix-57-60/ bug 1551291
3. Firefox 61-65 https://addons.mozilla.org/en-US/firefox/addon/disabled-add-on-fix-61-65/ bug 1551290

For earlier versions of Firefox (pre-52) the Firefox 52-56 extension fix on AMO has not been widely tested on pre-52 versions of Firefox, but has been reported to work on some machines. https://addons.mozilla.org/en-US/firefox/addon/disabled-add-on-fix-52-56/