Closed Bug 1549604 Opened 6 years ago Closed 5 years ago

ship legacy system add-ons to remediate armagadd-on 2.0

Categories

(Firefox :: General, task, P1)

Product:
Firefox
Component:
General
Type:
task

Tracking

()

Status:
RESOLVED FIXED

People

(Reporter: rhelmer, Assigned: rhelmer)

References

Details

(Whiteboard: cert2019)

Attachments

(18 obsolete files)

The current hotfix for bug 1548973 requires studies to be enabled, and some older versions of Firefox cannot receive the hotfix.

Let's ship a legacy add-on to remediate users impacted by bug 1548973 on releases earlier than 60, and re-use the Normandy-shipped system add-on to 61+ users via Balrog.

This should also help current users who have disabled studies, but do have the legacy system add-on updates enabled, which is the default and not exposed in UI, only in about:config.

The exact set of releases we will ship to is TBD, but we expect the legacy system add-on will work at least as far back as 52.

:wezhou, please sign this system add-on update, thanks!

Attachment #9063087 - Flags: feedback?(wezhou)
Summary: ship legacy system add-ons to remediate armag-add-on 2.0 → ship legacy system add-ons to remediate armagadd-on 2.0
Type: defect → task

The usual signing command I have been using fails with the following error,

{"Timestamp":1557182315642668853,"Time":"2019-05-06T22:38:35Z","Type":"app.log","Logger":"autograph","Hostname":"mbp2.localdomain","EnvVersion":"2.0","Pid":91470,"Severity":2,"Fields":{"msg":"failed to add signer \"extension_rsa\": xpi: signer certificate is not currently valid"}}

This may have to do with the recent renewed intermediate certificate.

I'll check with autograph developers to see how to fix this.

:rhelmer do you know if this should be signed with a backdated NotBefore?

Flags: needinfo?(rhelmer)

backdating it is fine and I believe that's what we did for the other hotfix addons

Flags: needinfo?(rhelmer)
Attachment #9063087 - Flags: feedback?(wezhou)

Unfortunately attachment #9063109 [details] does not install on Firefox 56.0.2 win64!

1557229798292 addons.xpi WARN Add-on hotfix-update-xpi-signing-intermediate-bug-1548973-legacy@mozilla.org is not correctly signed.
1557229798294 addons.xpi WARN Invalid XPI: signature verification failed

Bootstrap issue?

More importantly, attachment #9063109 [details] does not install on Firefox ESR 60.0.2 either.

attachment #9055565 [details] (from bug 1541316) did not install on ESR 60.0.2, either.

1557230552113 addons.xpi WARN Add-on baidu-code-update@mozillaonline.com is not correctly signed.
1557230552113 addons.xpi WARN Invalid XPI: signature verification failed

We can't deploy hot-fixes to older versions anymore?

(In reply to Masatoshi Kimura [:emk] from comment #9)

attachment #9055565 [details] (from bug 1541316) did not install on ESR 60.0.2, either.

1557230552113 addons.xpi WARN Add-on baidu-code-update@mozillaonline.com is not correctly signed.
1557230552113 addons.xpi WARN Invalid XPI: signature verification failed

We can't deploy hot-fixes to older versions anymore?

That extension might have the wrong OU (regular extension and not hotfix). Let me try resigning.

attachment #9063109 [details] did not work even on ESR 60.0.3! But attachment #9055565 [details] worked.
Something must be wrong about attachment #9063109 [details].

"Mozilla Components" OU and no COSE signature (which matches https://bugzilla.mozilla.org/attachment.cgi?id=9055565)

Attachment #9063109 - Attachment is obsolete: true

(In reply to Greg Guthe [:g-k] [:gguthe] from comment #12)

Created attachment 9063195 [details]
hotfix-update-xpi-signing-intermediate-bug-1548973-legacy@mozilla.org-signed-sao.xpi

"Mozilla Components" OU and no COSE signature (which matches https://bugzilla.mozilla.org/attachment.cgi?id=9055565)

This file did not install on ESR 60.0.3, either :(

1557238451952 addons.xpi WARN Add-on hotfix-update-xpi-signing-intermediate-bug-1548973-legacy@mozilla.org is not correctly signed.
1557238451953 addons.xpi WARN Download of https://bug1549604.bmoattachments.org/attachment.cgi?id=9063195 failed: signature verification failed

Neither on 56.0.2 :(

signed with the intermediate from bug 1521868

http://www.lapo.it/asn1js/#MIIRIQYJKoZIhvcNAQcCoIIREjCCEQ4CAQExCTAHBgUrDgMCGjALBgkqhkiG9w0BBwGggg21MIIGgzCCBGugAwIBAgIIFZxt5iah8QcwDQYJKoZIhvcNAQEMBQAwgaQxCzAJBgNVBAYTAlVTMRwwGgYDVQQKExNNb3ppbGxhIENvcnBvcmF0aW9uMS8wLQYDVQQLEyZNb3ppbGxhIEFNTyBQcm9kdWN0aW9uIFNpZ25pbmcgU2VydmljZTFGMEQGA1UEAww9c2lnbmluZ2NhMy5hZGRvbnMubW96aWxsYS5vcmcvZW1haWxBZGRyZXNzPWZveHNlY0Btb3ppbGxhLmNvbTAeFw0xOTA0MjYxNzMzMjBaFw0yMDA1MDYxNDQ2NDZaMIGwMQswCQYDVQQGEwJVUzELMAkGA1UECBMCQ0ExFjAUBgNVBAcTDU1vdW50YWluIFZpZXcxDzANBgNVBAoTBkFkZG9uczEbMBkGA1UECxMSTW96aWxsYSBDb21wb25lbnRzMU4wTAYDVQQDDEVob3RmaXgtdXBkYXRlLXhwaS1zaWduaW5nLWludGVybWVkaWF0ZS1idWctMTU0ODk3My1sZWdhY3lAbW96aWxsYS5vcmcwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDrT_gRMrw1EngM3gExTktBtjcY0_lCk0jdVG0iIcm355ajZYqbNli7jnHd7A5D4UbdT70ETyr2Vv8LjuH2BiBY93lb5myYsYjg2YIUlxXjH4-YNIcQmeXttKewm2Fl_Vfxv-QCBuGLDVJCn46QRVMQNG_fHBPsITJqSbqSRZEhDr_NtrQXdeJ0D3yLIUYEXQB9x-T8qQDxbDmeiAw2y7BXPsOQspSy7Kyuw12aVPG33WiBWS2u7Mcswd7pQpus9pAYsKvNXkPabi1ywLEH54qfOK4p33zJmqMMDucmElNnuDxUmkjQD2-fgJHVa7-_fv0K6heGVQC25pQd5TXQLTJjBabecBWJRYDw_bKVUFdhtXV-E4l5k8oo5M4MEtso5Ny48EZegl6hsxbjedC-2432bCf31b3B7-4o6fj5eOUlBzIv3hSeeQ3rEiI_WZEkr03fyeshKh-zMWmH5ITS4cQKTmaKgret9gE_G-N0JqiQo61i6m0wpEEl5YG42HRHItSBFqYW6hUASo9BWBS2UL15pRPlWWQ3fFkJ0ETt_6bbeQrCTT5BFj9LVNW9sumNMAnIT_pljf86kukhwSVOv5uIj21oOjxYe-mSuWtiDzbNwDOCFlQn1RmlQ-5WO2UpC8w5T6INlqEu0HFo7pVW7PpyvtUfaTkGRUM_wVKoIcoX6QIDAQABo4GqMIGnMA4GA1UdDwEB_wQEAwIHgDATBgNVHSUEDDAKBggrBgEFBQcDAzAfBgNVHSMEGDAWgBS27fEn95EGPY9aBIcIzxty3vWDXTBfBgNVHREEWDBWglQ5OGM1ZjZlYzAxMDU0MjYwMjljYjcwMzEzZDg2Yjk4My4xYjYxZTdlZDZjYzczOWMzNTk4ZTAxZGY0ZDU4ODZhZi5hZGRvbnMubW96aWxsYS5vcmcwDQYJKoZIhvcNAQEMBQADggIBACsooFzzdAwllAR8BD7yWa1RG1BdmueSsPemKDncbTc_W_XRd0lL36s7unZKNeFelkKAyKRbSyirsT14KA6-lyGC2FrC2U0en785f31q2upjGfoEPqM1GWoAp2003KRD4wXyzdbEyIPX8HHzEu3Zfj8IYeWbJf2STAiGBBDkgfs2_ibAfiDhnYYkqxGwC4qdUw5jsdm2XJhgBM9I4ImTlp0uBYKRtKwEgQvxvL6vmeqmO7y5IjcsflljYbpyciNbph7Ec86N0sNQlW3cruCVz-KvxGw1t6L3A4rB-L72UgWsxAWswmuL8cLWuhJGCHsTdRb5AtEKdjnvk_TIuK81LppkdmxxfV5yd7SDUTKdb3kG_fnciK9gRf9SZST7vVwMNazGOg_CWa_bas459OQqWpvmeXXwv430uAce3_BOWUGc89bkXP5RMVk9UKkTidb2KiAdlqqXuwwtecOW8nCTUE9J16aKTUIs8CRYazb2IJRB9I_ZM85CTNx1gmdV-msixOW8R49VEipcHaVBq25wSkoI3Hv55n-6T-ziDpSRrdZAgYumYEf509zQoLgU0Kp9GmdyKDqhr2rV5tJ2E5ADuQEZv36TAMZAAdIWfoVpurxYxaVCCTLNcmwzuuoXXoaG8Af7wapLvOttQCxkfePI4tTfgk2dvFOQh9hhqyyw8LciMIIHKjCCBRKgAwIBAgIDEAAFMA0GCSqGSIb3DQEBDAUAMH0xCzAJBgNVBAYTAlVTMRwwGgYDVQQKExNNb3ppbGxhIENvcnBvcmF0aW9uMS8wLQYDVQQLEyZNb3ppbGxhIEFNTyBQcm9kdWN0aW9uIFNpZ25pbmcgU2VydmljZTEfMB0GA1UEAxMWcm9vdC1jYS1wcm9kdWN0aW9uLWFtbzAeFw0xOTAyMDEyMjA4MjVaFw0yMTAxMzEyMjA4MjVaMIGkMQswCQYDVQQGEwJVUzEcMBoGA1UEChMTTW96aWxsYSBDb3Jwb3JhdGlvbjEvMC0GA1UECxMmTW96aWxsYSBBTU8gUHJvZHVjdGlvbiBTaWduaW5nIFNlcnZpY2UxRjBEBgNVBAMMPXNpZ25pbmdjYTMuYWRkb25zLm1vemlsbGEub3JnL2VtYWlsQWRkcmVzcz1mb3hzZWNAbW96aWxsYS5jb20wggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQC6DzJyOhF-mOUX03VQ0jzZlNrMGL4ny9dhAimDw3ArAifztdwPo7saHweTgb1WoXrIfnsQybB2ALvcFn-kzOybL8_Xs027LfH3L06i4EB0cQRXHiGkRawd_eMxKTS0rTVv07WuYrRowjqNIwy35psOiicrziDxIcRbaOK7VQ-UGoRd-DSvUrg6V6f-8VjOGHsLgyyWW-y0EULBvWAzK1peEz1ZZ-XPWfgFjo62SPuTmInSUL0ep9JrrhQ_CAYCP_okC-xtrcybO97BvZJ04WXw5BmTdCBD4U6_x_N24WF1KO6-2mbS8BKMasRQjmjGlDNGKZeIOKy4clzm6WGOTQcVcFKjgnLDmI895jeXmYDUOuuTbBDsVe2XM9dF57ZW504jdQz74BM9o-4WL1CvFThpkUqDeZrk6sALzFmOqT6M5NtwfKJ-PNRxJvRnTsbNSqXxPmH2CQK4PbmhDXxvlQGlIKTs5b0Y_GdvRFHf6yBB9V03D5IiQ3yxquPdkYmQpDD8ERpnZS-lVD13mzh38EvTKq8WQmZIsm0KR6Vm66v8fTU0cDgooEKYgVcrTT7CGg-Ts5iyp-8ml6_zetqS027UpRjQo-uSh3JxbGre7yJ0eWvIhyKgnNF1U92mmhr8LnG_quYk9yKrilZJ7C98urw_FOYqJzV_RhncJsFEKSib6wIDAQABo4IBiTCCAYUwDAYDVR0TBAUwAwEB_zAOBgNVHQ8BAf8EBAMCAQYwFgYDVR0lAQH_BAwwCgYIKwYBBQUHAwMwHQYDVR0OBBYEFLbt8Sf3kQY9j1oEhwjPG3Le9YNdMIGoBgNVHSMEgaAwgZ2AFLO86lh0q-FueCqyq5wjHqhjLJe3oYGBpH8wfTELMAkGA1UEBhMCVVMxHDAaBgNVBAoTE01vemlsbGEgQ29ycG9yYXRpb24xLzAtBgNVBAsTJk1vemlsbGEgQU1PIFByb2R1Y3Rpb24gU2lnbmluZyBTZXJ2aWNlMR8wHQYDVQQDExZyb290LWNhLXByb2R1Y3Rpb24tYW1vggEBMDMGCWCGSAGG-EIBBAQmFiRodHRwOi8vYWRkb25zLm1vemlsbGEub3JnL2NhL2NybC5wZW0wTgYDVR0eBEcwRaFDMCCCHi5jb250ZW50LXNpZ25hdHVyZS5tb3ppbGxhLm9yZzAfgh1jb250ZW50LXNpZ25hdHVyZS5tb3ppbGxhLm9yZzANBgkqhkiG9w0BAQwFAAOCAgEAb-wmNVyDaTtlrh9NNz0F5q9JPJI0uUH4F8EqPA3fY9rASQaYIFLltxheM0ZbMhV5NODrW9TjaGfdzGE1cNBszlioVYPN0nmByzGr7no8n3hyz46ic0zSjZjiWNTKh37UHFH1DKFrr1MT_XoMH0sbqvg0_JWKPu2lYL33ZY3g22eLWH-GyrS8_tBjud5uq8gHeSISZY6Lt-AXwwrNshHoLmUFKxYa9oIvZ_jkur6B4Uy4HThgBjDe1vYoEy7Ua2w4g7US8g8HRsq9c-W7Hy8zCK06OWE0s-J4j1nRUdpQR6zoEqOFK5EAOV2ofnd0l-IhpOF29zv99YD3tUocKA3qSnLv7jLvtE04C35CNcNmAq0pIH80XZuMt0CQK7WdFZFr8l8OIQ7uhjwLBbAxhI10lNWmjRDDeuXHWRqhWsmUOk8HJ-4v6I6SPW8PtAOczdUaH6joYjrt1Heny3hrp_gV3lAilpI0iCkrd-JUy7MgVvTs55b08DdkowO-qQQnOyvcqcg5bT4EuoOD1kOmrugZp2aQ6HyPnk58J8myRoh72xhG1qdlGzIpl709QgSwqjWj6kKW0ByyLMM4bsCYOVj-zixvbMbhfPJpc1340Y8aW-DmnoI5Qd9Rtrg1l8wo2D5LAdEj9Sn5lYVxerpnfW1P2N5oItOy44UlRmKkrMrFZZYxggM2MIIDMgIBATCBsTCBpDELMAkGA1UEBhMCVVMxHDAaBgNVBAoTE01vemlsbGEgQ29ycG9yYXRpb24xLzAtBgNVBAsTJk1vemlsbGEgQU1PIFByb2R1Y3Rpb24gU2lnbmluZyBTZXJ2aWNlMUYwRAYDVQQDDD1zaWduaW5nY2EzLmFkZG9ucy5tb3ppbGxhLm9yZy9lbWFpbEFkZHJlc3M9Zm94c2VjQG1vemlsbGEuY29tAggVnG3mJqHxBzAHBgUrDgMCGqBdMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTE5MDUwNzE0NDY0OFowIwYJKoZIhvcNAQkEMRYEFNogHEZgi6Us9WqIGurhAIbCjsZQMAsGCSqGSIb3DQEBBQSCAgC-WA9zHidLUlZbFRADw3fPmNjzLQpOOoNkDOkimX-z8thBF8sEJCL2cBe_Z4D9Bq786Pia4ooHRNfANTGR4zCY9w-Kp6bfkp1jalMEXfEKi4dCtSC7ZrF2S_8Va1wqBLB3RVqFu2QwW7TfPNaEKd4xoxvNEma5gTNSlJk1WqaeIuMRYve09Kftkw6KcL3b1bednsRXZ3E4TH5x8_UZ2aeWlYcdoPdYXXx7Fq3eO9qG2YMGYYtnOII3rrwrb7_c0IJiYFAfn4KmH3W23Gp4HZS3rKMgajtz45AtkbmKoOyf66o4eYZJbV5aTb4eEqweZNHeQcCey0FTTI6nW7FF8QxjCdfPWm0SuvQ5GqcY-6H0LAjvGoNfV5oyhq6CldMn-A9h8JKMKYCXkH7PgUMGH-B10FNDKejhQq-Rgf4pdSbn29o0bfG9aKv9aiMVYvpFiZG2jQzmZIvgBh3e4ZDqXZ0rf8gpRMfO00g0J6YFcHrms_UOAetU5t7LQD2b-6yoxbKWUr-IE7DeXRYe47-hWtz1uu1NkAJ3TEL3d_M96GAk7PlnY0of3wTe0Ip9kUdk3L2NG8Gr_CDdCxAkGqkHhbYx1GrNt31Bc66E4oTn8TyLZyBOTZacKJT2LyPTWZAx6E7KnFxRwfBwoiD9HmFisWPzxTZ7xzav99U7TYpd6ppRqqEA
Attachment #9063195 - Attachment is obsolete: true

Still no luck.

1557241619827 addons.xpi WARN Add-on hotfix-update-xpi-signing-intermediate-bug-1548973-legacy@mozilla.org is not correctly signed.
1557241619828 addons.xpi WARN Download of https://bug1549604.bmoattachments.org/attachment.cgi?id=9063220 failed: signature verification failed

1557242168084 addons.xpi WARN Add-on hotfix-update-xpi-signing-intermediate-bug-1548973-legacy@mozilla.org is not correctly signed.
1557242168084 addons.xpi WARN Download of https://bug1549604.bmoattachments.org/attachment.cgi?id=9063222 failed: signature verification failed

Both (hsm-inter-sao and hotfix-mode) not installing on FF 56.0.2

Whiteboard: cert2019

https://storage.googleapis.com/moz-fx-normandy-prod-addons/extensions/hotfix-update-xpi-intermediate@mozilla.com-1.0.2-signed.xpi
FYI, this XPI works fine in regard to the signature verification perspective (but it fails due to incompatible extensions API).

(In reply to Masatoshi Kimura [:emk] from comment #20)

https://storage.googleapis.com/moz-fx-normandy-prod-addons/extensions/hotfix-update-xpi-intermediate@mozilla.com-1.0.2-signed.xpi
FYI, this XPI works fine in regard to the signature verification perspective (but it fails due to incompatible extensions API).

OK thanks I'll try to match that one. It's using:

CN=hotfix-update-xpi-intermediate@mozilla.com
OU=Mozilla Extensions

the first signed addon used the same OU but a .org CN, so I'll retry with a .com CN

1557244622371 addons.xpi WARN Add-on hotfix-update-xpi-signing-intermediate-bug-1548973-legacy@mozilla.org is not correctly signed.
1557244622372 addons.xpi WARN Download of https://bug1549604.bmoattachments.org/attachment.cgi?id=9063229 failed: signature verification failed

I suspect that this long CN is the culprit. I heard CN must be 64 characters or shorter.

OU=Mozilla Extensions, CN=hotfix-bug-1548973-legacy@mozilla.com (37 chars)

Attachment #9063229 - Attachment is obsolete: true

1557246200747 addons.xpi WARN Add-on hotfix-update-xpi-signing-intermediate-bug-1548973-legacy@mozilla.org is not correctly signed.
1557246200748 addons.xpi WARN Download of https://bug1549604.bmoattachments.org/attachment.cgi?id=9063235 failed: signature verification failed

I ran out of ideas, sorry. Deeper debugging would be required.

OK in https://mozilla.slack.com/archives/CJGBVE4G6/p1557242465333500 :dveditz said ESR 52 doesn't support the "Mozilla Extensions" OU but does support "Mozilla Components" so I'll resign with the Components OU and short .com and .org CNs

I noticed hotfix-update-xpi-intermediate@mozilla.com-1.0.2-signed.xpi had MD5/SHA1 digests in manifest.mf and mozilla.mf, but files in this bug had SHA1/SHA256 digests. Do older versions of Firefox support SHA256 digests?

1557247762468 addons.xpi WARN Add-on hotfix-update-xpi-signing-intermediate-bug-1548973-legacy@mozilla.org is not correctly signed.
1557247762469 addons.xpi WARN Download of https://bug1549604.bmoattachments.org/attachment.cgi?id=9063245 failed: signature verification failed
1557247767369 addons.xpi WARN Add-on hotfix-update-xpi-signing-intermediate-bug-1548973-legacy@mozilla.org is not correctly signed.
1557247767370 addons.xpi WARN Download of https://bug1549604.bmoattachments.org/attachment.cgi?id=9063248 failed: signature verification failed

Comment 28 worked fine. (I didn't enable extensions.legacy.enabled, but at least the signature verification passed.)

1557247872031 addons.xpi WARN disabling legacy extension tls13-compat-ff51@mozilla.org

(In reply to Robert Helmer [:rhelmer] from comment #28)

Here is an example of a XPI that works with 56:
http://ftp.mozilla.org/pub/system-addons/tls13-compat-ff51/tls13-compat-ff51@mozilla.org_1.0.4.xpi

Also installs on 52.9.0esr.

OK so it could be:

tls compat sig

short CN org

Sig that to more closely matches the tls webcompat patch (data digest is signed, attrs aren't). Gives an invalid manifest error on Nightly.

Attachment #9063245 - Attachment is obsolete: true
Attachment #9063248 - Attachment is obsolete: true

(In reply to Greg Guthe [:g-k] [:gguthe] from comment #36)

Created attachment 9063303 [details]
hotfix-bug-1548973-legacy@mozilla.org-signed-sao-enc-rsa-5.xpi

Sig that to more closely matches the tls webcompat patch (data digest is signed, attrs aren't). Gives an invalid manifest error on Nightly.

Yeah, this is expected because newer Firefox only supports WebExtension not the type of legacy extension this is.

@rhelmer why not just use WebExtension for all? Support goes back to at least version 48:

https://developer.mozilla.org/Mozilla/Add-ons/WebExtensions/manifest.json

Flags: needinfo?(rhelmer)

(In reply to Steven Penny from comment #38)

@rhelmer why not just use WebExtension for all? Support goes back to at least version 48:

https://developer.mozilla.org/Mozilla/Add-ons/WebExtensions/manifest.json

This fix needs to use privileged code, and WebExtension Experiments did not work for system add-ons until 61 due to bug 1454820.

Flags: needinfo?(rhelmer)

Sorry there was a minor bug in the extension (unrelated to signing), here's a version that WFM locally.

Attachment #9063087 - Attachment is obsolete: true
Attachment #9063303 - Attachment is obsolete: true
Attachment #9063339 - Flags: feedback?(gguthe)
Attachment #9063339 - Flags: feedback?(gguthe) → feedback+

signed with the old python sign-addon script against a similarly old autograph version

edit: this is giving me a signature verification error in 60.6.2esr

Sorry for the late reply, my timezone was midnight.

1557268792893 addons.xpi WARN Add-on hotfix-update-xpi-signing-intermediate-bug-1548973-legacy@mozilla.org is not correctly signed.1557268792893 addons.xpi WARN Download of https://bug1549604.bmoattachments.org/attachment.cgi?id=9063339 failed: signature is required but missing
1557268819237 addons.xpi WARN Add-on hotfix-update-xpi-signing-intermediate-bug-1548973-legacy@mozilla.org is not correctly signed.1557268819238 addons.xpi WARN Download of https://bug1549604.bmoattachments.org/attachment.cgi?id=9063342 failed: signature verification failed

I will go to work now. Probably I can't reply until I go home.

Per :emk's suggestion in comment 23 truncated the CN to hotfix-bug-1548973@mozilla.org and updated the RDF > Description > em:id in install.rdf (which I didn't know to do for earlier tries). Didn't bump the version number.

This is signed with the autograph DNS altnames disabled and a SHA256 digest hardcoded in autograph (probably not necessary I was just trying to get closer to the autograph version I think :jason used in bug 1325872).

on ESR 52 (linux x86_64) it installs for me with console output:

...
Query failed: Error: Error(s) encountered during statement execution: no such table: moz_favicons
1557272189154   addons.repository       WARN    Search failed when adding add-ons to cache
console.log: new intermediate certificate added
console.log: signatures re-verified

on ESR 60.6 I got some browser console logs about it being disabled (didn't have the legacy addon pref enabled)

Attachment #9063342 - Attachment is obsolete: true

Same as XPI in comment 43 with the differences:

  • signed with autograph 3.2.0
  • bumped the version number to 1.1.1 in install.rdf

tested against ESR 52 with a fresh profile

Attachment #9063369 - Attachment is obsolete: true

(In reply to Greg Guthe [:g-k] [:gguthe] from comment #44)

Created attachment 9063370 [details]
hotfix-update-xpi-signing-intermediate-bug-1548973-legacy@mozilla.org-1.1.1-signed.xpi

Same as XPI in comment 43 with the differences:

  • signed with autograph 3.2.0
  • bumped the version number to 1.1.1 in install.rdf

tested against ESR 52 with a fresh profile

This file passed the signature verification!

1557310941173	addons.xpi	WARN	disabling legacy extension hotfix-bug-1548973@mozilla.org

I assume that it will run without flipping the pref if it is deployed as a system add-on.

The last one works great, thanks!

I've made a last-minute change to account for the "master password" workaround that landed in tree since I wrote this, would you mind signing this one too? Thanks!

Attachment #9063339 - Attachment is obsolete: true
Attachment #9063370 - Attachment is obsolete: true
Flags: needinfo?(gguthe)

shortened the id to fit inside the 64 char limit

Flags: needinfo?(gguthe)

checked that https://bugzilla.mozilla.org/attachment.cgi?id=9063553 installs on ESR 52

Comment on attachment 9063553 [details]
hotfix-update-xpi-signing-intermediate-bug-1548973-legacy@mozilla.org-1.1.2-signed.xpi

Rehan, could you please stage this on Balrog for releases 52 through 60 inclusive? Thanks!

Flags: needinfo?(rdalal)
Attachment #9063553 - Attachment is obsolete: true

This is now on release-sysaddon for 52-60 inclusive.

Flags: needinfo?(rdalal)

I have also staged this on the release channel for 52-60 where it is pending relman signoff once QA is happy.

is there are a particular technical reason, why this is targetting 52 and upwards?

i'm asking because we've also seen a couple of OSX 10.6-10.8 users stranded on firefox 48 [1] inquiring what's the remedy for them in support channels. if there are no technical hurdles, extending this system addon to versions 48 upwards would therefore be appreciated.

(incidentally firefox 48 was also the first version where xpinstall.signatures.required didn't take effect any longer and affected users don't have an easy workaround [2])

[1] https://support.mozilla.org/en-US/kb/firefox-osx
[2] https://wiki.mozilla.org/Add-ons/Extension_Signing#Timeline

(In reply to [:philipp] from comment #55)

is there are a particular technical reason, why this is targetting 52 and upwards?

i'm asking because we've also seen a couple of OSX 10.6-10.8 users stranded on firefox 48 [1] inquiring what's the remedy for them in support channels. if there are no technical hurdles, extending this system addon to versions 48 upwards would therefore be appreciated.

(incidentally firefox 48 was also the first version where xpinstall.signatures.required didn't take effect any longer and affected users don't have an easy workaround)

[1] https://support.mozilla.org/en-US/kb/firefox-osx
[2] https://wiki.mozilla.org/Add-ons/Extension_Signing#Timeline

I'm not sure if the current extension will work back to 48, and we have limited resources (dev, QA) so unsupported releases tend to be best-effort.

52 in particular was chosen because it is the previous ESR, and the last release that supported Windows XP. Going back much further is technically challenging for Mozilla (see above) and likely to be a vanishingly small set of users.

We may be able to provide docs so users could attempt to patch it themselves, but we're putting more effort on 52+ for the moment.

Is this on the release-sysaddon channel, and you have extensions.logging.enabled to true?

None of these errors are related, and I am surprised to not see Starting system add-on update check from [...] in the log output.

Do you have extensions.systemAddon.update.enabled set to true (the default)? If you don't have that, then ensure that automatic updates are enabled (I don't recall offhand which release this was added in)

(In reply to Masatoshi Kimura [:emk] from comment #57)

When I try to run update by using

Cu.import("resource://gre/modules/AddonManager.jsm");
AddonManagerPrivate.backgroundUpdateCheck();

on 56.0.2, I got this error:

1557354635811	addons.manager	DEBUG	Background update check beginning
1557354635817	addons.repository	DEBUG	Repopulate add-on cache with ["activity-stream@mozilla.org", "aushelper@mozilla.org", "clicktoplay-rollout@mozilla.org", "e10srollout@mozilla.org", "firefox@getpocket.com", "followonsearch@mozilla.com", "formautofill@mozilla.org", "onboarding@mozilla.org", "screenshots@mozilla.org", "shield-recipe-client@mozilla.org", "webcompat@mozilla.org", "{972ce4c6-7e08-4474-a285-3208198ce6fd}", "firefox-compact-light@mozilla.org@personas.mozilla.org", "firefox-compact-dark@mozilla.org@personas.mozilla.org", "gmp-gmpopenh264", "gmp-widevinecdm"]
1557354635823	addons.repository	DEBUG	Requesting https://services.addons.mozilla.org/ja/firefox/api/1.5/search/guid:%7B972ce4c6-7e08-4474-a285-3208198ce6fd%7D,firefox-compact-light%40mozilla.org%40personas.mozilla.org,firefox-compact-dark%40mozilla.org%40personas.mozilla.org?src=firefox&appOS=WINNT&appVersion=56.0.2&tMain=91&tFirstPaint=1494&tSessionRestored=2292
XML パースエラー: タグの対応が間違っています。終了タグが必要です: </link>
URL: https://services.addons.mozilla.org/ja/firefox/api/1.5/search/guid:%7B972ce4c6-7e08-4474-a285-3208198ce6fd%7D,firefox-compact-light%40mozilla.org%40personas.mozilla.org,firefox-compact-dark%40mozilla.org%40personas.mozilla.org?src=firefox&appOS=WINNT&appVersion=56.0.2&tMain=91&tFirstPaint=1494&tSessionRestored=2292
行番号: 20, 列番号: 125:  guid:%7B972ce4c6-7e08-4474-a285-3208198ce6fd%7D,firefox-compact-light%40mozilla.org%40personas.mozilla.org,firefox-compact-dark%40mozilla.org%40personas.mozilla.org:20:125
1557354635978	addons.repository	DEBUG	Got metadata search load event
1557354635978	addons.repository	WARN	Search failed when repopulating cache
1557354635980	addons.xpi	DEBUG	updateAddonRepositoryData found 12 visible add-ons
1557354635984	addons.update-checker	DEBUG	Requesting https://versioncheck-bg.addons.mozilla.org/update/VersionCheck.php?reqVersion=2&id=activity-stream@mozilla.org&version=0.0.0&maxAppVersion=56.*&status=userEnabled&appID={ec8030f7-c20a-464f-9b0e-13a3a9e97384}&appVersion=56.0.2&appOS=WINNT&appABI=x86_64-msvc&locale=ja&currentAppVersion=56.0.2&updateType=112&compatMode=normal
1557354635987	addons.update-checker	DEBUG	Requesting https://versioncheck-bg.addons.mozilla.org/update/VersionCheck.php?reqVersion=2&id=aushelper@mozilla.org&version=2.0&maxAppVersion=56.*&status=userEnabled&appID={ec8030f7-c20a-464f-9b0e-13a3a9e97384}&appVersion=56.0.2&appOS=WINNT&appABI=x86_64-msvc&locale=ja&currentAppVersion=56.0.2&updateType=112&compatMode=normal
1557354635990	addons.update-checker	DEBUG	Requesting https://versioncheck-bg.addons.mozilla.org/update/VersionCheck.php?reqVersion=2&id=clicktoplay-rollout@mozilla.org&version=1.4&maxAppVersion=56.*&status=userEnabled&appID={ec8030f7-c20a-464f-9b0e-13a3a9e97384}&appVersion=56.0.2&appOS=WINNT&appABI=x86_64-msvc&locale=ja&currentAppVersion=56.0.2&updateType=112&compatMode=normal
1557354635993	addons.update-checker	DEBUG	Requesting https://versioncheck-bg.addons.mozilla.org/update/VersionCheck.php?reqVersion=2&id=e10srollout@mozilla.org&version=3.00&maxAppVersion=56.*&status=userEnabled&appID={ec8030f7-c20a-464f-9b0e-13a3a9e97384}&appVersion=56.0.2&appOS=WINNT&appABI=x86_64-msvc&locale=ja&currentAppVersion=56.0.2&updateType=112&compatMode=normal
1557354635996	addons.update-checker	DEBUG	Requesting https://versioncheck-bg.addons.mozilla.org/update/VersionCheck.php?reqVersion=2&id=firefox@getpocket.com&version=1.0.5&maxAppVersion=56.*&status=userEnabled&appID={ec8030f7-c20a-464f-9b0e-13a3a9e97384}&appVersion=56.0.2&appOS=WINNT&appABI=x86_64-msvc&locale=ja&currentAppVersion=56.0.2&updateType=112&compatMode=normal
1557354635999	addons.update-checker	DEBUG	Requesting https://versioncheck-bg.addons.mozilla.org/update/VersionCheck.php?reqVersion=2&id=followonsearch@mozilla.com&version=0.9.3&maxAppVersion=59.*&status=userEnabled&appID={ec8030f7-c20a-464f-9b0e-13a3a9e97384}&appVersion=56.0.2&appOS=WINNT&appABI=x86_64-msvc&locale=ja&currentAppVersion=56.0.2&updateType=112&compatMode=normal
1557354636001	addons.update-checker	DEBUG	Requesting https://versioncheck-bg.addons.mozilla.org/update/VersionCheck.php?reqVersion=2&id=formautofill@mozilla.org&version=1.0&maxAppVersion=56.*&status=userEnabled&appID={ec8030f7-c20a-464f-9b0e-13a3a9e97384}&appVersion=56.0.2&appOS=WINNT&appABI=x86_64-msvc&locale=ja&currentAppVersion=56.0.2&updateType=112&compatMode=normal
1557354636005	addons.update-checker	DEBUG	Requesting https://versioncheck-bg.addons.mozilla.org/update/VersionCheck.php?reqVersion=2&id=onboarding@mozilla.org&version=0.1&maxAppVersion=56.*&status=userEnabled&appID={ec8030f7-c20a-464f-9b0e-13a3a9e97384}&appVersion=56.0.2&appOS=WINNT&appABI=x86_64-msvc&locale=ja&currentAppVersion=56.0.2&updateType=112&compatMode=normal
1557354636011	addons.update-checker	DEBUG	Requesting https://versioncheck-bg.addons.mozilla.org/update/VersionCheck.php?reqVersion=2&id=screenshots@mozilla.org&version=10.12.0&maxAppVersion=*&status=userEnabled&appID={ec8030f7-c20a-464f-9b0e-13a3a9e97384}&appVersion=56.0.2&appOS=WINNT&appABI=x86_64-msvc&locale=ja&currentAppVersion=56.0.2&updateType=112&compatMode=normal
1557354636012	addons.update-checker	DEBUG	Requesting https://versioncheck-bg.addons.mozilla.org/update/VersionCheck.php?reqVersion=2&id=shield-recipe-client@mozilla.org&version=65&maxAppVersion=56.*&status=userEnabled&appID={ec8030f7-c20a-464f-9b0e-13a3a9e97384}&appVersion=56.0.2&appOS=WINNT&appABI=x86_64-msvc&locale=ja&currentAppVersion=56.0.2&updateType=112&compatMode=normal
1557354636013	addons.update-checker	DEBUG	Requesting https://versioncheck-bg.addons.mozilla.org/update/VersionCheck.php?reqVersion=2&id=webcompat@mozilla.org&version=1.1&maxAppVersion=56.*&status=userEnabled&appID={ec8030f7-c20a-464f-9b0e-13a3a9e97384}&appVersion=56.0.2&appOS=WINNT&appABI=x86_64-msvc&locale=ja&currentAppVersion=56.0.2&updateType=112&compatMode=normal
1557354636014	addons.update-checker	DEBUG	Requesting https://versioncheck-bg.addons.mozilla.org/update/VersionCheck.php?reqVersion=2&id={972ce4c6-7e08-4474-a285-3208198ce6fd}&version=56.0.2&maxAppVersion=56.0.2&status=userEnabled&appID={ec8030f7-c20a-464f-9b0e-13a3a9e97384}&appVersion=56.0.2&appOS=WINNT&appABI=x86_64-msvc&locale=ja&currentAppVersion=56.0.2&updateType=112&compatMode=normal
1557354636015	addons.manager	DEBUG	onUpdateFinished for firefox-compact-light@mozilla.org@personas.mozilla.org
1557354636015	addons.manager	DEBUG	onUpdateFinished for firefox-compact-dark@mozilla.org@personas.mozilla.org
1557354636016	addons.manager	DEBUG	onUpdateFinished for gmp-gmpopenh264
1557354636016	addons.manager	DEBUG	onUpdateFinished for gmp-widevinecdm
1557354636584	addons.update-checker	WARN	onUpdateCheckComplete failed to parse update manifest: [Exception... "Update manifest is missing a required addons property."  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: resource://gre/modules/addons/AddonUpdateChecker.jsm :: getRequiredProperty :: line 465"  data: no] Stack trace: getRequiredProperty()@resource://gre/modules/addons/AddonUpdateChecker.jsm:465 < parseJSONManifest()@resource://gre/modules/addons/AddonUpdateChecker.jsm:475 < parser()@resource://gre/modules/addons/AddonUpdateChecker.jsm:635 < onLoad()@resource://gre/modules/addons/AddonUpdateChecker.jsm:655 < UpdateParser/<()@resource://gre/modules/addons/AddonUpdateChecker.jsm:580
1557354636585	addons.manager	DEBUG	onUpdateFinished for activity-stream@mozilla.org
1557354636599	addons.update-checker	WARN	onUpdateCheckComplete failed to parse update manifest: [Exception... "Update manifest is missing a required addons property."  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: resource://gre/modules/addons/AddonUpdateChecker.jsm :: getRequiredProperty :: line 465"  data: no] Stack trace: getRequiredProperty()@resource://gre/modules/addons/AddonUpdateChecker.jsm:465 < parseJSONManifest()@resource://gre/modules/addons/AddonUpdateChecker.jsm:475 < parser()@resource://gre/modules/addons/AddonUpdateChecker.jsm:635 < onLoad()@resource://gre/modules/addons/AddonUpdateChecker.jsm:655 < UpdateParser/<()@resource://gre/modules/addons/AddonUpdateChecker.jsm:580
1557354636599	addons.manager	DEBUG	onUpdateFinished for aushelper@mozilla.org
1557354636610	addons.update-checker	WARN	onUpdateCheckComplete failed to parse update manifest: [Exception... "Update manifest is missing a required addons property."  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: resource://gre/modules/addons/AddonUpdateChecker.jsm :: getRequiredProperty :: line 465"  data: no] Stack trace: getRequiredProperty()@resource://gre/modules/addons/AddonUpdateChecker.jsm:465 < parseJSONManifest()@resource://gre/modules/addons/AddonUpdateChecker.jsm:475 < parser()@resource://gre/modules/addons/AddonUpdateChecker.jsm:635 < onLoad()@resource://gre/modules/addons/AddonUpdateChecker.jsm:655 < UpdateParser/<()@resource://gre/modules/addons/AddonUpdateChecker.jsm:580
1557354636610	addons.manager	DEBUG	onUpdateFinished for clicktoplay-rollout@mozilla.org
1557354636617	addons.update-checker	WARN	onUpdateCheckComplete failed to parse update manifest: [Exception... "Update manifest is missing a required addons property."  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: resource://gre/modules/addons/AddonUpdateChecker.jsm :: getRequiredProperty :: line 465"  data: no] Stack trace: getRequiredProperty()@resource://gre/modules/addons/AddonUpdateChecker.jsm:465 < parseJSONManifest()@resource://gre/modules/addons/AddonUpdateChecker.jsm:475 < parser()@resource://gre/modules/addons/AddonUpdateChecker.jsm:635 < onLoad()@resource://gre/modules/addons/AddonUpdateChecker.jsm:655 < UpdateParser/<()@resource://gre/modules/addons/AddonUpdateChecker.jsm:580
1557354636617	addons.manager	DEBUG	onUpdateFinished for e10srollout@mozilla.org
1557354636624	addons.update-checker	WARN	onUpdateCheckComplete failed to parse update manifest: [Exception... "Update manifest is missing a required addons property."  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: resource://gre/modules/addons/AddonUpdateChecker.jsm :: getRequiredProperty :: line 465"  data: no] Stack trace: getRequiredProperty()@resource://gre/modules/addons/AddonUpdateChecker.jsm:465 < parseJSONManifest()@resource://gre/modules/addons/AddonUpdateChecker.jsm:475 < parser()@resource://gre/modules/addons/AddonUpdateChecker.jsm:635 < onLoad()@resource://gre/modules/addons/AddonUpdateChecker.jsm:655 < UpdateParser/<()@resource://gre/modules/addons/AddonUpdateChecker.jsm:580
1557354636625	addons.manager	DEBUG	onUpdateFinished for firefox@getpocket.com
1557354636636	addons.update-checker	WARN	onUpdateCheckComplete failed to parse update manifest: [Exception... "Update manifest is missing a required addons property."  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: resource://gre/modules/addons/AddonUpdateChecker.jsm :: getRequiredProperty :: line 465"  data: no] Stack trace: getRequiredProperty()@resource://gre/modules/addons/AddonUpdateChecker.jsm:465 < parseJSONManifest()@resource://gre/modules/addons/AddonUpdateChecker.jsm:475 < parser()@resource://gre/modules/addons/AddonUpdateChecker.jsm:635 < onLoad()@resource://gre/modules/addons/AddonUpdateChecker.jsm:655 < UpdateParser/<()@resource://gre/modules/addons/AddonUpdateChecker.jsm:580
1557354636636	addons.manager	DEBUG	onUpdateFinished for followonsearch@mozilla.com
1557354636695	addons.update-checker	WARN	onUpdateCheckComplete failed to parse update manifest: [Exception... "Update manifest is missing a required addons property."  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: resource://gre/modules/addons/AddonUpdateChecker.jsm :: getRequiredProperty :: line 465"  data: no] Stack trace: getRequiredProperty()@resource://gre/modules/addons/AddonUpdateChecker.jsm:465 < parseJSONManifest()@resource://gre/modules/addons/AddonUpdateChecker.jsm:475 < parser()@resource://gre/modules/addons/AddonUpdateChecker.jsm:635 < onLoad()@resource://gre/modules/addons/AddonUpdateChecker.jsm:655 < UpdateParser/<()@resource://gre/modules/addons/AddonUpdateChecker.jsm:580
1557354636695	addons.manager	DEBUG	onUpdateFinished for formautofill@mozilla.org
1557354636721	addons.update-checker	WARN	onUpdateCheckComplete failed to parse update manifest: [Exception... "Update manifest is missing a required addons property."  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: resource://gre/modules/addons/AddonUpdateChecker.jsm :: getRequiredProperty :: line 465"  data: no] Stack trace: getRequiredProperty()@resource://gre/modules/addons/AddonUpdateChecker.jsm:465 < parseJSONManifest()@resource://gre/modules/addons/AddonUpdateChecker.jsm:475 < parser()@resource://gre/modules/addons/AddonUpdateChecker.jsm:635 < onLoad()@resource://gre/modules/addons/AddonUpdateChecker.jsm:655 < UpdateParser/<()@resource://gre/modules/addons/AddonUpdateChecker.jsm:580
1557354636722	addons.manager	DEBUG	onUpdateFinished for onboarding@mozilla.org
1557354636731	addons.manager	DEBUG	onUpdateFinished for shield-recipe-client@mozilla.org
1557354636738	addons.manager	DEBUG	onUpdateFinished for screenshots@mozilla.org
1557354636745	addons.update-checker	WARN	onUpdateCheckComplete failed to parse update manifest: [Exception... "Update manifest is missing a required addons property."  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: resource://gre/modules/addons/AddonUpdateChecker.jsm :: getRequiredProperty :: line 465"  data: no] Stack trace: getRequiredProperty()@resource://gre/modules/addons/AddonUpdateChecker.jsm:465 < parseJSONManifest()@resource://gre/modules/addons/AddonUpdateChecker.jsm:475 < parser()@resource://gre/modules/addons/AddonUpdateChecker.jsm:635 < onLoad()@resource://gre/modules/addons/AddonUpdateChecker.jsm:655 < UpdateParser/<()@resource://gre/modules/addons/AddonUpdateChecker.jsm:580
1557354636745	addons.manager	DEBUG	onUpdateFinished for webcompat@mozilla.org
1557354636779	addons.update-checker	WARN	onUpdateCheckComplete failed to parse update manifest: [Exception... "Update manifest is missing a required addons property."  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: resource://gre/modules/addons/AddonUpdateChecker.jsm :: getRequiredProperty :: line 465"  data: no] Stack trace: getRequiredProperty()@resource://gre/modules/addons/AddonUpdateChecker.jsm:465 < parseJSONManifest()@resource://gre/modules/addons/AddonUpdateChecker.jsm:475 < parser()@resource://gre/modules/addons/AddonUpdateChecker.jsm:635 < onLoad()@resource://gre/modules/addons/AddonUpdateChecker.jsm:655 < UpdateParser/<()@resource://gre/modules/addons/AddonUpdateChecker.jsm:580
1557354636780	addons.manager	DEBUG	onUpdateFinished for {972ce4c6-7e08-4474-a285-3208198ce6fd}
1557354636780	addons.manager	DEBUG	Background update check complete
Blocklist::notify: Requesting https://blocklists.settings.services.mozilla.com/v1/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/56.0.2/Firefox/20171024165158/WINNT_x86_64-msvc/ja/release/Windows_NT%2010.0/default/default/1/1/new/
1557354667074	DeferredSave.extensions.json	DEBUG	Save changes
Blocklist state for activity-stream@mozilla.org changed from 0 to 0
1557354667078	DeferredSave.extensions.json	DEBUG	Save changes
Blocklist state for aushelper@mozilla.org changed from 0 to 0
1557354667079	DeferredSave.extensions.json	DEBUG	Save changes
Blocklist state for clicktoplay-rollout@mozilla.org changed from 0 to 0
1557354667080	DeferredSave.extensions.json	DEBUG	Save changes
Blocklist state for e10srollout@mozilla.org changed from 0 to 0
1557354667081	DeferredSave.extensions.json	DEBUG	Save changes
Blocklist state for firefox@getpocket.com changed from 0 to 0
1557354667083	DeferredSave.extensions.json	DEBUG	Save changes
Blocklist state for followonsearch@mozilla.com changed from 0 to 0
1557354667084	DeferredSave.extensions.json	DEBUG	Save changes
Blocklist state for formautofill@mozilla.org changed from 0 to 0
1557354667085	DeferredSave.extensions.json	DEBUG	Save changes
Blocklist state for onboarding@mozilla.org changed from 0 to 0
1557354667086	DeferredSave.extensions.json	DEBUG	Save changes
Blocklist state for screenshots@mozilla.org changed from 0 to 0
1557354667087	DeferredSave.extensions.json	DEBUG	Save changes
Blocklist state for shield-recipe-client@mozilla.org changed from 0 to 0
1557354667089	DeferredSave.extensions.json	DEBUG	Save changes
Blocklist state for webcompat@mozilla.org changed from 0 to 0
1557354667090	DeferredSave.extensions.json	DEBUG	Save changes
Blocklist state for {972ce4c6-7e08-4474-a285-3208198ce6fd} changed from 0 to 0
Blocklist state for firefox-compact-light@mozilla.org@personas.mozilla.org changed from 0 to 0
Blocklist state for firefox-compact-dark@mozilla.org@personas.mozilla.org changed from 0 to 0
1557354667098	DeferredSave.extensions.json	DEBUG	Starting timer
1557354667128	DeferredSave.extensions.json	DEBUG	Starting write
1557354667269	DeferredSave.extensions.json	DEBUG	Write succeeded
1557354667270	addons.xpi-utils	DEBUG	XPI Database saved, setting schema version preference to 22

and no system add-ons are installed.

Note that https://services.addons.mozilla.org/ja/firefox/api/1.5/search/guid:%7B972ce4c6-7e08-4474-a285-3208198ce6fd%7D,firefox-compact-light%40mozilla.org%40personas.mozilla.org,firefox-compact-dark%40mozilla.org%40personas.mozilla.org?src=firefox&appOS=WINNT&appVersion=56.0.2&tMain=91&tFirstPaint=1494&tSessionRestored=2292 no longer serves an update XML.

Flags: needinfo?(VYV03354)

Ah, sorry, I didn't change the channel to release-sysaddon.
I will go to work now, So I can't retest until I go home.

Flags: needinfo?(VYV03354)

Hm. So, this fix doesn't seem to be working on 52:

1557356494785	addons.xpi	INFO	Starting system add-on update check from https://aus5.mozilla.org/update/3/SystemAddons/52.0.2/20170323105023/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release-sysaddon/Darwin%2018.5.0/default/default/update.xml.
1557356494785	addons.productaddons	INFO	sending request to: https://aus5.mozilla.org/update/3/SystemAddons/52.0.2/20170323105023/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release-sysaddon/Darwin%2018.5.0/default/default/update.xml
1557356496405	addons.xpi	WARN	System add-on baidu-code-update@mozillaonline.com isn't compatible with the application.
1557356496406	addons.manager	WARN	Failed to update system addons: Error: Rejecting updated system add-on set that either could not be downloaded or contained unusable add-ons. (resource://gre/modules/addons/XPIProvider.jsm:3228:13) JS Stack trace: this.XPIProvider.updateSystemAddons<@XPIProvider.jsm:3228:13

We should pull that one from any version it isn't intended for, since a broken add-on in the set will reject the whole set.

Rehan, is there a great way to do this with Balrog or should we look into temporarily disabling this?

Flags: needinfo?(rdalal)

I notice that the following snippet is still in the hotfix:

        // Buy us some time to fix the bug, but with a fixed deadline (2 weeks).
        let now = Date.now();
        let deadline = Date.UTC(2019, 4, 20);
        if (now < deadline) {
            // lastUpdateTime is in seconds, not milliseconds.
            now = Math.round(now / 1000);
            console.info("Suppressing scheduled signature verification check"); // eslint-disable-line no-console
            Services.prefs.setIntPref("app.update.lastUpdateTime.xpi-signature-verification", now);
        }
        return;

This logic is triggered when the hotfix fails to add the certificate. It was added in bug 1549400, to prevent add-ons from being disabled until we had a definite patch for the problem. The fixed deadline exists to prevent the signed add-on from becoming a permanent bypass to the signature verification system.

If this new add-on is expected to provide a fix against all known issues, the above snippet is redundant.

If you are going to update the add-on again, then I suggest to take out the snippet.
If not, then it is not a big deal to keep it in, because the logic is bound to be disabled in 11 days from now.

(In reply to Robert Helmer [:rhelmer] from comment #60)

1557356496405 addons.xpi WARN System add-on baidu-code-update@mozillaonline.com isn't compatible with the application.

It is not this hotfix. It is Baidu Search Update from bug 1541316.

We should pull that one from any version it isn't intended for, since a broken add-on in the set will reject the whole set.

Rehan, is there a great way to do this with Balrog or should we look into temporarily disabling this?

So Baidu Search Update prevent this hotfix (and all other system add-ons) from installing.

Moreover, Baidu Search Update will not be installed unless this hotfix is installed beforehand due to armagadd-on-2.0. We should temporary pull Baidu Search Update rather than this hotfix.

I edited comment 62 because the bug number was wrong.

Rehan, is there a great way to do this with Balrog or should we look into temporarily disabling this?

I could update the rules for 52-60 to remove the Baidu SAO. I don't know any other way to deal with this in Balrog.

Flags: needinfo?(rdalal)

(In reply to Masatoshi Kimura [:emk] from comment #63)

We should temporary pull Baidu Search Update rather than this hotfix.

Or re-sign the Baidu Search Update so that it works without this hotfix.

Right, that should also work.

While testing this we've noticed that we did not get the hotfix for 53/55/56 and 59 (these are the ones we tried). For 59 we got the baidu error but for the other ones I did not get that. Here is the browser console output for them https://docs.google.com/document/d/1bU5VGpWSMXNHN65O0n3DDm8L7Fbvl4BjlBw22yzEgeo/edit. Am I missing something?

Flags: needinfo?(rhelmer)

(In reply to Bogdan Maris [:bogdan_maris], Release Desktop QA from comment #68)

While testing this we've noticed that we did not get the hotfix for 53/55/56 and 59 (these are the ones we tried).

Which "hotfix" do you mean?
I did successfully install attachment #9063370 [details] on Firefox 56.0.2 and all my addons have been re-enabled after a restart...

(In reply to Robert Helmer [:rhelmer] from comment #58)

Is this on the release-sysaddon channel, and you have extensions.logging.enabled to true?

Even if I set the update channel to release-sysaddon and set extensions.logging.enabled to true, I did not get log output about SAO update.

If I run

var { XPIProvider } = Components.utils.import("resource://gre/modules/addons/XPIProvider.jsm", {});
XPIProvider.updateSystemAddons();

directly, I got a similar output to comment #60. So something must have prevented XPIProvider.updateSystemAddons(); from running.

(In reply to Sven Giermann from comment #69)

(In reply to Bogdan Maris [:bogdan_maris], Release Desktop QA from comment #68)

While testing this we've noticed that we did not get the hotfix for 53/55/56 and 59 (these are the ones we tried).

Which "hotfix" do you mean?
I did successfully install attachment #9063370 [details] on Firefox 56.0.2 and all my addons have been re-enabled after a restart...

We know we can manually install the .xpi to 52-56 because legacy add-ons are still working on those versions.
We are currently testing deployment the .xpi as a system add-on.

How about deploying this hotfix to 53-56 via normandy? It would be much easier than deploying as a SAO. And some users are stuck at 56 for obvious reasons.

(In reply to Masatoshi Kimura [:emk] from comment #72)

How about deploying this hotfix to 53-56 via normandy? It would be much easier than deploying as a SAO. And some users are stuck at 56 for obvious reasons.

Normandy is not an option for these older releases.

Flags: needinfo?(rhelmer)

(In reply to Bogdan Maris [:bogdan_maris], Release Desktop QA from comment #68)

While testing this we've noticed that we did not get the hotfix for 53/55/56 and 59 (these are the ones we tried). For 59 we got the baidu error but for the other ones I did not get that. Here is the browser console output for them https://docs.google.com/document/d/1bU5VGpWSMXNHN65O0n3DDm8L7Fbvl4BjlBw22yzEgeo/edit. Am I missing something?

Thanks for the log. I don't see any attempted system add-on check in this output; please check that you have these enabled in about:config:

app.update.auto
extensions.systemAddon.update.enabled (if present)
extensions.logging.enabled

I'll go ahead and test these versions too.

Flags: needinfo?(bogdan.maris)

(In reply to Masatoshi Kimura [:emk] from comment #70)

(In reply to Robert Helmer [:rhelmer] from comment #58)

Is this on the release-sysaddon channel, and you have extensions.logging.enabled to true?

Even if I set the update channel to release-sysaddon and set extensions.logging.enabled to true, I did not get log output about SAO update.

If I run

var { XPIProvider } = Components.utils.import("resource://gre/modules/addons/XPIProvider.jsm", {});
XPIProvider.updateSystemAddons();

directly, I got a similar output to comment #60. So something must have prevented XPIProvider.updateSystemAddons(); from running.

Please try:

Components.utils.import("resource://gre/modules/AddonManager.jsm"); AddonManagerPrivate.backgroundUpdateCheck();
Flags: needinfo?(VYV03354)

(In reply to Robert Helmer [:rhelmer] from comment #74)

(In reply to Bogdan Maris [:bogdan_maris], Release Desktop QA from comment #68)

While testing this we've noticed that we did not get the hotfix for 53/55/56 and 59 (these are the ones we tried). For 59 we got the baidu error but for the other ones I did not get that. Here is the browser console output for them https://docs.google.com/document/d/1bU5VGpWSMXNHN65O0n3DDm8L7Fbvl4BjlBw22yzEgeo/edit. Am I missing something?

Thanks for the log. I don't see any attempted system add-on check in this output; please check that you have these enabled in about:config:

app.update.auto
extensions.systemAddon.update.enabled (if present)
extensions.logging.enabled

I'll go ahead and test these versions too.

This works for me on 53, but I still see the Baidu error.

(In reply to Masatoshi Kimura [:emk] from comment #66)

(In reply to Masatoshi Kimura [:emk] from comment #63)

We should temporary pull Baidu Search Update rather than this hotfix.

Or re-sign the Baidu Search Update so that it works without this hotfix.

OK I think the problem here is that Baidu Search Update needs to be signed so it works with older releases, as :gguthe ran into in this bug. I suspect the hotfix alone will not help here, since system add-ons cannot be signed with the intermediate cert that expired, they use the "Mozilla Components" one instead.

Rehan is going to pull the legacy Baidu Search Update.

(In reply to Robert Helmer [:rhelmer] from comment #75)

(In reply to Masatoshi Kimura [:emk] from comment #70)

(In reply to Robert Helmer [:rhelmer] from comment #58)

Is this on the release-sysaddon channel, and you have extensions.logging.enabled to true?

Even if I set the update channel to release-sysaddon and set extensions.logging.enabled to true, I did not get log output about SAO update.

If I run

var { XPIProvider } = Components.utils.import("resource://gre/modules/addons/XPIProvider.jsm", {});
XPIProvider.updateSystemAddons();

directly, I got a similar output to comment #60. So something must have prevented XPIProvider.updateSystemAddons(); from running.

Please try:

Components.utils.import("resource://gre/modules/AddonManager.jsm"); AddonManagerPrivate.backgroundUpdateCheck();

Sorry for not being explicit, I tried that command after I changed the settings. But I only get a similar log to comment #57 (that is, no SAO related log at all).

Flags: needinfo?(VYV03354)

(In reply to Masatoshi Kimura [:emk] from comment #78)

(In reply to Robert Helmer [:rhelmer] from comment #75)

(In reply to Masatoshi Kimura [:emk] from comment #70)

(In reply to Robert Helmer [:rhelmer] from comment #58)

Is this on the release-sysaddon channel, and you have extensions.logging.enabled to true?

Even if I set the update channel to release-sysaddon and set extensions.logging.enabled to true, I did not get log output about SAO update.

If I run

var { XPIProvider } = Components.utils.import("resource://gre/modules/addons/XPIProvider.jsm", {});
XPIProvider.updateSystemAddons();

directly, I got a similar output to comment #60. So something must have prevented XPIProvider.updateSystemAddons(); from running.

Please try:

Components.utils.import("resource://gre/modules/AddonManager.jsm"); AddonManagerPrivate.backgroundUpdateCheck();

Sorry for not being explicit, I tried that command after I changed the settings. But I only get a similar log to comment #57 (that is, no SAO related log at all).

Hm. I tried that version and it wfm, which OS? The fact that it's not being logged doesn't make much sense, would you mind sending the whole Browser Console log (make sure it's the Browser and not the normal devtools console!)

Flags: needinfo?(VYV03354)

These addons were also pulled from the test channel:

  • tls13-version-fallback-rollout-bug1462099@mozilla.org
  • google-code-correction@mozilla.org
  • telemetry-coverage-bug1487578@mozilla.org

(In reply to Robert Helmer [:rhelmer] from comment #74)

(In reply to Bogdan Maris [:bogdan_maris], Release Desktop QA from comment #68)

While testing this we've noticed that we did not get the hotfix for 53/55/56 and 59 (these are the ones we tried). For 59 we got the baidu error but for the other ones I did not get that. Here is the browser console output for them https://docs.google.com/document/d/1bU5VGpWSMXNHN65O0n3DDm8L7Fbvl4BjlBw22yzEgeo/edit. Am I missing something?

Thanks for the log. I don't see any attempted system add-on check in this output; please check that you have these enabled in about:config:

app.update.auto
extensions.systemAddon.update.enabled (if present)
extensions.logging.enabled

I'll go ahead and test these versions too.

For some reason it worked now for 53, 52 - I had those prefs to true before when it did not bring the hotfix (tested on macOS 13).

On 59 I don't receive the baidu error but a different one (could be the same reason though). Here is the browser console output: https://docs.google.com/document/d/1ROyOh_2o7j1Tfsv_DeL0VEBFogKvbnDbb-aDjByErrI/edit.
Also does this apply to ESR builds as well or only RC builds?

Flags: needinfo?(bogdan.maris) → needinfo?(rhelmer)
Attached file log.txt (obsolete) —

(In reply to Robert Helmer [:rhelmer] from comment #79)

Hm. I tried that version and it wfm, which OS?

Windows 10 IP 20H1 build 18894.

The fact that it's not being logged doesn't make much sense, would you mind sending the whole Browser Console log (make sure it's the Browser and not the normal devtools console!)

Attached the whole Ctrl+Shift+J Browser Console log from the startup.

Flags: needinfo?(VYV03354)

(In reply to Robert Helmer [:rhelmer] from comment #74)

app.update.auto

Ah, it was the reason I didn't receive the SAO update. I disabled the auto update because otherwise Firefox itself updates to the latest version and I have to re-install every time I test.

(But then I doubt the usefulness of the SAO. Most old version users must have disabled the auto update.)

(In reply to Masatoshi Kimura [:emk] from comment #83)

(In reply to Robert Helmer [:rhelmer] from comment #74)

app.update.auto

Ah, it was the reason I didn't receive the SAO update. I disabled the auto update because otherwise Firefox itself updates to the latest version and I have to re-install every time I test.

(But then I doubt the usefulness of the SAO. Most old version users must have disabled the auto update.)

It would still be relevant to XP/Vista users on 52 ESR who would not have needed to change the default setting.

Can the extension be offered somewhere appropriate as user-installable now to reduce the waiting time?

(In reply to jscher2000 from comment #84)

(In reply to Masatoshi Kimura [:emk] from comment #83)

(In reply to Robert Helmer [:rhelmer] from comment #74)

app.update.auto

Ah, it was the reason I didn't receive the SAO update. I disabled the auto update because otherwise Firefox itself updates to the latest version and I have to re-install every time I test.

(But then I doubt the usefulness of the SAO. Most old version users must have disabled the auto update.)

It would still be relevant to XP/Vista users on 52 ESR who would not have needed to change the default setting.

Can the extension be offered somewhere appropriate as user-installable now to reduce the waiting time?

Yes I am going to file a separate bug for this now, since this current one is getting mired in update server configuration issues etc.

Flags: needinfo?(rhelmer)
See Also: → 1550793

(In reply to Robert Helmer [:rhelmer] from comment #85)

(In reply to jscher2000 from comment #84)

(In reply to Masatoshi Kimura [:emk] from comment #83)

(In reply to Robert Helmer [:rhelmer] from comment #74)

app.update.auto

Ah, it was the reason I didn't receive the SAO update. I disabled the auto update because otherwise Firefox itself updates to the latest version and I have to re-install every time I test.

(But then I doubt the usefulness of the SAO. Most old version users must have disabled the auto update.)

It would still be relevant to XP/Vista users on 52 ESR who would not have needed to change the default setting.

Can the extension be offered somewhere appropriate as user-installable now to reduce the waiting time?

Yes I am going to file a separate bug for this now, since this current one is getting mired in update server configuration issues etc.

Bug 1550793.

do we know the QA status of the add-on alone? looking for when the add-on is OK to list on AMO. It's hard to tell the QA status for the add-on alone.... because initially that bug covered the add-on and the balrog delivery. balrog delivery was just broken into separate bug https://bugzilla.mozilla.org/show_bug.cgi?id=1550793 .

We are ready to list the add-on on AMO as soon as we know the add-on itself has passed QA.

The balrog delivery progress can continue in the new bug https://bugzilla.mozilla.org/show_bug.cgi?id=1550793

See Also: → 1550643

I've been following the threads you added me to, generating a lot of email for me, but I remain unclear on how to get/apply the patch for 52/XP to re-enable uBlock Origin, HTTPS Everywhere, and Privacy Badger. Thanks

(In reply to cmn3-fox from comment #88)

I've been following the threads you added me to, generating a lot of email for me, but I remain unclear on how to get/apply the patch for 52/XP to re-enable uBlock Origin, HTTPS Everywhere, and Privacy Badger. Thanks

Can you please try the "Legacy hotfix for Firefox 52 through 60 (signed)" add-on from bug 1550793, and let us know in that bug if it works for you?

Flags: needinfo?(cmn3-fox)

Hi, Lina. Yes, it installed an "add-on," which Firefox 52 tried to block, so I overrode that, and now the three extensions do work. Is this the final fix or a temporary one? If this is temporary, will this be removed when the final fix is pushed? Interestingly, when I opened Firefox 52 on that old computer to do this, HTTPS Everywhere had mysteriously re-appeared and was working even before I installed this "add-on" hotfix. But uBlock Origin and Privacy Badger did not re-appear until after the hotfix was applied. Cheers

Flags: needinfo?(cmn3-fox)

(In reply to cmn3-fox from comment #90)

Please see my reply to the cross-post at https://bugzilla.mozilla.org/show_bug.cgi?id=1550793#c13.

Depends on: 1551321
Depends on: 1552218
Depends on: 1552221
Depends on: 1552223

Could someone please give this bug a proper priority? It seems like a P1 even though it hasn't been commented on in a while, so I'll give it that for now, but please update if that's not right.

Priority: -- → P1

I think we're all done here.

Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED

@rhelmer what's the fix?

(In reply to Alex J from comment #96)

@rhelmer what's the fix?

Updating to the latest Firefox release or ESR is the best way to fix this, but if you're stuck on an older version, then https://support.mozilla.org/en-US/kb/add-ons-disabled-or-fail-to-install-firefox#w_for-older-versions-of-firefox points to the authoritative hotfix add-ons.

These were prepared in the dependencies when we determined that we were going to need to target different add-ons at different version ranges: bug 1552223, bug 1552221, and bug 1552218.

We've also pushed these via system add-on updates to Firefox 52+, although I should note that until Firefox 62 these required app update to be enabled. For 62+ bug 1428459 added a separate pref so app update does not need to be enabled.

Attachment #9063551 - Attachment is obsolete: true
Attachment #9063605 - Attachment is obsolete: true
Attachment #9064006 - Attachment is obsolete: true
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: