System add-ons deployed with a build do not update an existing system add-on that was pushed.

VERIFIED FIXED in Firefox 53

Status

()

Toolkit
Add-ons Manager
VERIFIED FIXED
a month ago
18 days ago

People

(Reporter: rstrong, Assigned: rhelmer)

Tracking

unspecified
mozilla55
Points:
---

Firefox Tracking Flags

(firefox52 affected, firefox53 verified, firefox54 verified, firefox55 verified)

Details

MozReview Requests

()

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(2 attachments)

I think we figured out one of the problems with the aushelper system add-on success rate. Turns out that when a Windows 51.0.1 Firefox client gets the aushelper 1.5 system add-on pushed to them and then updates to 52.0.1 that the aushelper 2.0 system add-on that comes with 52.0.1 is not successfully installed and they continue running the aushelper 1.5 system add-on.

Not sure if there are other cases but this
BTW: running a 52.0.1 build using a profile that doesn't have the aushelper 1.5 system add-on installed (e.g. received it as a push) does install the aushelper 2.0 system add-on that was included in the build.

Comment 2

a month ago
My query against longitudinal[1] suggests that about 1 in 7 Windows clients on 52.x are still running the old aushelper version.

[1]: https://sql.telemetry.mozilla.org/queries/3850/source
(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #0)
> I think we figured out one of the problems with the aushelper system add-on
> success rate. Turns out that when a Windows 51.0.1 Firefox client gets the
> aushelper 1.5 system add-on pushed to them and then updates to 52.0.1 that
> the aushelper 2.0 system add-on that comes with 52.0.1 is not successfully
> installed and they continue running the aushelper 1.5 system add-on.
> 
> Not sure if there are other cases but this
s/Not sure if there are other cases but this/Not sure if there are other cases of this/

Comment 4

a month ago
Ack. Only 1 in 7 Windows clients on 52.x are running the _NEW_ aushelper version. Most are stuck on the old one.
BTW: Those are likely new installations.
(Assignee)

Comment 6

a month ago
The current set of updates is stored in pref extensions.systemAddonSet - this is supposed to be reset when the application version changes.
Assignee: nobody → rhelmer
Which it wasn't for me and it appears that it isn't getting reset for the majority of 52.0.1 users as well. From my profile that I upgraded from 51.0.1 to 52.0.1

{"schema":1,"directory":"{1dfafdf0-3361-4735-a240-466a1dc48b71}","addons":{"aushelper@mozilla.org":{"version":"1.5"},"diagnostics@mozilla.org":{"version":"1.0"},"disableSHA1rollout@mozilla.org":{"version":"1.3"},"hsts-priming@mozilla.org":{"version":"1.0"}}}
(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #5)
> BTW: Those are likely new installations.
I should have said that the one in seven that are reporting the new aushelper which is aushelper 2.0 are likely new installations (e.g. new profile) since 52.0.1 installed the aushelper 2.0 system add-on for me when I didn't have the aushelper 1.5 system add-on installed.
(Assignee)

Comment 9

a month ago
As a workaround while I look into this, could we serve aushelper 2.0 to 52.0.1 users?

Once we figure out what's going on we can ship a fix and get it uplifted so reverting to the built-in works correctly on the next app update.
Flags: needinfo?(robert.strong.bugs)
I sent an email to ddurst and felipe suggesting that we push an aushelper 2.5 to 52.x and possibly future versions as well. I'd prefer going with a higher version so we can differentiate the clients in telemetry. As soon as we have a chance to discuss it I'll let you know what we'll do.
Flags: needinfo?(robert.strong.bugs)
(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #10)
> I sent an email to ddurst and felipe suggesting that we push an aushelper
> 2.5 to 52.x and possibly future versions as well. I'd prefer going with a
> higher version so we can differentiate the clients in telemetry. As soon as
> we have a chance to discuss it I'll let you know what we'll do.
For clarity, I'd prefer going with a higher *add-on* version so we can differentiate the clients in telemetry.
(Assignee)

Comment 12

a month ago
sgtm. I can reproduce this, so I suspect it's a regression shipped with 52.

We have a test for this in http://searchfox.org/mozilla-central/rev/c48398abd9f0f074c69f2223260939e30e8f99a8/toolkit/mozapps/extensions/test/xpcshell/test_system_reset.js so I am going to start by trying to trigger this there, and make sure it's not something Windows-specific.
Status: NEW → ASSIGNED
Since the test didn't catch it perhaps this bug has been there for longer and might explain some of the low uptake in 47.0.2.
I didn't comment on the email, but the 2.5 plan has a thumbs-up from rstrong, felipe, and me, so let's do.
Comment hidden (mozreview-request)
(Assignee)

Comment 16

a month ago
(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #13)
> Since the test didn't catch it perhaps this bug has been there for longer
> and might explain some of the low uptake in 47.0.2.

It's possible, but I suspect it's not the case since the reset path was much simpler before... certainly possible though. I think we would've seen this behavior in telemetry at least, we should still see a bunch of old versions hanging around.

I'll grab an old build and confirm though instead of speculating.
(Assignee)

Comment 17

a month ago
I'll request uplift once we get this landed - that'll get it fixed for good on the next app update.

In the meantime, we can work around this using the update mechanism - I'll take a look at what's new in http://searchfox.org/mozilla-central/source/browser/extensions that got updates in the 51.* cycle and see what's impacted.
Filed bug 1350186 to create the aushelper system add-on that will be pushed to 52.* clients.

Comment 19

a month ago
mozreview-review
Comment on attachment 8850796 [details]
Bug 1350064 - test that updated system add-ons are reset on app update

https://reviewboard.mozilla.org/r/123312/#review125698

::: commit-message-20018:1
(Diff revision 1)
> +Bug 1350064 - test that updates system add-ons are reset on app update r?aswan

updates -> updated
Attachment #8850796 - Flags: review?(aswan) → review+
Comment hidden (mozreview-request)

Comment 21

a month ago
Pushed by rhelmer@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/45b68fe4d4be
test that updated system add-ons are reset on app update r=aswan
(Assignee)

Comment 22

a month ago
Created attachment 8850831 [details]
uplift request

Approval Request Comment
[Feature/Bug causing the regression]: bug 1204156
[User impact if declined]: system add-on updates are not cleared on app update, causing extra downloads for users and hassle for system add-on releases
[Is this code covered by automated tests?]: yes
[Has the fix been verified in Nightly?]: no
[Needs manual test from QE? If yes, steps to reproduce]: steps to repro would be to ensure that 51.0.1 has updated system add-ons and then to upgrade to 52.0.1 and observe that updates are still present in about:support (the bug is that aushelper would still be using the 1.0 update served to 51.0.1, rather than the built-in 2.0 version)
[List of other uplifts needed for the feature/fix]: none
[Is the change risky?]: no
[Why is the change risky/not risky?]: trivial/obvious fix, easy to test for
[String changes made/needed]: none
Attachment #8850831 - Flags: approval-mozilla-release?
Attachment #8850831 - Flags: approval-mozilla-beta?
Attachment #8850831 - Flags: approval-mozilla-aurora?
(Assignee)

Comment 23

a month ago
I'd expect this to be a ridealong if uplifted to release as we can work around it in the meantime and it should not cause any harm.
sorry had to back this out for eslint failure in test_system_reset.js like https://treeherder.mozilla.org/logviewer.html#?job_id=86114398&repo=autoland&lineNumber=4374

https://hg.mozilla.org/integration/autoland/rev/3368df04e574
Flags: needinfo?(rhelmer)
Comment hidden (mozreview-request)
(Assignee)

Comment 26

a month ago
(In reply to Iris Hsiao [:ihsiao] from comment #24)
> sorry had to back this out for eslint failure in test_system_reset.js like
> https://treeherder.mozilla.org/logviewer.
> html#?job_id=86114398&repo=autoland&lineNumber=4374
> 
> https://hg.mozilla.org/integration/autoland/rev/3368df04e574

Thanks! I pushed the whitespace change to fix this, doing a try run to make sure it sticks this time:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=5b1df9bf5a4c
Flags: needinfo?(rhelmer)
(Assignee)

Comment 27

a month ago
I took a small sample of the longitudinal table for the last 7 days to get an idea of what system add-ons are active on 52.0.1:

https://sql.telemetry.mozilla.org/queries/2678/source#table

The only add-ons affected by this other than aushelper are these two which are active (their install.rdf has 52.*):

* hsts-priming@mozilla.org v1.0
* disableSHA1rollout@mozilla.org v1.3

Cory, due to this bug these two were left active in profiles of 51.* users upgrading to 52.*

I've tested that any leftover system add-ons like this that are not part of the update set
get removed restartlessly the next time we push an update, so that is working correctly and 
should clean up any lingering system add-on updates.

It looks like we're planning to ship these same add-on versions, I just wanted to make sure
everyone is aware that these two were activated earlier than expected.
Flags: needinfo?(cprice)

Comment 28

a month ago
Pushed by rhelmer@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/ec80a3f5cebc
test that updated system add-ons are reset on app update r=aswan

Comment 29

a month ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/ec80a3f5cebc
Status: ASSIGNED → RESOLVED
Last Resolved: a month ago
status-firefox55: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla55
(Assignee)

Updated

a month ago
Flags: needinfo?(cprice)

Updated

a month ago
status-firefox53: --- → affected
status-firefox54: --- → affected

Updated

a month ago
status-firefox52: --- → affected
Hi Brindusa, could you help find someone to verify if this issue was fixed as expected on a latest Nightly build? Thanks!
Flags: qe-verify+
Flags: needinfo?(brindusa.tot)
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:55.0) Gecko/20100101 Firefox/55.0

I can confirm the fix on the latest Nightly 55.0a1, build ID 20170326030204 on Windows 10 x64, Ubuntu 14.04 x64 and Mac OS X 10.12. When updating from a earlier Nightly release(51.0a1,52.0a1, etc.) the system addon aushelper is successfully updated from 1.0 to 2.0 version.
Status: RESOLVED → VERIFIED
status-firefox55: fixed → verified
Flags: needinfo?(brindusa.tot)
Hi Rhelmer, given the critical nature of this problem, I think we should uplift this fix to Aurora54 and eta53. If you agree, please nominate for uplift to Aurora and Beta.
Flags: needinfo?(rhelmer)
Comment on attachment 8850831 [details]
uplift request

Fix a system addon update issue and was verified. Aurora54+ & Beta53+.
Attachment #8850831 - Flags: approval-mozilla-beta?
Attachment #8850831 - Flags: approval-mozilla-beta+
Attachment #8850831 - Flags: approval-mozilla-aurora?
Attachment #8850831 - Flags: approval-mozilla-aurora+

Comment 34

a month ago
bugherderuplift
https://hg.mozilla.org/releases/mozilla-aurora/rev/2365a3af90a4
status-firefox54: affected → fixed

Comment 35

a month ago
bugherderuplift
https://hg.mozilla.org/releases/mozilla-beta/rev/6c66fac00e39
status-firefox53: affected → fixed
(Assignee)

Updated

a month ago
Flags: needinfo?(rhelmer)
I confirm that the version of Application Update Service Helper is updated from 1.0 to 2.0 when Firefox is updated to latest builds. 
The tests were performed for Firefox Aurora (updated from 51.0a2(2016-11-01) to 54.0a2 (2017-04-06)) and for Firefox Beta (updated from 51.0b9 to 53.0b9).
Tests were performed under Windows 10x64.
status-firefox53: fixed → verified
status-firefox54: fixed → verified
Flags: qe-verify+
Comment on attachment 8850831 [details]
uplift request

This got fixed in beta 53, so I'm clearing the m-r approval flag.
Attachment #8850831 - Flags: approval-mozilla-release? → approval-mozilla-release-
You need to log in before you can comment on or make changes to this bug.