Closed Bug 1350064 Opened 3 years ago Closed 3 years ago

System add-ons deployed with a build do not update an existing system add-on that was pushed.

Categories

(Toolkit :: Add-ons Manager, defect)

defect
Not set

Tracking

()

VERIFIED FIXED
mozilla55
Tracking Status
firefox52 --- affected
firefox53 --- verified
firefox54 --- verified
firefox55 --- verified

People

(Reporter: rstrong, Assigned: rhelmer)

Details

Attachments

(2 files)

I think we figured out one of the problems with the aushelper system add-on success rate. Turns out that when a Windows 51.0.1 Firefox client gets the aushelper 1.5 system add-on pushed to them and then updates to 52.0.1 that the aushelper 2.0 system add-on that comes with 52.0.1 is not successfully installed and they continue running the aushelper 1.5 system add-on.

Not sure if there are other cases but this
BTW: running a 52.0.1 build using a profile that doesn't have the aushelper 1.5 system add-on installed (e.g. received it as a push) does install the aushelper 2.0 system add-on that was included in the build.
My query against longitudinal[1] suggests that about 1 in 7 Windows clients on 52.x are still running the old aushelper version.

[1]: https://sql.telemetry.mozilla.org/queries/3850/source
(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #0)
> I think we figured out one of the problems with the aushelper system add-on
> success rate. Turns out that when a Windows 51.0.1 Firefox client gets the
> aushelper 1.5 system add-on pushed to them and then updates to 52.0.1 that
> the aushelper 2.0 system add-on that comes with 52.0.1 is not successfully
> installed and they continue running the aushelper 1.5 system add-on.
> 
> Not sure if there are other cases but this
s/Not sure if there are other cases but this/Not sure if there are other cases of this/
Ack. Only 1 in 7 Windows clients on 52.x are running the _NEW_ aushelper version. Most are stuck on the old one.
BTW: Those are likely new installations.
The current set of updates is stored in pref extensions.systemAddonSet - this is supposed to be reset when the application version changes.
Assignee: nobody → rhelmer
Which it wasn't for me and it appears that it isn't getting reset for the majority of 52.0.1 users as well. From my profile that I upgraded from 51.0.1 to 52.0.1

{"schema":1,"directory":"{1dfafdf0-3361-4735-a240-466a1dc48b71}","addons":{"aushelper@mozilla.org":{"version":"1.5"},"diagnostics@mozilla.org":{"version":"1.0"},"disableSHA1rollout@mozilla.org":{"version":"1.3"},"hsts-priming@mozilla.org":{"version":"1.0"}}}
(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #5)
> BTW: Those are likely new installations.
I should have said that the one in seven that are reporting the new aushelper which is aushelper 2.0 are likely new installations (e.g. new profile) since 52.0.1 installed the aushelper 2.0 system add-on for me when I didn't have the aushelper 1.5 system add-on installed.
As a workaround while I look into this, could we serve aushelper 2.0 to 52.0.1 users?

Once we figure out what's going on we can ship a fix and get it uplifted so reverting to the built-in works correctly on the next app update.
Flags: needinfo?(robert.strong.bugs)
I sent an email to ddurst and felipe suggesting that we push an aushelper 2.5 to 52.x and possibly future versions as well. I'd prefer going with a higher version so we can differentiate the clients in telemetry. As soon as we have a chance to discuss it I'll let you know what we'll do.
Flags: needinfo?(robert.strong.bugs)
(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #10)
> I sent an email to ddurst and felipe suggesting that we push an aushelper
> 2.5 to 52.x and possibly future versions as well. I'd prefer going with a
> higher version so we can differentiate the clients in telemetry. As soon as
> we have a chance to discuss it I'll let you know what we'll do.
For clarity, I'd prefer going with a higher *add-on* version so we can differentiate the clients in telemetry.
sgtm. I can reproduce this, so I suspect it's a regression shipped with 52.

We have a test for this in http://searchfox.org/mozilla-central/rev/c48398abd9f0f074c69f2223260939e30e8f99a8/toolkit/mozapps/extensions/test/xpcshell/test_system_reset.js so I am going to start by trying to trigger this there, and make sure it's not something Windows-specific.
Status: NEW → ASSIGNED
Since the test didn't catch it perhaps this bug has been there for longer and might explain some of the low uptake in 47.0.2.
I didn't comment on the email, but the 2.5 plan has a thumbs-up from rstrong, felipe, and me, so let's do.
(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #13)
> Since the test didn't catch it perhaps this bug has been there for longer
> and might explain some of the low uptake in 47.0.2.

It's possible, but I suspect it's not the case since the reset path was much simpler before... certainly possible though. I think we would've seen this behavior in telemetry at least, we should still see a bunch of old versions hanging around.

I'll grab an old build and confirm though instead of speculating.
I'll request uplift once we get this landed - that'll get it fixed for good on the next app update.

In the meantime, we can work around this using the update mechanism - I'll take a look at what's new in http://searchfox.org/mozilla-central/source/browser/extensions that got updates in the 51.* cycle and see what's impacted.
Filed bug 1350186 to create the aushelper system add-on that will be pushed to 52.* clients.
Comment on attachment 8850796 [details]
Bug 1350064 - test that updated system add-ons are reset on app update

https://reviewboard.mozilla.org/r/123312/#review125698

::: commit-message-20018:1
(Diff revision 1)
> +Bug 1350064 - test that updates system add-ons are reset on app update r?aswan

updates -> updated
Attachment #8850796 - Flags: review?(aswan) → review+
Pushed by rhelmer@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/45b68fe4d4be
test that updated system add-ons are reset on app update r=aswan
Attached file uplift request
Approval Request Comment
[Feature/Bug causing the regression]: bug 1204156
[User impact if declined]: system add-on updates are not cleared on app update, causing extra downloads for users and hassle for system add-on releases
[Is this code covered by automated tests?]: yes
[Has the fix been verified in Nightly?]: no
[Needs manual test from QE? If yes, steps to reproduce]: steps to repro would be to ensure that 51.0.1 has updated system add-ons and then to upgrade to 52.0.1 and observe that updates are still present in about:support (the bug is that aushelper would still be using the 1.0 update served to 51.0.1, rather than the built-in 2.0 version)
[List of other uplifts needed for the feature/fix]: none
[Is the change risky?]: no
[Why is the change risky/not risky?]: trivial/obvious fix, easy to test for
[String changes made/needed]: none
Attachment #8850831 - Flags: approval-mozilla-release?
Attachment #8850831 - Flags: approval-mozilla-beta?
Attachment #8850831 - Flags: approval-mozilla-aurora?
I'd expect this to be a ridealong if uplifted to release as we can work around it in the meantime and it should not cause any harm.
(In reply to Iris Hsiao [:ihsiao] from comment #24)
> sorry had to back this out for eslint failure in test_system_reset.js like
> https://treeherder.mozilla.org/logviewer.
> html#?job_id=86114398&repo=autoland&lineNumber=4374
> 
> https://hg.mozilla.org/integration/autoland/rev/3368df04e574

Thanks! I pushed the whitespace change to fix this, doing a try run to make sure it sticks this time:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=5b1df9bf5a4c
Flags: needinfo?(rhelmer)
I took a small sample of the longitudinal table for the last 7 days to get an idea of what system add-ons are active on 52.0.1:

https://sql.telemetry.mozilla.org/queries/2678/source#table

The only add-ons affected by this other than aushelper are these two which are active (their install.rdf has 52.*):

* hsts-priming@mozilla.org v1.0
* disableSHA1rollout@mozilla.org v1.3

Cory, due to this bug these two were left active in profiles of 51.* users upgrading to 52.*

I've tested that any leftover system add-ons like this that are not part of the update set
get removed restartlessly the next time we push an update, so that is working correctly and 
should clean up any lingering system add-on updates.

It looks like we're planning to ship these same add-on versions, I just wanted to make sure
everyone is aware that these two were activated earlier than expected.
Flags: needinfo?(cprice)
Pushed by rhelmer@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/ec80a3f5cebc
test that updated system add-ons are reset on app update r=aswan
https://hg.mozilla.org/mozilla-central/rev/ec80a3f5cebc
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla55
Flags: needinfo?(cprice)
Hi Brindusa, could you help find someone to verify if this issue was fixed as expected on a latest Nightly build? Thanks!
Flags: qe-verify+
Flags: needinfo?(brindusa.tot)
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:55.0) Gecko/20100101 Firefox/55.0

I can confirm the fix on the latest Nightly 55.0a1, build ID 20170326030204 on Windows 10 x64, Ubuntu 14.04 x64 and Mac OS X 10.12. When updating from a earlier Nightly release(51.0a1,52.0a1, etc.) the system addon aushelper is successfully updated from 1.0 to 2.0 version.
Status: RESOLVED → VERIFIED
Flags: needinfo?(brindusa.tot)
Hi Rhelmer, given the critical nature of this problem, I think we should uplift this fix to Aurora54 and eta53. If you agree, please nominate for uplift to Aurora and Beta.
Flags: needinfo?(rhelmer)
Comment on attachment 8850831 [details]
uplift request

Fix a system addon update issue and was verified. Aurora54+ & Beta53+.
Attachment #8850831 - Flags: approval-mozilla-beta?
Attachment #8850831 - Flags: approval-mozilla-beta+
Attachment #8850831 - Flags: approval-mozilla-aurora?
Attachment #8850831 - Flags: approval-mozilla-aurora+
Flags: needinfo?(rhelmer)
I confirm that the version of Application Update Service Helper is updated from 1.0 to 2.0 when Firefox is updated to latest builds. 
The tests were performed for Firefox Aurora (updated from 51.0a2(2016-11-01) to 54.0a2 (2017-04-06)) and for Firefox Beta (updated from 51.0b9 to 53.0b9).
Tests were performed under Windows 10x64.
Comment on attachment 8850831 [details]
uplift request

This got fixed in beta 53, so I'm clearing the m-r approval flag.
Attachment #8850831 - Flags: approval-mozilla-release? → approval-mozilla-release-
You need to log in before you can comment on or make changes to this bug.