1639979 - 20200519160549 Nightly can't update with security.cancel_non_local_systemprincipal

Status: RESOLVED FIXED

(Toolkit :: Application Update, defect)

Description

[Jeff Muizelaar [:jrmuizel]]

In the about dialog I get "Update failed." and I get a pop up that says "Nightly can't update to the latest version. Download a fresh copy of Nightly and we'll help you install it."

I started getting this on macOS today.

Comment 1

[Jeff Muizelaar [:jrmuizel]]

Perhaps related to bug 1639429?

Comment 2

[Jeff Muizelaar [:jrmuizel]]

Setting security.cancel_non_local_systemprincipal=false fixed it

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

https://sql.telemetry.mozilla.org/queries/71356/source#179234 takes a look at how many telemetry pings there are from the busted 20200519160549 build. RyanVM, would you mind monitoring that and deciding if we need to take any further action ?

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Console update log

This log illustrates why safe mode doesn't help - it's for a fresh install of the broken build with no addons. The system principal change somehow breaks the download of the mar files. The urls themselves are fine but Firefox cannot download it.

Setting the pref in comment #2 resolves the issue, which we might do with a hotfix or similar if there is significant population there.

Comment 5

[Jeff Muizelaar [:jrmuizel]]

https://sql.telemetry.mozilla.org/queries/71368/source#179258 is another view of who's left behind.

Comment 7

[Frederik Braun [:freddy]]

Flipping security.cancel_non_local_loads_triggered_by_systemprincipal should resolve this for whoever is left behind.

Comment 9

[Julien Cristau [:jcristau]]

As far as I can tell from telemetry there's no evidence at this point of more people being stuck on this particular build than others around the same date, so it looks like we may not need further action.

Comment 10

[Vicky Chin [:vchin]]

If someone doesn't find this bug and flip the pref, updates won't work and they are either broken or stuck on an old build.
While there's no evidence that there are more people on this build than others, there's also no way for them to move forward.

Comment 11

[Jeff Muizelaar [:jrmuizel]]

I've updated: https://sql.telemetry.mozilla.org/queries/71368/source#179258. It shows that there about twice as many people stuck on this build as nearby ones. That being said, there continues to be a steady decline in telemetry pings so I think it's worth waiting a week to how the numbers look then.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

The ~2x more users seems to be on 20200519094847, the build I froze updates to when this first came up, instead of the problematic 20200519160549. I checked 20200519094847 can update fine (on mac 10.15 with a new profile).

There's another bump at 20200521093657 from another nightly update freeze so this might be a feature of these watersheds in the update paths. Or there is some subtlety in interpreting this data that I've missed.

It would be great if we had a play book ready to put into action quickly, avoiding nightly users having to re-install. Maybe a normandy recipe for channel==nightly and buildID==<some value or range> and then <recipe to solve> ?

Comment 13

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

FWIW, this issue happened to me too. I was stuck on a May 19th release of Firefox nightly (specifically: 78.0a1 (2020-05-19) (64-bit))) on Mac as a result. Flipping security.cancel_non_local_systemprincipal to false and restarting nightly allowed it to recover.

Comment 14

[Adam Gashlin (he/him) [:agashlin] (ex-moco)]

I don't know if there's anything more to be done here, duping to the other bug with more duplicates. I'll make a comment there about the pref for anyone still stuck who runs across it.

Comment 15

[Adam Gashlin (he/him) [:agashlin] (ex-moco)]

On second thought, this really is more about update completely failing even on a fresh profile, so it's somewhat of a separate case. Fixed by the backout of bug 1613609, fixable with the pref flip.

Comment 16

[Jeff Muizelaar [:jrmuizel]]

I updated https://sql.telemetry.mozilla.org/queries/71368/source#179258 again. While 20200519160549 is one of the builds in the range with the most remaining users it doesn't really stand out that much.