1325133 - [Mac] Unable to update nightly from December 19th build

Status: RESOLVED FIXED

(Release Engineering Graveyard :: Applications: Balrog (backend), defect)

Description

[Marcia Knous [:marcia]]

Seen while running Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:51.0) Gecko/20100101 Firefox/51.0 ID:20161219140923 CSet: 6e4843d1510be426212d31fec03ad4f2d70b1977

STR:
1. Check for a nightly update on Mac
2. Nothing happens.

I turned on app.update.log to grab some logging. Now I have to remember where to find the log.

Just checked my Windows machine and update is working fine there.

Comment 1

[Robert Strong (they/them - no direct email)]

The log is in the browser console. If you could get the line with the update url that is logged when checking for an update that would be helpful.

Comment 2

[Myk Melez [:myk] [@mykmelez]]

I'm seeing this as well.  Here's my log output:

10:44:36.764 AUS:SVC Checker: checkForUpdates, force: true
10:44:36.765 AUS:SVC Checker:getUpdateURL - update URL: https://aus5.mozilla.org/update/6/Firefox/53.0a1/20161219030207/Darwin_x86_64-gcc3/en-US/nightly/Darwin%2015.6.0/NA/default/default/update.xml?force=1
10:44:36.766 AUS:SVC Checker:checkForUpdates - sending request to: https://aus5.mozilla.org/update/6/Firefox/53.0a1/20161219030207/Darwin_x86_64-gcc3/en-US/nightly/Darwin%2015.6.0/NA/default/default/update.xml?force=1
10:44:36.954 AUS:SVC Checker:onLoad - request completed downloading document
10:44:36.954 AUS:SVC Checker:onLoad - number of updates available: 0

Comment 3

[Michael Kohler [:mkohler]]

It seems I'm stuck on 20161204030210. Same Mac OS X version.

19:44:34.621 AUS:SVC Checker:getUpdateURL - update URL: https://aus5.mozilla.org/update/6/Firefox/53.0a1/20161204030210/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/nightly/Darwin%2016.3.0/NA/default/default/update.xml?force=1
19:44:34.621 AUS:SVC Checker:checkForUpdates - sending request to: https://aus5.mozilla.org/update/6/Firefox/53.0a1/20161204030210/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/nightly/Darwin%2016.3.0/NA/default/default/update.xml?force=1
19:44:35.835 AUS:SVC Checker:onLoad - request completed downloading document
19:44:35.835 AUS:SVC Checker:onLoad - number of updates available: 0

Comment 4

[Robert Strong (they/them - no direct email)]

Ben and Rail, looks like some Macs aren't advertised updates for these builds.

Comment 5

[Marcia Knous [:marcia]]

Tracking 53+ for this issue - we need users on Mac with updated builds.

Comment 6

[Rail Aliiev [:rail]]

I see jlund pointed the nightlies to 20161221030226 2h ago. He may have more info.

Comment 7

[Chris AtLee [:catlee]]

dupe of bug 1324779?

Comment 8

[Jordan Lund (:jlund)]

(In reply to Rail Aliiev [:rail] ⌚️ET from comment #6)
> I see jlund pointed the nightlies to 20161221030226 2h ago. He may have more
> info.

hm, weird. The timing of this bug lines up with my changes. For those with access, I changed the official nightly channel rule to map to Firefox-mozilla-central-nightly-20161221030226 at https://aus4-admin.mozilla.org/rules/3

However, this serves users a buildid of 20161221030226 which is newer than the clients mentioned in this bug. And even rule has an update rate of 0%, force is true in https://bugzilla.mozilla.org/show_bug.cgi?id=1325133#c2 so that shouldn't block updating to 20161221030226

Maybe a 32bit osx nightly de-support is affecting a wider scope than we realize? or this is dup of, as catlee mentioned, to the ssl bug @ 1324779.

Comment 9

[Jordan Lund (:jlund)]

hm, looking at https://aus4-admin.mozilla.org/releases#Firefox-mozilla-central-nightly-latest and Firefox-mozilla-central-nightly-20161221030226 it seems that 20161221030226 was a nightly that uploaded to latest but wasn't 100% complete. Which meant we are missing bits like en-us for osx64.

I think it's better to point to yesterday's nightlies (dec 20th) than trying to find a complete one from today until "Bug 1325115 - multiple desktop and mobile nightlies, auroras are being triggered" gets resolved and we can unfreeze nightlies.

I'll do that now

Comment 10

[Jordan Lund (:jlund)]

tl;dr updates frozen now at 20161219030207 instead of 20161221030226

The last complete green set of nightlies we can lock to is the one from Dec 19th. I believe we are still trying to get a complete set from today (21st). So I have frozen nightlies against 20161219030207 (dec 19th). This means that comment 0 and 3 should now receive updates while comment 2 (myk) will not get an update as that is the latest, for now.

Please close this bug if you can now receive an update up until dec 19th.

Comment 11

[Michael Kohler [:mkohler]]

Just got an update from 20161204030210 to 20161219030207. This seems to work, therefore closing this bug.

Thanks!

Comment 12

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

(In reply to Jordan Lund (:jlund) from comment #10)
> So I have frozen nightlies against 20161219030207 (dec 19th). This means
> that comment 0 and 3 should now receive updates while comment 2 (myk) will
> not get an update as that is the latest, for now.

Disabling ("freezing") of updates is essentially an outage of our normal process, and I think it needs wide notification.  Not knowing that updates are disabled can cause serious confusion later, such as:

 * people looking at crash-stats will get incorrect regression windows if they try to find out when crashes started happening (potentially looking at data weeks or months after the fact)

 * people who see reports of bugs filed by nightly users will have an incorrect impression of when those bugs started (particularly for the sorts of bugs that are obvious enough that they'll be reported on the first nightly that has them)

I think disabling of nightly or aurora updates needs to be treated as an outage, and reported as such to lists such as stability, dev-platform, and firefox-dev.

Comment 13

[Jordan Lund (:jlund)]

(In reply to David Baron :dbaron: ⌚️UTC-8 from comment #12)
> (In reply to Jordan Lund (:jlund) from comment #10)
> > So I have frozen nightlies against 20161219030207 (dec 19th). This means
> > that comment 0 and 3 should now receive updates while comment 2 (myk) will
> > not get an update as that is the latest, for now.
> 
> Disabling ("freezing") of updates is essentially an outage of our normal
> process, and I think it needs wide notification.

I've added this issue to our monthly agenda to discuss how we can be better about communicating update (balrog) changes.

Comment 14

[Jordan Lund (:jlund)]

(In reply to Jordan Lund (:jlund) from comment #13)
> I've added this issue to our monthly agenda to discuss how we can be better
> about communicating update (balrog) changes.

monthly releng mtg agenda at the beginning of Jan, to be more accurate.