1325115 - multiple desktop and mobile nightlies, auroras are being triggered

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Jordan Lund (:jlund)]

likely from self-serve.

context:

WaltS48> Is there a problem with updates? My Firefox Nightly and the Thunderbird's I'm testing are all indicating the 12/19 builds are up to date when using Help > About.

...

Tomcat|sherifffduty> bhearsum: ok do we need to retrigger nightlys then ?
06:51:15 i guess so :)
06:51:17 
<•bhearsum> Tomcat|sherifffduty: that's up to you
06:51:25 
<Tomcat|sherifffduty> bhearsum: ok will retrigger then :)
06:51:31 to get users updated

...

philor> given that we now have between two and three nightlies running at once, *that* will certainly blow things up
07:38:58 
— philor kills them down to just one
Tomcat|sherifffduty: do you have any tabs open or any behavior going on that could possibly explain being up to four sets of nightlies triggered on m-c, and three on aurora?
<Tomcat|sherifffduty> philor: i think this was a crash that hit my nightly
07:48:25 and then restored the tabs
07:48:56 a mac crasher that is fixed ironically on the next nightly :/
...

•coop> aselagea|buildduty: if we're concerned about safety, just kill all the nightlies
08:31:57 
<jlund|away> Jordan Lund Should I freeze aurora and central updates?
08:32:05 Until we know what's going on
<•coop> we can trigger new ones
<•coop> jlund|away: yes, please
08:32:44 
<aselagea|buildduty> Alin Selagea coop: ok, on it

...

jlund|away> Jordan Lund fyi - frozen:
16:46:12 https://aus4-admin.mozilla.org/rules/4
16:46:14 https://aus4-admin.mozilla.org/rules/8
16:46:16 https://aus4-admin.mozilla.org/rules/10
16:46:19 https://aus4-admin.mozilla.org/rules/3
<jlund|away> Jordan Lund note, we will have to reverse the mapping and update rate. also, fennec nightlies ran recently because they are quick
<jlund|away> Jordan Lund so they are frozen against the most recent completed nightly

•catlee> Who does self-serve say is triggering them? 
<aselagea|buildduty> Alin Selagea "reason": "Self-serve: Requested by [e-mail]",

Comment 1

[Jordan Lund (:jlund)]

(In reply to Jordan Lund (:jlund) from comment #0)

> 
> WaltS48> Is there a problem with updates? My Firefox Nightly and the
> Thunderbird's I'm testing are all indicating the 12/19 builds are up to date
> when using Help > About.
> 
Tomcat|sherifffduty> bhearsum: hi! made a comment in the bug, nightlys still have problems
06:24:16 catlee: flod WaltS48 -> bug 1324779

where "Bug 1324779 - Balrog Admin reporting SSL errors" was the motivation for re-triggering nightlies.

Comment 2

[Jordan Lund (:jlund)]

(In reply to Jordan Lund (:jlund) from comment #0)
> jlund|away> Jordan Lund fyi - frozen:
> 16:46:12 https://aus4-admin.mozilla.org/rules/4
> 16:46:14 https://aus4-admin.mozilla.org/rules/8
> 16:46:16 https://aus4-admin.mozilla.org/rules/10
> 16:46:19 https://aus4-admin.mozilla.org/rules/3
> <jlund|away> Jordan Lund note, we will have to reverse the mapping and
> update rate. also, fennec nightlies ran recently because they are quick
> <jlund|away> Jordan Lund so they are frozen against the most recent
> completed nightly

to be actioned before closing this bug. reverse last history step for these rules ^

Comment 3

[Chris AtLee [:catlee]]

I only see two requests in self-serve for nightlies:

[e-mail] 	new_nightly_at_revision {'priority': 0, 'branch': 'mozilla-central', 'revision': '5b0afeaeebdd'} 2016-12-21 06:52:30 	2016-12-21 09:01:19 	{'body': {'msg': 'there was an error processing your request; please check logs on buildbot-master72.bb.releng.usw2.mozilla.com', 'errors': True}, 'request_id': 2746274}

[e-mail] 	new_nightly_at_revision {'priority': 0, 'branch': 'mozilla-aurora', 'revision': 'e6ec59a1128a'} 2016-12-21 06:52:15 	2016-12-21 09:10:11 	{'body': {'msg': 'Ok', 'errors': False}, 'request_id': 2746273}

The timings are strange. Each took several hours to complete, and one of them had errors.

Comment 4

[Chris AtLee [:catlee]]

It takes a really long time to get the list of nightly builder names:
2016-12-21 08:44:11,182 New nightly by [e-mail] of mozilla-central 5b0afeaeebdd
2016-12-21 09:01:16,968 buildernames are ['Android armv7 API 15+ mozilla-central nightly', 'Android 4.2 x86 mozilla-central nightly', 'Linux mozilla-central nightly', 'Linux x86-64 mozilla-central nightly', 'OS X 10.7 mozilla-central nightly', 'WINNT 5.2 mozilla-central nightly', 'WINNT 6.1 x86-64 mozilla-central nightly']
al nightly']

and acking the pulse message after this failed due to the connection timing out, so the message was requeued.

Comment 5

[Jordan Lund (:jlund)]

(In reply to Jordan Lund (:jlund) from comment #2)
> (In reply to Jordan Lund (:jlund) from comment #0)
> > jlund|away> Jordan Lund fyi - frozen:
> > 16:46:12 https://aus4-admin.mozilla.org/rules/4
> > 16:46:14 https://aus4-admin.mozilla.org/rules/8
> > 16:46:16 https://aus4-admin.mozilla.org/rules/10
> > 16:46:19 https://aus4-admin.mozilla.org/rules/3
> > <jlund|away> Jordan Lund note, we will have to reverse the mapping and
> > update rate. also, fennec nightlies ran recently because they are quick
> > <jlund|away> Jordan Lund so they are frozen against the most recent
> > completed nightly
> 
> to be actioned before closing this bug. reverse last history step for these
> rules ^

similar as osx central nightly issues from https://bugzilla.mozilla.org/show_bug.cgi?id=1325133#c10 I ended up changing the buildid to freeze on for desktop/mobile central/aurora to being based on the most previous complete platform set of nightlies: dec 19th. This doesn't really change what is required to 'unfreeze' nightlies back to '-latest' but wanted to document this change and its motivation

Comment 6

[Jordan Lund (:jlund)]

fyi:

* the instance of this bug has now been resolved
* bug 1325371 is going to make sure this doesn't happen again
* nightlies and auroras are now enabled again to latest