Stop automatic triggers of nightly builds on mozilla-aurora

VERIFIED FIXED

Status

Release Engineering
General
VERIFIED FIXED
a year ago
2 months ago

People

(Reporter: whimboo, Assigned: kmoir)

Tracking

unspecified

Firefox Tracking Flags

(firefox55 fixed)

Details

Attachments

(4 attachments)

(Reporter)

Description

a year ago
So since Tuesday last week we no longer land changes to the mozilla-aurora branch. But we are still creating nightly builds based on the fixed cron job in place. This is wasted machine time due to the amount of jobs we spin for that, and we should figure out a better way to only run this if really necessary. Which means if a security patch has be landed on that branch.

I would assume that letting sheriffs manually trigger a Nightly build in such a case should be fine.
Flags: needinfo?(bugspam.Callek)
(Reporter)

Comment 1

a year ago
Looks like this is also happening for all the other platforms.
Summary: Improve triggereing of TC Nightly builds on mozilla-central for Linux/Linux64 → Improve triggering of nightly builds on mozilla-aurora
(Reporter)

Updated

a year ago
Summary: Improve triggering of nightly builds on mozilla-aurora → Stop automatic triggers of nightly builds on mozilla-aurora
:dustin, opinion.

Disable the cron schedule on an aurora (and trunk) push, or disable the hook schedule in tc hooks? (or both)?
Flags: needinfo?(bugspam.Callek) → needinfo?(dustin)
The branch is dying, right?  Let's kill the hook.
Flags: needinfo?(dustin)
(Reporter)

Comment 4

a year ago
We might still get a build under emergency reasons, but as mentioned before I think it might be fine if sheriffs are triggering it manually?

Also what about Buildbot nightly builds?
(Reporter)

Comment 5

a year ago
Catlee or Dustin, who can do the next steps for this bug?
Flags: needinfo?(dustin)
Flags: needinfo?(catlee)
no idea
Flags: needinfo?(dustin) → needinfo?(hskupin)
Is this causing any problems by leaving them run?
Flags: needinfo?(catlee)
(Reporter)

Comment 8

a year ago
I assume the storage costs aren't that important, but what we actually loose here is a lot of machine power other integration branches could really benefit from. What we currently build are:

* Nightly builds (Build bot and TC based) for all platforms, and every supported locale
* For each locale + platform we run the funsize jobs to produce 4 partial update files
* For some locales + all platforms we run firefox-ui-update tests

Given that none of those builds and updates change, and that we run this once a day, I wonder what the benefit is beside just wasting developers time in waiting for jobs being run on try. Whereby even on integration branches we have large backlogs as it looks like.
Flags: needinfo?(hskupin)
Are they actually running every day at the moment? I'm not seeing any recent ones on Treeherder...
https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&filter-searchStr=nightly
(Reporter)

Comment 10

a year ago
Yes, it's all under the last revision:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=94b7e538af7dbb9a6567cad9125c2f5411e7fb80&filter-tier=1&filter-tier=2&filter-tier=3

Keep in mind due to the massive amount of jobs for this changeset it takes ages to load all the data.
Ah yes, the loading delay was throwing me. Then yes, given our ever-present backlog issues, this is certainly a set of useless tests that would be great to eliminate.
(Reporter)

Comment 12

a year ago
(In reply to Ryan VanderMeulen [:RyanVM] from comment #11)
> Ah yes, the loading delay was throwing me. Then yes, given our ever-present
> backlog issues, this is certainly a set of useless tests that would be great
> to eliminate.

Justin, you helped in setting up the crontab or whatever else is necessary triggered jobs for oak. Is this bug something you could also help?
Flags: needinfo?(bugspam.Callek)
Two ways to do this for taskcluster, you can edit the in-tree file at https://hg.mozilla.org/releases/mozilla-aurora/file/tip/.cron.yml to remove the 'when' (for aurora only if desired) -- you'll want an empty array as the value:  `[]`

*or* you can get :dustin or someone to disable the cron schedule on aurora https://tools.taskcluster.net/hooks/#project-releng/cron-task-releases-mozilla-aurora

This is a "DO NOT EDIT" hook that is configured automatically by taskcluster admin.

We want to leave the manual-nightly hooks:
https://tools.taskcluster.net/hooks/#project-releng/nightly-fennec%252fmozilla-aurora
and
https://tools.taskcluster.net/hooks/#project-releng/nightly-desktop%252fmozilla-aurora

And the supporting code to trigger them in the .cron.yml (which by modifying just the `when` will do so) until/unless we are certain we don't need a new nightly on aurora.
Flags: needinfo?(bugspam.Callek)
(Reporter)

Comment 14

a year ago
Given that Justin is away right now, who could actually do those steps? Jordan, could you help out?
Flags: needinfo?(jlund)
(Reporter)

Comment 15

a year ago
Justin informed me that Jordan is also away. So Greg, could you please help us to figure out who from the Taskcluster team could help? Thanks.
Flags: needinfo?(garndt)
(Reporter)

Updated

a year ago
Flags: needinfo?(jlund)
It sounds like just modifying the when clause as described is the best approach for now.  I'll be happy to r+ that patch.
I don't know that we have a good way to disable the buildbot nightly schedulers without also disabling the builders.
(Assignee)

Comment 18

a year ago
Created attachment 8865995 [details] [diff] [review]
bug1358976tc.patch
Attachment #8865995 - Flags: review?(dustin)
(Assignee)

Comment 19

a year ago
Created attachment 8866017 [details] [diff] [review]
bug1358976bb.patch

This disables the buildbot tests on aurora + some of the build jobs.  This bug doesn't happen very often so this is just a stopgap patch until we can get rid of aurora completely.

< Rev7 MacOSX Yosemite 10.10.5 mozilla-aurora debug test mochitest-e10s-devtools-chrome-7 ScriptFactory
< Rev7 MacOSX Yosemite 10.10.5 mozilla-aurora debug test mochitest-e10s-devtools-chrome-8 ScriptFactory
< Rev7 MacOSX Yosemite 10.10.5 mozilla-aurora debug test mochitest-e10s-1 ScriptFactory
< Rev7 MacOSX Yosemite 10.10.5 mozilla-aurora debug test mochitest-e10s-2 ScriptFactory
< Rev7 MacOSX Yosemite 10.10.5 mozilla-aurora debug test mochitest-e10s-3 

102,110d101
< OS X 10.7 mozilla-aurora nightly SigningScriptFactory
< Firefox mozilla-aurora macosx64 l10n nightly-1 SigningScriptFactory
< Firefox mozilla-aurora macosx64 l10n nightly-2 SigningScriptFactory
< Firefox mozilla-aurora macosx64 l10n nightly-3 SigningScriptFactory
< Firefox mozilla-aurora macosx64 l10n nightly-4 SigningScriptFactory
< Firefox mozilla-aurora macosx64 l10n nightly-5 SigningScriptFactory
< Firefox mozilla-aurora macosx64 l10n nightly-6 SigningScriptFactory
< Firefox mozilla-aurora macosx64 l10n nightly-7 SigningScriptFactory
< Firefox mozilla-aurora macosx64 l10n nightly-8 SigningScriptFactory
113,119d103
< WINNT 5.2 mozilla-aurora nightly SigningScriptFactory
< Firefox mozilla-aurora win32 l10n nightly-1 SigningScriptFactory
< Firefox mozilla-aurora win32 l10n nightly-2 SigningScriptFactory
< Firefox mozilla-aurora win32 l10n nightly-3 SigningScriptFactory
< Firefox mozilla-aurora win32 l10n nightly-4 SigningScriptFactory
< Firefox mozilla-aurora win32 l10n nightly-5 SigningScriptFactory
< Firefox mozilla-aurora win32 l10n nightly-6 SigningScriptFactory
122,128d105
< WINNT 6.1 x86-64 mozilla-aurora nightly SigningScriptFactory
< Firefox mozilla-aurora win64 l10n nightly-1 SigningScriptFactory
< Firefox mozilla-aurora win64 l10n nightly-2 SigningScriptFactory
< Firefox mozilla-aurora win64 l10n nightly-3 SigningScriptFactory
< Firefox mozilla-aurora win64 l10n nightly-4 SigningScriptFactory
< Firefox mozilla-aurora win64 l10n nightly-5 SigningScriptFactory
< Firefox mozilla-aurora win64 l10n nightly-6 SigningScriptFactory
Attachment #8866017 - Flags: review?(spacurar)
removing ni? looks like comment 18 is the patch needed
Flags: needinfo?(garndt)
(Assignee)

Updated

a year ago
Assignee: nobody → kmoir
Attachment #8865995 - Flags: review?(dustin) → review+

Comment 21

a year ago
Pushed by kmoir@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/590b949a5e4a
Stop automatic triggers of nightly builds on mozilla-aurora r=dustin DONTBUILD
(In reply to Kim Moir [:kmoir] from comment #18)
> Created attachment 8865995 [details] [diff] [review]
> bug1358976tc.patch

We'll need to back this out if we never need to build another aurora nightly to fix a security issue.
Attachment #8866017 - Flags: review?(spacurar) → review+
(Reporter)

Comment 23

a year ago
So there is no way to just stop the automatic generation of nightly builds, but leave the feature in place to let sheriffs trigger a nightly build?

Comment 24

a year ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/590b949a5e4a
Status: NEW → RESOLVED
Last Resolved: a year ago
status-firefox55: --- → fixed
Resolution: --- → FIXED
(Reporter)

Comment 25

a year ago
(In reply to Henrik Skupin (:whimboo) from comment #23)
> So there is no way to just stop the automatic generation of nightly builds,
> but leave the feature in place to let sheriffs trigger a nightly build?

Justin replied yesterday via email and mentioned the following:

> No we won't, the manual nightly trigger will still work with this. 

It means that whenever we need a new nightly build due to security fixes, it can be triggered by hand.
(Reporter)

Comment 26

a year ago
I assume we have to uplift this patch to mozilla-aurora?
Flags: needinfo?(kmoir)
(Assignee)

Comment 27

a year ago
yes but the current code on m-a is different so I need a different patch, will look at this today.
Flags: needinfo?(kmoir)
(Assignee)

Updated

a year ago
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Updated

a year ago
Status: REOPENED → RESOLVED
Last Resolved: a year agoa year ago
Resolution: --- → FIXED
(Reporter)

Comment 29

a year ago
I still see Nightly builds being triggered for Android. Where those missed?
Flags: needinfo?(kmoir)
(Reporter)

Comment 31

a year ago
Also not directly related to this bug but what are those nightly builds on mozilla-beta?

https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&selectedJob=98571880

I thought that the new devedition builds will be produced daily, but now we have two types of nightly builds?
(Assignee)

Comment 32

a year ago
Created attachment 8867171 [details] [diff] [review]
bug1358976android.patch

patch to disable android builds
Flags: needinfo?(kmoir)
Attachment #8867171 - Flags: review?(hskupin)
(Assignee)

Comment 33

a year ago
I'm not sure of the state of dev edition nightlies asked about in comment 31 so I'll ask bhearsum to provide context
Flags: needinfo?(bhearsum)
(In reply to Henrik Skupin (:whimboo) from comment #31)
> Also not directly related to this bug but what are those nightly builds on
> mozilla-beta?
> 
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&selectedJob=98571880
> 
> I thought that the new devedition builds will be produced daily, but now we
> have two types of nightly builds?

We are producing per-push builds of DevEdition as well as nightlies on mozilla-beta. (Last I checked, we need per-push builds so that regressions can be more easily tracked.)

In buildbot and Taskcluster we're treating these as separate "platforms" that have a different update channel set, if that matters to you.
Flags: needinfo?(bhearsum)
(Reporter)

Updated

a year ago
Attachment #8867171 - Flags: review?(hskupin) → review?(dustin)
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Attachment #8867171 - Flags: review?(dustin) → review+

Comment 35

a year ago
Pushed by kmoir@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/3a47e27d4010
Stop automatic triggers of nightly builds on mozilla-aurora r=dustin DONTBUILD

Comment 36

a year ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/3a47e27d4010
Status: REOPENED → RESOLVED
Last Resolved: a year agoa year ago
Resolution: --- → FIXED
(Reporter)

Comment 37

a year ago
Looks like the follow-up patch for Android also landed on aurora via:
https://hg.mozilla.org/releases/mozilla-aurora/rev/d38398e5144e312301a5635da5e27cb6054ecdaa

Since then no new nightly build for any platform. Thanks Kim!
Status: RESOLVED → VERIFIED
Component: General Automation → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.