1358976 - Stop automatic triggers of nightly builds on mozilla-aurora

Status: VERIFIED FIXED

(Release Engineering :: General, enhancement)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

So since Tuesday last week we no longer land changes to the mozilla-aurora branch. But we are still creating nightly builds based on the fixed cron job in place. This is wasted machine time due to the amount of jobs we spin for that, and we should figure out a better way to only run this if really necessary. Which means if a security patch has be landed on that branch.

I would assume that letting sheriffs manually trigger a Nightly build in such a case should be fine.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Looks like this is also happening for all the other platforms.

Comment 2

[Justin Wood (:Callek)]

:dustin, opinion.

Disable the cron schedule on an aurora (and trunk) push, or disable the hook schedule in tc hooks? (or both)?

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

The branch is dying, right?  Let's kill the hook.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

We might still get a build under emergency reasons, but as mentioned before I think it might be fine if sheriffs are triggering it manually?

Also what about Buildbot nightly builds?

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Catlee or Dustin, who can do the next steps for this bug?

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

no idea

Comment 7

[Chris AtLee [:catlee]]

Is this causing any problems by leaving them run?

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I assume the storage costs aren't that important, but what we actually loose here is a lot of machine power other integration branches could really benefit from. What we currently build are:

* Nightly builds (Build bot and TC based) for all platforms, and every supported locale
* For each locale + platform we run the funsize jobs to produce 4 partial update files
* For some locales + all platforms we run firefox-ui-update tests

Given that none of those builds and updates change, and that we run this once a day, I wonder what the benefit is beside just wasting developers time in waiting for jobs being run on try. Whereby even on integration branches we have large backlogs as it looks like.

Comment 9

[Ryan VanderMeulen [:RyanVM]]

Are they actually running every day at the moment? I'm not seeing any recent ones on Treeherder...
https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&filter-searchStr=nightly

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Yes, it's all under the last revision:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=94b7e538af7dbb9a6567cad9125c2f5411e7fb80&filter-tier=1&filter-tier=2&filter-tier=3

Keep in mind due to the massive amount of jobs for this changeset it takes ages to load all the data.

Comment 11

[Ryan VanderMeulen [:RyanVM]]

Ah yes, the loading delay was throwing me. Then yes, given our ever-present backlog issues, this is certainly a set of useless tests that would be great to eliminate.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #11)
> Ah yes, the loading delay was throwing me. Then yes, given our ever-present
> backlog issues, this is certainly a set of useless tests that would be great
> to eliminate.

Justin, you helped in setting up the crontab or whatever else is necessary triggered jobs for oak. Is this bug something you could also help?

Comment 13

[Justin Wood (:Callek)]

Two ways to do this for taskcluster, you can edit the in-tree file at https://hg.mozilla.org/releases/mozilla-aurora/file/tip/.cron.yml to remove the 'when' (for aurora only if desired) -- you'll want an empty array as the value:  `[]`

*or* you can get :dustin or someone to disable the cron schedule on aurora https://tools.taskcluster.net/hooks/#project-releng/cron-task-releases-mozilla-aurora

This is a "DO NOT EDIT" hook that is configured automatically by taskcluster admin.

We want to leave the manual-nightly hooks:
https://tools.taskcluster.net/hooks/#project-releng/nightly-fennec%252fmozilla-aurora
and
https://tools.taskcluster.net/hooks/#project-releng/nightly-desktop%252fmozilla-aurora

And the supporting code to trigger them in the .cron.yml (which by modifying just the `when` will do so) until/unless we are certain we don't need a new nightly on aurora.

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Given that Justin is away right now, who could actually do those steps? Jordan, could you help out?

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Justin informed me that Jordan is also away. So Greg, could you please help us to figure out who from the Taskcluster team could help? Thanks.

Comment 16

[Dustin J. Mitchell [:dustin] (he/him)]

It sounds like just modifying the when clause as described is the best approach for now.  I'll be happy to r+ that patch.

Comment 17

[Chris AtLee [:catlee]]

I don't know that we have a good way to disable the buildbot nightly schedulers without also disabling the builders.

Comment 18

[Kim Moir [:kmoir] ET]

Attachment: bug1358976tc.patch

Comment 19

[Kim Moir [:kmoir] ET]

Attachment: bug1358976bb.patch

This disables the buildbot tests on aurora + some of the build jobs.  This bug doesn't happen very often so this is just a stopgap patch until we can get rid of aurora completely.

< Rev7 MacOSX Yosemite 10.10.5 mozilla-aurora debug test mochitest-e10s-devtools-chrome-7 ScriptFactory
< Rev7 MacOSX Yosemite 10.10.5 mozilla-aurora debug test mochitest-e10s-devtools-chrome-8 ScriptFactory
< Rev7 MacOSX Yosemite 10.10.5 mozilla-aurora debug test mochitest-e10s-1 ScriptFactory
< Rev7 MacOSX Yosemite 10.10.5 mozilla-aurora debug test mochitest-e10s-2 ScriptFactory
< Rev7 MacOSX Yosemite 10.10.5 mozilla-aurora debug test mochitest-e10s-3 

102,110d101
< OS X 10.7 mozilla-aurora nightly SigningScriptFactory
< Firefox mozilla-aurora macosx64 l10n nightly-1 SigningScriptFactory
< Firefox mozilla-aurora macosx64 l10n nightly-2 SigningScriptFactory
< Firefox mozilla-aurora macosx64 l10n nightly-3 SigningScriptFactory
< Firefox mozilla-aurora macosx64 l10n nightly-4 SigningScriptFactory
< Firefox mozilla-aurora macosx64 l10n nightly-5 SigningScriptFactory
< Firefox mozilla-aurora macosx64 l10n nightly-6 SigningScriptFactory
< Firefox mozilla-aurora macosx64 l10n nightly-7 SigningScriptFactory
< Firefox mozilla-aurora macosx64 l10n nightly-8 SigningScriptFactory
113,119d103
< WINNT 5.2 mozilla-aurora nightly SigningScriptFactory
< Firefox mozilla-aurora win32 l10n nightly-1 SigningScriptFactory
< Firefox mozilla-aurora win32 l10n nightly-2 SigningScriptFactory
< Firefox mozilla-aurora win32 l10n nightly-3 SigningScriptFactory
< Firefox mozilla-aurora win32 l10n nightly-4 SigningScriptFactory
< Firefox mozilla-aurora win32 l10n nightly-5 SigningScriptFactory
< Firefox mozilla-aurora win32 l10n nightly-6 SigningScriptFactory
122,128d105
< WINNT 6.1 x86-64 mozilla-aurora nightly SigningScriptFactory
< Firefox mozilla-aurora win64 l10n nightly-1 SigningScriptFactory
< Firefox mozilla-aurora win64 l10n nightly-2 SigningScriptFactory
< Firefox mozilla-aurora win64 l10n nightly-3 SigningScriptFactory
< Firefox mozilla-aurora win64 l10n nightly-4 SigningScriptFactory
< Firefox mozilla-aurora win64 l10n nightly-5 SigningScriptFactory
< Firefox mozilla-aurora win64 l10n nightly-6 SigningScriptFactory

Comment 20

[Greg Arndt [:garndt]]

removing ni? looks like comment 18 is the patch needed

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/590b949a5e4a
Stop automatic triggers of nightly builds on mozilla-aurora r=dustin DONTBUILD

Comment 22

[Chris AtLee [:catlee]]

(In reply to Kim Moir [:kmoir] from comment #18)
> Created attachment 8865995 [details] [diff] [review]
> bug1358976tc.patch

We'll need to back this out if we never need to build another aurora nightly to fix a security issue.

Comment 23

[Henrik Skupin [:whimboo][⌚️UTC+1]]

So there is no way to just stop the automatic generation of nightly builds, but leave the feature in place to let sheriffs trigger a nightly build?

Comment 24

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/590b949a5e4a

Comment 25

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Henrik Skupin (:whimboo) from comment #23)
> So there is no way to just stop the automatic generation of nightly builds,
> but leave the feature in place to let sheriffs trigger a nightly build?

Justin replied yesterday via email and mentioned the following:

> No we won't, the manual nightly trigger will still work with this. 

It means that whenever we need a new nightly build due to security fixes, it can be triggered by hand.

Comment 26

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I assume we have to uplift this patch to mozilla-aurora?

Comment 27

[Kim Moir [:kmoir] ET]

yes but the current code on m-a is different so I need a different patch, will look at this today.

Comment 28

[Kim Moir [:kmoir] ET]

Attachment: bug1358976m-a.patch

https://hg.mozilla.org/releases/mozilla-aurora/rev/6a80fe909738915dcac79fe5cbbfc722ad93afd4

Comment 29

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I still see Nightly builds being triggered for Android. Where those missed?

Comment 30

[Henrik Skupin [:whimboo][⌚️UTC+1]]

https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=6a80fe909738915dcac79fe5cbbfc722ad93afd4&filter-tier=1&filter-tier=2&filter-tier=3

Comment 31

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Also not directly related to this bug but what are those nightly builds on mozilla-beta?

https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&selectedJob=98571880

I thought that the new devedition builds will be produced daily, but now we have two types of nightly builds?

Comment 32

[Kim Moir [:kmoir] ET]

Attachment: bug1358976android.patch

patch to disable android builds

Comment 33

[Kim Moir [:kmoir] ET]

I'm not sure of the state of dev edition nightlies asked about in comment 31 so I'll ask bhearsum to provide context

Comment 34

[[e-mail] (:bhearsum)]

(In reply to Henrik Skupin (:whimboo) from comment #31)
> Also not directly related to this bug but what are those nightly builds on
> mozilla-beta?
> 
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&selectedJob=98571880
> 
> I thought that the new devedition builds will be produced daily, but now we
> have two types of nightly builds?

We are producing per-push builds of DevEdition as well as nightlies on mozilla-beta. (Last I checked, we need per-push builds so that regressions can be more easily tracked.)

In buildbot and Taskcluster we're treating these as separate "platforms" that have a different update channel set, if that matters to you.

Comment 35

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/3a47e27d4010
Stop automatic triggers of nightly builds on mozilla-aurora r=dustin DONTBUILD

Comment 36

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/3a47e27d4010

Comment 37

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Looks like the follow-up patch for Android also landed on aurora via:
https://hg.mozilla.org/releases/mozilla-aurora/rev/d38398e5144e312301a5635da5e27cb6054ecdaa

Since then no new nightly build for any platform. Thanks Kim!