991707 - Don't generate nightly builds on a tree if no new changes have landed since the previous nightly

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Ed Morley [:emorley]]

(In reply to Phil Ringnalda (:philor) from bug 991476 comment #0)
> Last week, we switched the default for nightlies from building on the last
> good rev, which also switched them from looking at whether or not a nightly
> had been built on the last rev, to just building on the tip.
> 
> Since then, we've built six pointless nightlies on the Ionmonkey tree, and
> triggered six full sets of pointless mostly-unrunnable tests on it. The fact
> that we're out $60 for that is minimal, compared to the fact that we're out
> the use of those slaves during a critical time of the day.

We should make sure we don't build new nightlies if there were already ones on that rev (l10n repo changes excepted, for repos where we do non-en-us builds too).

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [buildbotcustom] Add config to override triggerBuildIfNoChanges for nightlies

lastRevFunc looks up the last revision build for builders belonging to the current scheduler, and has support to not build if that's tip, we just default to not using that for nightlies. This gives us the option to.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [buildbot-configs] Project branches only build nightlies when code changes

This uses triggerBuildIfNoChanges=False for oak and ux branches, and leaves alone central/aurora/b2g-*. This and the buildbotcustom patch pass checkconfig on a build scheduler master, but I haven't figured out any way to test in more depth than that.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

I made my changes based on oak example in bug 1019976, but if we could also do this everywhere except the active b2g branch(es).

Comment 5

[Rail Aliiev [:rail]]

Comment on attachment 8436575 [details] [diff] [review]
[buildbot-configs] Project branches only build nightlies when code changes

... .... .. .--.   .. -

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8436574 [details] [diff] [review]
[buildbotcustom] Add config to override triggerBuildIfNoChanges for nightlies

https://hg.mozilla.org/build/buildbotcustom/rev/9a67b2c5288f

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8436575 [details] [diff] [review]
[buildbot-configs] Project branches only build nightlies when code changes

https://hg.mozilla.org/build/buildbot-configs/rev/5160e18a75fc

Comment 8

[Simone Bruno [:simone]]

In prod with reconfig on 2014-06-12 10:46 PT

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

This is working on Oak and UX.  

Ed/philor, would you like to extend this to other branches ? We have nightlies enabled on m-c, m-a, m-esr24, m-b2g28_v1_3, m-b2g28_v1_3t, m-b2g30_v1_4. IIRC QA wants b2g to always build, but esr24 is probably idle enough that it's worth changing.

Comment 10

[Phil Ringnalda (:philor)]

I can't remember where the "but build it anyway if there were l10n changes even if there weren't code changes" comes in to know whether it's safe to do this to m-a without messing that up, but I'd think esr24 has been l10n frozen for quite some time and it'd be safe there no matter what.

Comment 11

[Ed Morley [:emorley]]

I'd like to do this everywhere where it's possible, as long as the l10n feature that philor mentioned isn't regressed :-)

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 02/17 - 02/21)]

So I have seen this problem over the weekend for the Aurora builds. No code changes have been landed between Saturday and Sunday but we got a completely identical build on Sunday.

http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015/08/2015-08-15-00-40-04-mozilla-aurora/firefox-42.0a2.en-US.linux-i686.txt

http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015/08/2015-08-16-00-40-12-mozilla-aurora/firefox-42.0a2.en-US.linux-i686.txt

Given that a lot of tests are reporting to treeherder those days, it will cause confusion. Especially with tools like our firefox-ui-tests which run on pulse messages and then report multiple times to the same changeset. An example you can see here:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&filter-job_group_symbol=Ff&filter-job_group_symbol=Fr&filter-job_group_symbol=Fu&exclusion_profile=false

This bug is lingering around for the last year, so I wonder if we can finally finish it up? Nick, what's left to do here to get it also working for the other branches?

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Did you see comment #10 ? There's a valid use case for aurora, where l10n changes should generate new nightly builds even if gecko hasn't changed.

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 02/17 - 02/21)]

Sure but that only repacks the en-US build for the changed locales. If en-US has not been changed ideally no new build would be necessary. But I assume the l10n repack scripts want a build of the same build id and that's why we took the route to build them all? Out ouf interest what happens if only a single locale has been changed? Do we build all, or only en-US and that locale?

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

It's just how the scheduling was implemented in buildbot. And en-US nightly triggers l10n nightlies, with no provision to just do l10n in some cases. I'm not sure if we are considering all locales at this point.

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

Current state:

ash                            only changes
oak                            only changes
mozilla-esr38                  only changes
mozilla-central                every time
mozilla-aurora                 every time

mozilla-b2g37_v2_2             every time
mozilla-b2g37_v2_2r            every time
mozilla-b2g44_v2_5             every time
b2g-ota                        every time

comm-central                   every time
comm-aurora                    every time
comm-esr38                     every time

b2g is a special case, and you could argue that Thunderbird is too (AFAIK comm builds don't track if gecko changed).