983756 - Figure out a way to prevent falling behind wrt to b2g nightly builds

Status: RESOLVED DUPLICATE of bug 983731

(Release Engineering :: General, defect)

Description

[Armen [:armenzg]]

> Why didn't the system generate nightlies from the 4:37Am merge tomcat did to
> m-c? That was all green.

Simplest reason: it was merged after the nightlies for b2g triggered (we trigger them at 4am and again at 4pm)
per: http://mxr.mozilla.org/build/source/buildbot-configs/mozilla/b2g_config.py#999

Even if the merge was pushed at say, 3:45am we wouldn't have built nightlies on it because we didn't [as of time of nightly triggering] have finished jobs.

Additionally, even if it was pushed at 3:00am we wouldn't have built nightlies on it because wouldn't have gotten the full set of device builds (which we do on a periodic cadence).

This is the same model we have used for Desktop-Nightlies for Firefox (e.g. periodic PGO, to avoid surprise nightly issues)
################################

Possible solutions:
* make m-c approval-only [1]
* make m-c's periodic interval to be every 3 hours instead of 6 [2]

Early-warning solution:
* setup a nagios check on the freshness of the pvt builds [3]

[1]
"I'd rather see us move forward with making m-c an approval-only branch as discussed before so we can reasonably drop the "last good rev" logic from it and run nightlies off tip. We've discussed it before, but it seems we can never drive it to completion."

[2]
"Another easy fix for now would be to make the m-c interval between periodic builds be 3 hours like it is on every other branch instead of 6. Then KWierso's merges would have had periodic builds that finished before the nightlies started."

[3]
https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/mozilla-central-hamachi/latest/

Comment 1

[Aki Sasaki (not active)]

pvtbuilds nagios wouldn't help here, since the issue is the freshness of http://hg.mozilla.org/mozilla-central/file/default/b2g/config/gaia.json , not the freshness of the nightly build.