1332930 - Shutting off buildbot nighties shut off periodicupdates

Status: VERIFIED FIXED

(Release Engineering :: General, defect)

Description

[Phil Ringnalda (:philor)]

As attachment 8827707 [details] says, shutting off buildbot nightlies also shut off the periodicupdate builder. Since the expiration on the last update that landed is April 26th, and 53 doesn't get replaced until June 13th, and we only do HPKP updates on aurora, not on beta/release, we have six weeks to get updates going on aurora.

Comment 1

[Phil Ringnalda (:philor)]

[Tracking Requested - why for this release]:
If we don't fix it (or manually land an update to the expiration), HPKP will be broken two weeks into the 53 release lifespan.

Comment 2

[Marcia Knous [:marcia]]

Tracking 53+ to prevent brokenness.

Comment 3

[Kim Moir [:kmoir] ET]

So renabling them in buildbot is problematic because I haven't found a way to enable the periodic builds without renabling the nightly ones as well.  I think I'm going to enable them in tc via the new work for cron jobs in bug 1252948.

Comment 4

[Kim Moir [:kmoir] ET]

Attachment: bug1332930bb.patch

wip buildbot patch

Fixing this is very difficult because the way the build dependencies work. I can't seem to leave the file update builder there while removing the generic builder

Comment 5

[Kim Moir [:kmoir] ET]

Attachment: bug1332930bcustom.patch

(build6)[[e-mail] build6]$ diff old new
72a73,74
> Linux x86-64 mozilla-central build SigningScriptFactory
> Linux x86-64 mozilla-central periodic file update ScriptFactory
120a123,124
> Linux x86-64 mozilla-aurora build SigningScriptFactory
> Linux x86-64 mozilla-aurora periodic file update ScriptFactory

still need to remove the  build SigningScriptFactory for the branches

Comment 6

[Aki Sasaki (not active)]

I tried taking a look, but

* I was unsuccessful in getting a buildbot env installed on my mac laptop, then
* I tried creating a buildbot centos docker image https://github.com/escapewindow/bbdocker but am currently stymied by pycrypto and pyOpenSSL not finding -lpython2.7 .

Rather than sink more hours into shaving the yak of reproducing the buildbot builder lists, I'm going to hazard a guess you're going to need one of these ugly |if| blocks to delete the generic builder: https://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla/config.py#l2698

Comment 7

[Kim Moir [:kmoir] ET]

Thanks Aki, I will take another look

Comment 8

[Kim Moir [:kmoir] ET]

Attachment: bug1332930.patch

builder diff 

72a73
> Linux x86-64 mozilla-central periodic file update ScriptFactory
120a122
> Linux x86-64 mozilla-aurora periodic file update ScriptFactory

Comment 9

[Aki Sasaki (not active)]

Comment on attachment 8830772 [details] [diff] [review]
bug1332930.patch

Blame/annotate will show people why this block is here, but it might be nice to add a comment to the |else:| block to say this is to keep periodicupdates.

Good work, thanks!  These if blocks aren't pretty, but they're part of moving off buildbot.  At some point we'll be able to get rid of these config files altogether.

Comment 10

[Kim Moir [:kmoir] ET]

Comment on attachment 8830772 [details] [diff] [review]
bug1332930.patch

caused the tests to fail

Comment 11

[Kim Moir [:kmoir] ET]

Attachment: bug1332930-4.patch

landed and merged to production, tests pass

Comment 12

[Kim Moir [:kmoir] ET]

verified the builder exists here
http://buildbot-master70.bb.releng.use1.mozilla.com:8001/builders

Comment 14

[Phil Ringnalda (:philor)]

Either we were astonishingly unlucky, getting a bad URL from the taskcluster index for the lastest build on both mozilla-central and mozilla-aurora, or the change to TC builds also broke that part of periodicupdates, since both runs this morning failed by getting a 404 URL for the builds.

Comment 15

[Aki Sasaki (not active)]

Attachment: fix_periodic.diff

Whomever is around first gets to review :)
We're not finding the artifacts because they're not pretty named: https://tools.taskcluster.net/task-inspector/#S_1gifVoSVyRo8JXHmyfuA/0
I'm guessing this patch will fix, but I'm not 100% sure.

Comment 16

[Rail Aliiev [:rail]]

Comment on attachment 8831313 [details] [diff] [review]
fix_periodic.diff

lgtm

Comment 17

[Aki Sasaki (not active)]

https://hg.mozilla.org/build/tools/rev/832aa943661555bdb4d9275a18c0232c76e279f3
bug 1332930 - use simple filenames for tc periodic updates. r=rail

Comment 18

[Aki Sasaki (not active)]

Attachment: fix beta, release, esr

Comment 19

[Aki Sasaki (not active)]

https://hg.mozilla.org/build/tools/rev/5dec772a8d528f5f76f7c17566eb5c3b2e3dd102
bug 1332930 - fix m-b m-r m-esr periodic updates. r=rail

Comment 20

[Phil Ringnalda (:philor)]

Total victory.