1353819 - Linux nightly builds are not being created on oak

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Robert Strong (they/them - no direct email)]

Possibly due to taskcluster.

This is needed to try to diagnose bug 1303834

Comment 1

[Alin Selagea [:aselagea]]

Attachment: bug_1353819.patch

Comment 2

[Alin Selagea [:aselagea]]

$ diff -U 0 -p old_tasks new_tasks | grep -v @@
--- old_tasks   2017-04-06 14:38:27.055414200 +0300
+++ new_tasks   2017-04-06 18:07:56.980976500 +0300
+build-linux-nightly/opt
+build-linux64-nightly/opt

I basically added only the two linux builds to the list of already existing tasks.

Comment 3

[Alin Selagea [:aselagea]]

Comment on attachment 8855345 [details] [diff] [review]
bug_1353819.patch

My bad, I omitted the fact that Callek is on pto today. Moving the review request to Dustin.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8855345 [details] [diff] [review]
bug_1353819.patch

No bmo login atm, but r-

This bug needs the nightlies added to oak on the cron.yml schedule, not per checkin.

It needs balrog,beetmover,signing,etc.  Adding it to cron.yml at sametime as oaks bb nightlies should get us most of the way, rest would be in the scriptworker utility hardcodes in the taskcluster folder.

-Callek (sent from mobile)

Comment 5

[Kim Moir [:kmoir] ET]

just as a fyi Alin, Callek is referring to the .cron.yml in the root of a tree like mozilla-inbound.  These are triggered by hooks that are defined automatically by taskcluster-admin

https://tools.taskcluster.net/hooks/#project-releng/cron-task-projects-oak

Comment 6

[Alin Selagea [:aselagea]]

Thanks for the suggestions. 

If my understanding is correct, we should align the start time for the TC Linux nightlies with the BB ones. I spent some time investigating where we define those values but, unfortunately, I had no luck :|

Comment 7

[Alin Selagea [:aselagea]]

Per IRC:

"
<aselagea|buildduty> Alin Selagea I wonder where we define the start time for the buildbot nightly builds
14:35:12 I don't seem to be able to find that in our configs
15:28:30 R<rail> aselagea|buildduty: this one https://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla/config.py#l1943 ?
"

https://hg.mozilla.org/mozilla-central/file/tip/.cron.yml#l18 suggests that we should buildbot starts.
but that doesn't seem to be true (at least for the date branch), since we have:
      date: [{hour: 15, minute: 0}]

while on buildbot:
      h = BRANCHES['date']['start_hour'][0]
      m = BRANCHES['date']['start_minute'][0]
      print 'hour:', h, ';', 'minute', m

      hour: 4 ; minute 2

Comment 8

[Justin Wood (:Callek)]

Attachment: Bug 1353819 - Run nightly jobs on Oak.

Review commit: https://reviewboard.mozilla.org/r/127770/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/127770/

Comment 9

[Aki Sasaki (not active)]

Comment on attachment 8855897 [details]
Bug 1353819 - Run nightly jobs on Oak.

https://reviewboard.mozilla.org/r/127770/#review130476

Comment 10

[Aki Sasaki (not active)]

If we follow the current pattern of date, we'd just land this on oak.
I'm not sure what the merging pattern is there, though.

Comment 11

[Robert Strong (they/them - no direct email)]

I can land that on oak if you like. I tend to merge m-c to oak each day.

Comment 12

[Justin Wood (:Callek)]

https://hg.mozilla.org/projects/oak/rev/7938c51ab68b136727b8d3b96b2cc8b7d43da0c8

I suggest leaving this open until we verify that the builds complete, and maybe that older nightlies can update to these.

Comment 13

[Aki Sasaki (not active)]

https://hg.mozilla.org/projects/oak/rev/1d8669872a3403804f3aee1661b1a7b0f2b1db38
Bug 1353819 - disable android nightlies on oak.

Comment 14

[Aki Sasaki (not active)]

Hitting a level 2 vs 3 issue.

We also need to disable android, and linux AB, so likely we'll need to:

- custom target_task_method to weed out android + linux AB per push
- debug the level 2 vs 3 issue on nightlies

I need to prioritize beta bustage, so this may not all happen by my EOD.

Comment 16

[[e-mail] (:bhearsum)]

I'm increasing the severity of this because it is blocking proper testing of Dawn work, which is a top priority.

Comment 17

[Jordan Lund (:jlund)]

(In reply to Aki Sasaki [:aki] from comment #14)

> - custom target_task_method to weed out android + linux AB per push
afaict, at least android isn't blocking oak nightlies anymore since we are allowing them as a stopgap: https://hg.mozilla.org/projects/oak/rev/ecda67e5bb01
> - debug the level 2 vs 3 issue on nightlies
looks like oak is level 3 thanks to bug 1355131 and bug 1355132

iiuc, we can try to run nightlies again on oak? I'll reach out on irc to see if that's the case.

Comment 18

[[e-mail] (:bhearsum)]

I'm unblocked now - whatever changes were made were enough to get Oak nightlies going.