Closed Bug 1353819 Opened 3 years ago Closed 3 years ago

Linux nightly builds are not being created on oak

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task, critical)

task
Not set
critical

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: robert.strong.bugs, Assigned: Callek)

References

Details

Attachments

(2 files)

Possibly due to taskcluster.

This is needed to try to diagnose bug 1303834
Flags: needinfo?(bugspam.Callek)
Component: General Automation → Buildduty
QA Contact: catlee → bugspam.Callek
See Also: → 1353642
Assignee: nobody → aselagea
Attachment #8855345 - Flags: review?(bugspam.Callek)
$ diff -U 0 -p old_tasks new_tasks | grep -v @@
--- old_tasks   2017-04-06 14:38:27.055414200 +0300
+++ new_tasks   2017-04-06 18:07:56.980976500 +0300
+build-linux-nightly/opt
+build-linux64-nightly/opt

I basically added only the two linux builds to the list of already existing tasks.
Comment on attachment 8855345 [details] [diff] [review]
bug_1353819.patch

My bad, I omitted the fact that Callek is on pto today. Moving the review request to Dustin.
Attachment #8855345 - Flags: review?(bugspam.Callek) → review?(dustin)
Comment on attachment 8855345 [details] [diff] [review]
bug_1353819.patch

No bmo login atm, but r-

This bug needs the nightlies added to oak on the cron.yml schedule, not per checkin.

It needs balrog,beetmover,signing,etc.  Adding it to cron.yml at sametime as oaks bb nightlies should get us most of the way, rest would be in the scriptworker utility hardcodes in the taskcluster folder.

-Callek (sent from mobile)
Attachment #8855345 - Flags: review?(dustin)
just as a fyi Alin, Callek is referring to the .cron.yml in the root of a tree like mozilla-inbound.  These are triggered by hooks that are defined automatically by taskcluster-admin

https://tools.taskcluster.net/hooks/#project-releng/cron-task-projects-oak
Thanks for the suggestions. 

If my understanding is correct, we should align the start time for the TC Linux nightlies with the BB ones. I spent some time investigating where we define those values but, unfortunately, I had no luck :|
Per IRC:

"
<aselagea|buildduty> Alin Selagea I wonder where we define the start time for the buildbot nightly builds
14:35:12 I don't seem to be able to find that in our configs
15:28:30 R<rail> aselagea|buildduty: this one https://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla/config.py#l1943 ?
"

https://hg.mozilla.org/mozilla-central/file/tip/.cron.yml#l18 suggests that we should buildbot starts.
but that doesn't seem to be true (at least for the date branch), since we have:
      date: [{hour: 15, minute: 0}]

while on buildbot:
      h = BRANCHES['date']['start_hour'][0]
      m = BRANCHES['date']['start_minute'][0]
      print 'hour:', h, ';', 'minute', m

      hour: 4 ; minute 2
Assignee: aselagea → bugspam.Callek
Flags: needinfo?(bugspam.Callek)
If we follow the current pattern of date, we'd just land this on oak.
I'm not sure what the merging pattern is there, though.
I can land that on oak if you like. I tend to merge m-c to oak each day.
https://hg.mozilla.org/projects/oak/rev/7938c51ab68b136727b8d3b96b2cc8b7d43da0c8

I suggest leaving this open until we verify that the builds complete, and maybe that older nightlies can update to these.
Hitting a level 2 vs 3 issue.

We also need to disable android, and linux AB, so likely we'll need to:

- custom target_task_method to weed out android + linux AB per push
- debug the level 2 vs 3 issue on nightlies

I need to prioritize beta bustage, so this may not all happen by my EOD.
Duplicate of this bug: 1355138
I'm increasing the severity of this because it is blocking proper testing of Dawn work, which is a top priority.
Blocks: 1353825
Severity: normal → critical
(In reply to Aki Sasaki [:aki] from comment #14)

> - custom target_task_method to weed out android + linux AB per push
afaict, at least android isn't blocking oak nightlies anymore since we are allowing them as a stopgap: https://hg.mozilla.org/projects/oak/rev/ecda67e5bb01
> - debug the level 2 vs 3 issue on nightlies
looks like oak is level 3 thanks to bug 1355131 and bug 1355132

iiuc, we can try to run nightlies again on oak? I'll reach out on irc to see if that's the case.
I'm unblocked now - whatever changes were made were enough to get Oak nightlies going.
No longer blocks: 1353825
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.