Linux nightly builds are not being created on oak

RESOLVED FIXED

Status

Release Engineering
Buildduty
--
critical
RESOLVED FIXED
8 months ago
4 months ago

People

(Reporter: rstrong, Assigned: Callek)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(2 attachments)

Possibly due to taskcluster.

This is needed to try to diagnose bug 1303834
Flags: needinfo?(bugspam.Callek)

Updated

8 months ago
Component: General Automation → Buildduty
QA Contact: catlee → bugspam.Callek
(Assignee)

Updated

8 months ago
See Also: → bug 1353642
Assignee: nobody → aselagea
Created attachment 8855345 [details] [diff] [review]
bug_1353819.patch
Attachment #8855345 - Flags: review?(bugspam.Callek)
$ diff -U 0 -p old_tasks new_tasks | grep -v @@
--- old_tasks   2017-04-06 14:38:27.055414200 +0300
+++ new_tasks   2017-04-06 18:07:56.980976500 +0300
+build-linux-nightly/opt
+build-linux64-nightly/opt

I basically added only the two linux builds to the list of already existing tasks.
Comment on attachment 8855345 [details] [diff] [review]
bug_1353819.patch

My bad, I omitted the fact that Callek is on pto today. Moving the review request to Dustin.
Attachment #8855345 - Flags: review?(bugspam.Callek) → review?(dustin)
Comment on attachment 8855345 [details] [diff] [review]
bug_1353819.patch

No bmo login atm, but r-

This bug needs the nightlies added to oak on the cron.yml schedule, not per checkin.

It needs balrog,beetmover,signing,etc.  Adding it to cron.yml at sametime as oaks bb nightlies should get us most of the way, rest would be in the scriptworker utility hardcodes in the taskcluster folder.

-Callek (sent from mobile)
Attachment #8855345 - Flags: review?(dustin)

Comment 5

8 months ago
just as a fyi Alin, Callek is referring to the .cron.yml in the root of a tree like mozilla-inbound.  These are triggered by hooks that are defined automatically by taskcluster-admin

https://tools.taskcluster.net/hooks/#project-releng/cron-task-projects-oak
Thanks for the suggestions. 

If my understanding is correct, we should align the start time for the TC Linux nightlies with the BB ones. I spent some time investigating where we define those values but, unfortunately, I had no luck :|
Per IRC:

"
<aselagea|buildduty> Alin Selagea I wonder where we define the start time for the buildbot nightly builds
14:35:12 I don't seem to be able to find that in our configs
15:28:30 R<rail> aselagea|buildduty: this one https://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla/config.py#l1943 ?
"

https://hg.mozilla.org/mozilla-central/file/tip/.cron.yml#l18 suggests that we should buildbot starts.
but that doesn't seem to be true (at least for the date branch), since we have:
      date: [{hour: 15, minute: 0}]

while on buildbot:
      h = BRANCHES['date']['start_hour'][0]
      m = BRANCHES['date']['start_minute'][0]
      print 'hour:', h, ';', 'minute', m

      hour: 4 ; minute 2
(Assignee)

Updated

8 months ago
Assignee: aselagea → bugspam.Callek
Flags: needinfo?(bugspam.Callek)
Comment hidden (mozreview-request)

Comment 9

8 months ago
mozreview-review
Comment on attachment 8855897 [details]
Bug 1353819 - Run nightly jobs on Oak.

https://reviewboard.mozilla.org/r/127770/#review130476
Attachment #8855897 - Flags: review?(aki) → review+

Comment 10

8 months ago
If we follow the current pattern of date, we'd just land this on oak.
I'm not sure what the merging pattern is there, though.
I can land that on oak if you like. I tend to merge m-c to oak each day.
(Assignee)

Comment 12

8 months ago
https://hg.mozilla.org/projects/oak/rev/7938c51ab68b136727b8d3b96b2cc8b7d43da0c8

I suggest leaving this open until we verify that the builds complete, and maybe that older nightlies can update to these.

Comment 13

8 months ago
https://hg.mozilla.org/projects/oak/rev/1d8669872a3403804f3aee1661b1a7b0f2b1db38
Bug 1353819 - disable android nightlies on oak.

Comment 14

8 months ago
Hitting a level 2 vs 3 issue.

We also need to disable android, and linux AB, so likely we'll need to:

- custom target_task_method to weed out android + linux AB per push
- debug the level 2 vs 3 issue on nightlies

I need to prioritize beta bustage, so this may not all happen by my EOD.

Updated

8 months ago
Depends on: 1355131

Updated

8 months ago
Duplicate of this bug: 1355138
I'm increasing the severity of this because it is blocking proper testing of Dawn work, which is a top priority.
Blocks: 1353825
Severity: normal → critical

Comment 17

8 months ago
(In reply to Aki Sasaki [:aki] from comment #14)

> - custom target_task_method to weed out android + linux AB per push
afaict, at least android isn't blocking oak nightlies anymore since we are allowing them as a stopgap: https://hg.mozilla.org/projects/oak/rev/ecda67e5bb01
> - debug the level 2 vs 3 issue on nightlies
looks like oak is level 3 thanks to bug 1355131 and bug 1355132

iiuc, we can try to run nightlies again on oak? I'll reach out on irc to see if that's the case.
I'm unblocked now - whatever changes were made were enough to get Oak nightlies going.

Updated

8 months ago
No longer blocks: 1353825
(Assignee)

Updated

4 months ago
Status: NEW → RESOLVED
Last Resolved: 4 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.