1252471 - ensure that we have linux64 debug tests running on all these branches for taskcluster tier2

Status: RESOLVED FIXED

(Taskcluster :: Services, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

branches which run linux64 debug tests with no tc tests running:
alder
ash
date
try
mozilla-aurora

branches with linux64 debug tc tests running:
b2g-inbound (running tc tests)
fx-team (running tc tests)
mozilla-central (running tc tests)
mozilla-inbound (running tc tests)

branches with no linux64 debug tc tests running, but no activity in 3+ months:
jamun (possibly defunct branch)
oak (possibly defunct branch)

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

:catlee, can you figure out if we need jamun and oak branches?  They exist and have builders defined but no action for 3+ months!

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

:dustin, what is required to make taskcluster start running linux64 debug test jobs on these branches:
alder
ash
date
try
mozilla-aurora

Comment 3

[Armen [:armenzg]]

We run TC jobs on try [1].
I believe we agreed (in an email thread with others) that we would not touch aurora until we're at tier1.

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=e72fb6b84026&filter-searchStr=tc%20desktop

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

I don't see evidence of tc jobs on try for many other pushes, take for example:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=1f376157cafd

is there a difference in try syntax that causes tc jobs to run?

For aurora, we don't want tier1 right away, but we should have tier2- especially after the merge which is upcoming next week.  I would rather get any small kinks worked out and get us more at capacity for scaling now than wait until after we are 100% ready and then hit new issues.  Maybe aurora is something we wait a week on- but the other branches should be running tests in parallel!

Comment 5

[Armen [:armenzg]]

(In reply to Joel Maher (:jmaher) from comment #4)
> I don't see evidence of tc jobs on try for many other pushes, take for
> example:
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=1f376157cafd
> 
The build has not finished :)
https://treeherder.mozilla.org/#/jobs?repo=try&revision=1f376157cafd&filter-searchStr=tc

> is there a difference in try syntax that causes tc jobs to run?

No. It works the same.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Adding more branches involves configuring them here:

  https://github.com/taskcluster/mozilla-taskcluster/blob/master/src/config/default.yml

I think you also have to insert them into the redis db or the mongo db, or maybe Azure?  Jonas, do you recall the details of configuring mozilla-taskcluster?

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Ah, looks like maybe we run an importer?  From treeherder?

https://github.com/taskcluster/mozilla-taskcluster/blob/master/src/bin/import_repositories.js

Comment 8

[Greg Arndt [:garndt]]

The importer adds them to redis so we can record the last push ID retrieved.  All branches are already configured in there since the last time we ran the importer.

For each branch that we want to run tests on we should be able to add them to the config linked in comment 6 once roles have been setup for them.

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

OK, thanks, I can take care of that.  I'm not clear from discussion above: should m-a be enabled yet?

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

next week we should enable aurora as the uplift takes place on Monday.  If these changes won't get rolled out until next week, then make it all at once.

Comment 11

[Chris AtLee [:catlee]]

(In reply to Joel Maher (:jmaher) from comment #1)
> :catlee, can you figure out if we need jamun and oak branches?  They exist
> and have builders defined but no action for 3+ months!

We don't need Jamun any more, but we do need Oak.

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

Thanks :catlee, please advise if we should open a bug to remove the linux64 debug test buildernames from buildbot for the jamun branch.

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

all our changes are on aurora, so firing up tests on there might be a worthwhile endeavor now.

Comment 14

[Nick Alexander :nalexander [he/him]]

There are TC jobs running on trunk that are not nearly as complex as these Linux tests: eslint and Android Lint / Unit tests.  I expected these to ride the trains and be scheduled on aurora (and all the way up the trains).  Do I need to file a ticket for those jobs in particular?  Or will this ticket unblock all TC jobs to ride the trains?

Comment 15

[Dustin J. Mitchell [:dustin] (he/him)]

This will cover those tasks.

Comment 16

[Joel Maher ( :jmaher ) (UTC -8)]

the goal is to ride the trains along with v.48.  possibly uplift to v.47 (current Aurora).  I would assume that if taskcluster starts scheduling jobs on aurora, then it would schedule all jobs on aurora.  Right now it is a limited set of branches.

:dustin, can you comment on Nick's question?  Also can you give us an idea of when we can see taskcluster running on all trunk based branches, and possibly aurora?

Comment 17

[Nick Alexander :nalexander [he/him]]

(In reply to Joel Maher (:jmaher) from comment #16)
> the goal is to ride the trains along with v.48.

Great news!

  possibly uplift to v.47
> (current Aurora).  I would assume that if taskcluster starts scheduling jobs
> on aurora, then it would schedule all jobs on aurora.  Right now it is a
> limited set of branches.
> 
> :dustin, can you comment on Nick's question?  Also can you give us an idea
> of when we can see taskcluster running on all trunk based branches, and
> possibly aurora?

Thanks everybody.

Comment 18

[Dustin J. Mitchell [:dustin] (he/him)]

All of those repositories are already in the mongo DB from the last import.  I'll add jamun and oak just to be safe - it doesn't hurt.

I set up all of the relevant roles.

Comment 19

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: https://github.com/taskcluster/mozilla-taskcluster/pull/51

To my understanding, once this lands, we'll have builds on all of these branches.

Comment 20

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8729092 [details] [review]
https://github.com/taskcluster/mozilla-taskcluster/pull/51

Greg was kind enough to deploy this, too.  So we should start seeing decision tasks pop up on all the named branches.

Comment 21

[Joel Maher ( :jmaher ) (UTC -8)]

we have taskcluster jobs on:
try
fx-team
mozilla-inbound
mozilla-central
mozilla-aurora

but we don't have it on:
ash
cedar
date
elm
jamun

Comment 22

[Dustin J. Mitchell [:dustin] (he/him)]

I don't see anything in the mozilla-taskcluster logs around the last push to Jamun about 44 hours ago.  It's monitoring the repo, but didn't seem to have any reaction to the push.  Greg, any idea what's up?

Comment 23

[Dustin J. Mitchell [:dustin] (he/him)]

Same for ash

https://hg.mozilla.org/projects/ash/pushloghtml?changeset=19a1743ceb2e - 07:58:38

next poll is

 May 06 07:59:24 mozilla-taskcluster app/pushlog.1: Fri, 06 May 2016 12:59:24 GMT superagent GET https://hg.mozilla.org/projects/ash/json-pushes/ 

but nothing further after that.  Worth noting that we should have been getting builds on cedar and elm even before comment 20.

I admit I really don't understand mozilla-taskcluster.  It has many, many layers of abstraction, especially for such a simple tool, and they're all pretty hard to penetrate.

Comment 24

[Dustin J. Mitchell [:dustin] (he/him)]

Greg fixed this for ash, by resetting the pushlog id in mozilla-taskcluster's DB back to the ID before the most recent.  Basically mozilla-taskcluster doesn't follow "resets" to repositories.

latest pushids
--------------
cedar - 139 (no change; last push was in March)
date  - 0 (reset)
elm   - 1 (reset)
jamun - 21 (reset)

so we should see decision tasks for elm and jamun shortly.

Comment 25

[Dustin J. Mitchell [:dustin] (he/him)]

I just restarted mozilla-taskcluster -- I forgot that was required

Comment 26

[Dustin J. Mitchell [:dustin] (he/him)]

I see a decision task on Jamun.  It failed, which I think means it needs caches generated.  I think greg can do that.

Elm wasn't in the original request (comment 0), so it didn't have anything set up.

Comment 27

[Dustin J. Mitchell [:dustin] (he/him)]

https://github.com/taskcluster/mozilla-taskcluster/pull/59

Comment 28

[Greg Arndt [:garndt]]

I created caches for those manually for now and Ash ran the decision task.  I added them to tc-vcs and will deploy a new package and update the hook on Monday.  The caches are good for 30 days so no harm in waiting until then.  Let me know if any issues come up.

Comment 29

[Greg Arndt [:garndt]]

Also, the changes from comment 27 have been deployed to prod and staging

Comment 30

[Joel Maher ( :jmaher ) (UTC -8)]

I see this working on ash!

I was looking in more detail, and oak has linux64 jobs, so we should ensure we have tc jobs on there as well.

Comment 31

[Joel Maher ( :jmaher ) (UTC -8)]

I have pinged recent commit authors to ask if they can land a fresh commit on these branches, ideally we will see success and mark this as closed.

Comment 32

[Joel Maher ( :jmaher ) (UTC -8)]

ack, a push this morning on elm didn't yield a gecko decision task, nor a taskcluster build:
https://treeherder.mozilla.org/#/jobs?repo=elm&revision=043082cb7bd8490c60815f67fbd1f33323ad7663

Comment 33

[Dustin J. Mitchell [:dustin] (he/him)]

It should get one now.

Comment 34

[Joel Maher ( :jmaher ) (UTC -8)]

and we are all good!
jamun- ok
cedar- ok (although the in-tree configs don't have up to date info
date- ok
oak- ok
elm- ok

Comment 35

[Joel Maher ( :jmaher ) (UTC -8)]

and this is fixed!