1364924 - re-image mac test machines to support move to tc workers on mac hardware

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Kim Moir [:kmoir] ET]

This work doesn't need to be implemented today, but will be implemented later this week, and in the following weeks.

We recently moved the macosx opt tests to run via buildbot bridge. The next step is to run macosx tests on a tc worker running on mac hardware.  To support this effort, we will need to disable a portion of the mac test pool and reimage them with the worker installed.  Puppet configs for this are here

https://hg.mozilla.org/build/puppet/file/2d33ae582e52/manifests/moco-nodes.pp#l8

Once we have a list of machines to migrate to this pool, we'll need to
1) prepare patches to increase the list of machines names in puppet that this configuration will apply to
2) disable machines so they don't accept new buildbot jobs
3) Once they have completed their jobs, they need to be reimaged with the usual bless command

Wander will then land some changes for in-tree scheduling to support this effort.  

We'd like this work to take place in earlier than ET morning so having the mac machines out of capacity doesn't impact wait times than it would later in the day.  

This activity will be repeated several times as we move groups of machines out of the buildbot pool and into the taskcluster pool.

We'll have more concrete dates for this work early this week.

Comment 1

[Kim Moir [:kmoir] ET]

Greg: Anything to add?  Also is their a bug for the item "Run Mac debug tests in TC on hardware"?  I looked but couldn't find one to add as a dependency to this bug.

Comment 2

[Greg Arndt [:garndt]]

I think the last pieces are determining how much we want to do per day.  I suggest the first time doing this that we start with a smaller % (maybe 10%) and let that ride for the day.  Subsequent days can be increased.

Also, when we say 10%, we mean 10% of the total tasks that run on trunk, esr52, and release branches.  Older branches, like esr45, will remain on buildbot until it's gone from I'm aware of.  So machines need to remain for them.  Are there other branches that will remain on buildbot?

Comment 3

[Greg Arndt [:garndt]]

Wander, is there a bug for running tests on hardware?

Comment 4

[Wander Lairson Costa]

(In reply to Greg Arndt [:garndt] from comment #3)
> Wander, is there a bug for running tests on hardware?

Do you mean running tests on generic-worker?

Comment 5

[Greg Arndt [:garndt]]

Test running generic-worker against builds produced by buildbot.  It's the work that we'll begin migrating once we have the CI patches uplifted to esr52 and beta.

Comment 6

[Wander Lairson Costa]

Really not sure, but I think it is Bug 1298431

Comment 7

[Kim Moir [:kmoir] ET]

wcosta: Is the the bug/pr with the changes that you need to do at your end to enable tests to be scheduled to run on taskcluster instead of of buildbot?  The tests bug just lists greening up the tests.

Comment 8

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: bug1364924_puppet.patch

Added the node with the 35 machines which will be changed in the first place.

Comment 9

[Wander Lairson Costa]

(In reply to Kim Moir [:kmoir] from comment #7)
> wcosta: Is the the bug/pr with the changes that you need to do at your end
> to enable tests to be scheduled to run on taskcluster instead of of
> buildbot?  The tests bug just lists greening up the tests.

Bug 1358533

Comment 10

[Greg Arndt [:garndt]]

(In reply to Wander Lairson Costa [:wcosta] from comment #9)
> (In reply to Kim Moir [:kmoir] from comment #7)
> > wcosta: Is the the bug/pr with the changes that you need to do at your end
> > to enable tests to be scheduled to run on taskcluster instead of of
> > buildbot?  The tests bug just lists greening up the tests.
> 
> Bug 1358533

This bug is for running builds natively.

Comment 11

[Kim Moir [:kmoir] ET]

Andrei, just as a fyi, we won't be implementing this Wednesday due to some other issues that arose.  We'll revisit when we can implement tomorrow.

Comment 12

[Kim Moir [:kmoir] ET]

This work is on hold until we have fix our monitoring tools to reflect the state of taskcluster queues and have tooling to reboot machines.

Comment 13

[Kim Moir [:kmoir] ET]

Wander, is there a bug for the taskcluster patches needed to enable tests on native hardware?  The bug in comment 9 is for builds, not tests.

Comment 14

[Wander Lairson Costa]

(In reply to Kim Moir [:kmoir] from comment #13)
> Wander, is there a bug for the taskcluster patches needed to enable tests on
> native hardware?  The bug in comment 9 is for builds, not tests.

I think bug 1298431, although afaik, things are green. I can make a try push to confirm before button press.

Comment 15

[Kim Moir [:kmoir] ET]

Andrei could you  please update this patch to reflect that the first batch we migrate will be 10 machines.  Also, the google doc I sent you has a list of the batches we will try to migrate so you could prepare those patches in advance if you like.

Comment 16

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: batch1.patch

Migration to tasckcluster first 10 machines (batch#1)

Comment 17

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: bug1364924_puppet.patch

patch for puppet including some comments of how the node will look at the end

Comment 18

[Alin Selagea [:aselagea]]

Comment on attachment 8877100 [details] [diff] [review]
bug1364924_puppet.patch

Looks good!

Comment 19

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

This is over,mac test machines were moved to tc.