661525 - Split up test masters by platform (again)

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Chris AtLee [:catlee]]

Our test master instances are huge (see bug 657954), and until we know why, we need to get them to be a bit slimmer.

Let's start by setting up 3 test master instances, each handling one of fedora32/64, leopard/snowleopard, and winxp/win7.

We'll also need separate pools in slavealloc for each of these platforms.

Comment 1

[Mike Taylor [:bear]]

filed bug 661650 for RelOps to create 3 new vm masters

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

So this basically means pools
 tests-scl1-linux
 tests-scl1-darwin
 tests-scl1-windows
which shouldn't be a problem.  Let me know when the masters are set up and ready to go, and I'll add the pools.  Then we can put the masters in those pools and start reassigning slaves to those pools, and finally delete the tests-scl1 pool.

Comment 3

[Aki Sasaki (not active)]

During the reconfig today, bm04-tests and bm06-tests (possibly other test-masters as well) were unresponsive via http, but still running. bm04 reconfiged after multiple attempts; bm06 I had to gracefully shut down.

The builder masters were also sluggish to reconfig, but on the whole I think the test masters overall needed a lot more help.  Pretty sure this bug will help solve that sluggishness.

Comment 4

[Chris AtLee [:catlee]]

Do the puppet manifests or json need to indicate which platforms are running on a test master?

Comment 5

[Chris AtLee [:catlee]]

Attachment: master manifests for buildbot-master{15,16}

Comment 6

[Chris AtLee [:catlee]]

Attachment: json for buildbot-master{15,16}

Comment 7

[[e-mail] (:bhearsum)]

Comment on attachment 538091 [details] [diff] [review]
json for buildbot-master{15,16}

Review of attachment 538091 [details] [diff] [review]:
-----------------------------------------------------------------

Comment 8

[Chris AtLee [:catlee]]

Attachment: teach tests_localconfig.py about limit_platforms

also fix slave port that should have landed a while ago

Comment 9

[[e-mail] (:bhearsum)]

Comment on attachment 538092 [details] [diff] [review]
teach tests_localconfig.py about limit_platforms

Review of attachment 538092 [details] [diff] [review]:
-----------------------------------------------------------------

Comment 10

[Chris AtLee [:catlee]]

Attachment: slavealloc csv for new masters

Comment 11

[Chris AtLee [:catlee]]

Comment on attachment 538094 [details]
slavealloc csv for new masters

imported this into slavealloc, and marked the masters as disabled for now

Comment 12

[Chris AtLee [:catlee]]

bm{15,16}-tests1-windows are now set up. they need the buildbot-configs patch above merged to production, and then updated on the masters.

also waiting on nagios checks and mysql access.

one those are ready, we can start the masters, and move some windows slaves into the tests-scl1-windows pool.

Comment 13

[Chris AtLee [:catlee]]

Attachment: enable bm15,16

Comment 14

[Chris AtLee [:catlee]]

Attachment: reduce platforms on existing test masters

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

We have a bit of a backlog of win64 talos & tests (currently 323) because the three active slaves are connected to buildbot-master05 and 06, which don't have win64 in the list of ACTIVE_PLATFORMS anymore. I'd shift them to test-master01, but I don't know how busy that is with the Tegras and don't want to destabilise that.

Comment 16

[Armen [:armenzg]]

(In reply to comment #15)
> We have a bit of a backlog of win64 talos & tests (currently 323) because
> the three active slaves are connected to buildbot-master05 and 06, which
> don't have win64 in the list of ACTIVE_PLATFORMS anymore. I'd shift them to
> test-master01, but I don't know how busy that is with the Tegras and don't
> want to destabilise that.

I had lost track of where I read the comment.
I dealt with these slaves yesterday by moving them to test-master01 for now.

Comment 17

[Chris AtLee [:catlee]]

Attachment: Remove slaves for non-active platforms

Doesn't make sense to have BuildSlaves created that will never be used. This patch prevents creation of BuildSlaves for non-active platforms, with a bit of ugliness for current projects that rely on platforms being defined.

Comment 18

[Chris AtLee [:catlee]]

Attachment: basic manifest for bm17

Comment 19

[[e-mail] (:bhearsum)]

Comment on attachment 540754 [details] [diff] [review]
Remove slaves for non-active platforms

Review of attachment 540754 [details] [diff] [review]:
-----------------------------------------------------------------

Comment 20

[Chris AtLee [:catlee]]

Splitting out just windows is good enough for now.