1292656 - Reduce 10.6 test pool to size appropriate for testing only mozilla-esr45 and comm-esr45

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Chris Cooper [:coop] (he/him)]

Only our ESR branches (Firefox and Thunderbird) are running on our 10.6 machines now. We have 50 machines configured right now, but can probably get by with around 10.

We'll need to remove the extra machines from our configs, etc.

Comment 1

[Chris Cooper [:coop] (he/him)]

(In reply to Chris Cooper [:coop] from comment #0)
> Only our ESR branches (Firefox and Thunderbird) are running on our 10.6
> machines now. We have 50 machines configured right now, but can probably get
> by with around 10.

Amy: how many minis per rack shelf? If reclaiming rack space is an issue, we could make it a multiple of that number.

Comment 2

[Amy Rich [:arr] [:arich]]

It's 2 per shelf, so just make it an even number. The rack view in inventory will show you which are on the same shelf so you can pick ones that are together. I think we have some intermingled with builders or new testers, but if we could free up a whole rack, that would be awesome. It means we could get rid of some infrastructure (switching gear, PDUs, racks) too.

Comment 3

[Phil Ringnalda (:philor)]

Whether or not it's supposed to, mozilla-release is still running 10.6 tests, so you're missing either a dependency on a bug about shutting that off, or a dependency on the *next* mergeday.

Comment 4

[Alin Selagea [:aselagea]]

At the moment, we still run OS X 10.6 tests on ESR branches + mozilla-release and try (if explicitly specified).

Comment 5

[Alin Selagea [:aselagea]]

Out of 50 enabled slaves, 48 are in "201-9 - scl3" rack and 2 in "201-8 - scl3", so I don't think we could free up a whole rack. Moreover, in bug 1279394 we disabled and removed from configs 69 minis, but they are still present in inventory with status=production (I suppose they weren't removed from the rack).

Coop: any ideas on how many minis we still want to keep enabled?

Comment 6

[Chris Cooper [:coop] (he/him)]

(In reply to Alin Selagea [:aselagea][:buildduty] from comment #4)
> At the moment, we still run OS X 10.6 tests on ESR branches +
> mozilla-release and try (if explicitly specified).

Hrmmm, you're right. We need to support 10.6 on release for at least the next 6 weeks in case of a chemspill in the Firefox 48 cycle.

In that case, we should step down to 20 machines for now, and then 10 machines once Firefox 49 is out.

Comment 7

[Alin Selagea [:aselagea]]

Disabled the two minis from "201-8 - scl3" (t-snow-r4-0127 and t-snow-r4-0128) and another 28 from "201-9 - scl3".

mysql> select count(*) from slaves where (name like 't-snow-r4%' and notes='Disabled in bug 1292656' and enabled=0);
+----------+
| count(*) |
+----------+
|       30 |
+----------+
1 row in set (0.00 sec)

mysql> select count(*) from slaves where name like 't-snow-r4%' and enabled=1;
+----------+
| count(*) |
+----------+
|       20 |
+----------+
1 row in set (0.00 sec)

Comment 8

[Alin Selagea [:aselagea]]

Attachment: bug_1292656.patch

Patch to remove 30 minis from bb-configs.

Comment 9

[Alin Selagea [:aselagea]]

From what I can see, most of the disabled minis are still present in inventory. Also, the Nagios checks are still up. 
@Amy: do you know if there are any plans for these?

Comment 10

[Amy Rich [:arr] [:arich]]

:allin: We'll need to file a decom bug with dcops for the entire list of 10.6 machines that have been shut down (both batches).

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

<nagios-releng> Tue 15:10:26 PDT [4112] nagios1.private.releng.scl3.mozilla.com:t-snow-r4-machines-ping cluster is WARNING: CLUSTER WARNING: t-snow-r4-machines-ping cluster: 72 ok, 0 warning, 0 unknown, 51 critical

Downtimed for 24 hours, could someone fix up the thresholds ?

Comment 12

[Amy Rich [:arr] [:arich]]

The hosts that are being decommissioned should just be removed frmo nagios, and that will clear up on its own.

Comment 13

[Phil Ringnalda (:philor)]

Attachment: Warn off trychooser users

Nobody reads, so this won't actually reduce our "startup, crash" try load to nothing, but at least it will feel useful.

Comment 14

[Alin Selagea [:aselagea]]

Comment on attachment 8788012 [details] [diff] [review]
Warn off trychooser users

Thanks!

Comment 15

[Alin Selagea [:aselagea]]

Updated TryChooser to include the change above.

Comment 16

[Alin Selagea [:aselagea]]

Minis have been unracked and removed from nagios, I also did some DB cleaning. Noticed that the inventory entries have not been updated/removed yet. 
@Amy: should that be done too?

Comment 17

[Amy Rich [:arr] [:arich]]

Van should be using invtool to mark them as decommed in inventory once he removes them. I've NIed him for that.

Comment 18

[Van Le [:van]]

all the retired r4s have been set to decommissioned and location as removed as we are donating these minis.

Comment 19

[Alin Selagea [:aselagea]]

Great, thanks!

Comment 20

[Alin Selagea [:aselagea]]

Per #c6, we should decommission another 10 t-snow-r4 machines since Firefox 49 is out.

Comment 21

[Alin Selagea [:aselagea]]

Attachment: bug_1292656_2.patch

Preferred to keep the last 10 machines of the current pool so that we don't include t-snow-r4-085 (which has RAM issues).

Comment 22

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/02b3c211cc48