802589 - Reduce underutilized Leopard pool

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[Armen [:armenzg]]

> We got 22 production Leopard slaves if I can count correctly.
> Yesterday we had only 56 Leopard jobs.
> If I divide that between 10 hours of the day comes to 5.6 jobs/hour.
>
> Any inconvenience if we go down to 15 Leopard slaves? I think we could go even
> lower but I want to accomodate for failing slaves and even higher loads.
> If you think we could go lower please let me know. I would rather see wait
> times on Leopard test jobs.

No one has raised objections to this plan.

I will disable a bunch of them and evaluate after a week or so.

Comment 1

[Armen [:armenzg]]

I noticed something interesting.
We run jetpack jobs for Leopard when we should not.
I will attach a patch to address that as well.

I have disabled a bunch of slaves and left 15 production ones and 1 staging up and running.

Comment 2

[Phil Ringnalda (:philor)]

Jetpack is bug 778969.

Comment 3

[Armen [:armenzg]]

On 10-19:
leopard: 88
  0:       83    94.32%
 15:        5     5.68%

On 10-20:
leopard: 17
  0:       17   100.00%

It seems that bug 778969 took a lot of the load.
I have disabled down to 7 production slaves and 1 staging slave.

We would be able to recuperate 17 rev3 minis.

Comment 4

[Armen [:armenzg]]

On 10-22:
leopard: 115
  0:       57    49.57%
 15:       39    33.91%
 30:       15    13.04%
 45:        2     1.74%
 60:        1     0.87%
 75:        0     0.00%
 90+:        1     0.87%

I have re-enabled 4 slaves back.

Our wait times would depend on how many changes land for:
* TB comm-esr10
* FF mozilla-esr10
* TB comm-release
* FF mozilla-release

I guess that as soon as version 17 goes into release and the esr-17 becomes the new esr we will be able to completely eliminate the pool.

Comment 5

[Armen [:armenzg]]

People keep on pushing.

Adding 1 more slave back and if we have wait times some days for the next 4 weeks let it be.

leopard: 163
  0:      106    65.03%
 15:       35    21.47%
 30:       12     7.36%
 45:        6     3.68%
 60:        3     1.84%
 75:        0     0.00%
 90+:        1     0.61%

Comment 6

[Armen [:armenzg]]

These need to be added:
talos-r3-leopard-002   talos-r3-xp-096
talos-r3-leopard-010   talos-r3-xp-097
talos-r3-leopard-017   talos-r3-xp-098
talos-r3-leopard-018   talos-r3-xp-099
talos-r3-leopard-019   talos-r3-xp-100

talos-r3-leopard-020   talos-r3-w7-100
talos-r3-leopard-021   talos-r3-w7-101
talos-r3-leopard-022   talos-r3-w7-102
talos-r3-leopard-023   talos-r3-w7-103
talos-r3-leopard-024   talos-r3-w7-104


mysql> select * from slaves where name like 'talos-r3-xp-095';
+---------+-----------------+----------+--------+---------+-----------+------+---------+-------+--------+----------------+-----------------+---------+------------------+-------+--------------+
| slaveid | name            | distroid | bitsid | speedid | purposeid | dcid | trustid | envid | poolid | basedir        | locked_masterid | enabled | current_masterid | notes | custom_tplid |
+---------+-----------------+----------+--------+---------+-----------+------+---------+-------+--------+----------------+-----------------+---------+------------------+-------+--------------+
|    2358 | talos-r3-xp-095 |        7 |      1 |       1 |         4 |    7 |       4 |     2 |     19 | C:\talos-slave |            NULL |       1 |               61 |       |         NULL |
+---------+-----------------+----------+--------+---------+-----------+------+---------+-------+--------+----------------+-----------------+---------+------------------+-------+--------------+

mysql> select * from slaves where name like 'talos-r3-w7-095';
+---------+-----------------+----------+--------+---------+-----------+------+---------+-------+--------+----------------+-----------------+---------+------------------+-------+--------------+
| slaveid | name            | distroid | bitsid | speedid | purposeid | dcid | trustid | envid | poolid | basedir        | locked_masterid | enabled | current_masterid | notes | custom_tplid |
+---------+-----------------+----------+--------+---------+-----------+------+---------+-------+--------+----------------+-----------------+---------+------------------+-------+--------------+
|    2374 | talos-r3-w7-095 |        8 |      1 |       1 |         4 |    7 |       4 |     2 |     19 | C:\talos-slave |            NULL |       1 |               61 | NULL  |         NULL |
+---------+-----------------+----------+--------+---------+-----------+------+---------+-------+--------+----------------+-----------------+---------+------------------+-------+--------------+

Comment 7

[Armen [:armenzg]]

Attachment: buildbot-configs

Comment 8

[Armen [:armenzg]]

Attachment: opsi

Comment 9

[Armen [:armenzg]]

Attachment: slavealloc

Comment 10

[Armen [:armenzg]]

Attachment: graphsdb

Comment 11

[Armen [:armenzg]]

Attachment: slavealloc as used

I updated the documentation to have less chances of doing it wrong.

Comment 12

[[e-mail] (:bhearsum)]

The buildbot-configs patch is in production.

Comment 13

[[e-mail] (:bhearsum)]

xp 097 was down and is now back up (bug 807320). Couldn't get it into the production pool because it's not in OPSI yet.

Comment 14

[Armen [:armenzg]]

The XP slaves are done:
https://secure.pub.build.mozilla.org/buildapi/recent/talos-r3-xp-096
https://secure.pub.build.mozilla.org/buildapi/recent/talos-r3-xp-097
https://secure.pub.build.mozilla.org/buildapi/recent/talos-r3-xp-098
https://secure.pub.build.mozilla.org/buildapi/recent/talos-r3-xp-099
https://secure.pub.build.mozilla.org/buildapi/recent/talos-r3-xp-100

Comment 15

[Armen [:armenzg]]

The Win7 slaves are done:
The XP slaves are done:
https://secure.pub.build.mozilla.org/buildapi/recent/talos-r3-w7-100
https://secure.pub.build.mozilla.org/buildapi/recent/talos-r3-w7-101
https://secure.pub.build.mozilla.org/buildapi/recent/talos-r3-w7-102
https://secure.pub.build.mozilla.org/buildapi/recent/talos-r3-w7-103
https://secure.pub.build.mozilla.org/buildapi/recent/talos-r3-w7-104