942206 - Setup in-house buildbot masters for mobile testers

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Chris Cooper [:coop] (he/him)]

These in-house masters have been provisioned and need to be setup to act as test masters for pandas and tegras.

buildbot-master99.srv.releng.scl3.mozilla.com
buildbot-master100.srv.releng.scl3.mozilla.com
buildbot-master101.srv.releng.scl3.mozilla.com
buildbot-master102.srv.releng.scl3.mozilla.com

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

The existing downtime on the service checks just expired, so I've set it again until Dec 4th (a week).

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

And for another week. What's the ETA on this work ?

Comment 3

[Justin Wood (:Callek)]

We currently have 5 panda masters, and 4 tegra masters.

We have 6 total new masters to play with, I'm going to give it a go with 2 tegra and 4 panda to see how things behave (thats ~220 pandas per master, and 125 tegras per master) if things are unstable we can rebalance. We can always check ganglia/graphite for graphs. If there is an objection lets hear it!

Comment 4

[Justin Wood (:Callek)]

Attachment: [config] add to production-masters.json disabled

add then to production-masters.json

Comment 5

[Justin Wood (:Callek)]

Did SQL for these masters just now as well, with r=jhopkins over IRC. [note the actual value-changing stuff is at end]:

mysql> describe pools;
+--------+--------------+------+-----+---------+----------------+
| Field  | Type         | Null | Key | Default | Extra          |
+--------+--------------+------+-----+---------+----------------+
| poolid | int(11)      | NO   | PRI | NULL    | auto_increment |
| name   | varchar(128) | NO   | UNI | NULL    |                |
+--------+--------------+------+-----+---------+----------------+
2 rows in set (0.01 sec)

mysql> select * from pools where name like "%tegra%";
+--------+------------------+
| poolid | name             |
+--------+------------------+
|     23 | tests-mtv1-tegra |
|     53 | tests-tegra      |
+--------+------------------+
2 rows in set (0.01 sec)

mysql> select * from pools where name like "%panda%";
+--------+------------------+
| poolid | name             |
+--------+------------------+
|     55 | tests-panda      |
|     29 | tests-scl1-panda |
+--------+------------------+
2 rows in set (0.00 sec)

mysql> select * from masters where poolid in (23,53);
+----------+-------------------+-----------------------------------------------+----------
-+---------+------+--------+---------+-------+
| masterid | nickname          | fqdn                                          | http_port
 | pb_port | dcid | poolid | enabled | notes |
+----------+-------------------+-----------------------------------------------+----------
-+---------+------+--------+---------+-------+
|       93 | bm19-tests1-tegra | buildbot-master19.build.mtv1.mozilla.com      |      8201
 |    9201 |    5 |     23 |       1 |       |
|       94 | bm20-tests1-tegra | buildbot-master20.build.mtv1.mozilla.com      |      8201
 |    9201 |    5 |     23 |       1 |       |
|       99 | bm22-tests1-tegra | buildbot-master22.build.mtv1.mozilla.com      |      8201
 |    9201 |    5 |     23 |       1 |       |
|      126 | bm10-tests1-tegra | buildbot-master10.build.mtv1.mozilla.com      |      8201
 |    9201 |    5 |     23 |       1 |       |
|      247 | bm95-tests1-tegra | buildbot-master95.srv.releng.usw2.mozilla.com |      8201
 |    9201 |   21 |     53 |       0 | NULL  |
|      249 | bm96-tests1-tegra | buildbot-master96.srv.releng.use1.mozilla.com |      8201
 |    9201 |   19 |     53 |       0 | NULL  |
|      251 | bm97-tests1-tegra | buildbot-master97.srv.releng.usw2.mozilla.com |      8201
 |    9201 |   21 |     53 |       0 | NULL  |
|      253 | bm98-tests1-tegra | buildbot-master98.srv.releng.use1.mozilla.com |      8201
 |    9201 |   19 |     53 |       0 | NULL  |
+----------+-------------------+-----------------------------------------------+----------
-+---------+------+--------+---------+-------+
8 rows in set (0.00 sec)

mysql> select * from masters where poolid in (29,55);
+----------+-------------------+-----------------------------------------------+----------
-+---------+------+--------+---------+------------+
| masterid | nickname          | fqdn                                          | http_port
 | pb_port | dcid | poolid | enabled | notes      |
+----------+-------------------+-----------------------------------------------+----------
-+---------+------+--------+---------+------------+
|      125 | bm29-tests1-panda | buildbot-master29.build.scl1.mozilla.com      |      8201
 |    9201 |    7 |     29 |       1 |            |
|      149 | bm43-tests1-panda | buildbot-master43.build.scl1.mozilla.com      |      8201
 |    9201 |    7 |     29 |       1 |            |
|      151 | bm44-tests1-panda | buildbot-master44.build.scl1.mozilla.com      |      8201
 |    9201 |    7 |     29 |       1 |            |
|      153 | bm42-tests1-panda | buildbot-master42.build.scl1.mozilla.com      |      8201
 |    9201 |    7 |     29 |       1 |            |
|      155 | bm45-tests1-panda | buildbot-master45.build.scl1.mozilla.com      |      8201
 |    9201 |    7 |     29 |       1 | NULL       |
|      237 | bm90-tests1-panda | buildbot-master90.srv.releng.use1.mozilla.com |      8201
 |    9201 |   19 |     55 |       0 | NULL       |
|      239 | bm91-tests1-panda | buildbot-master91.srv.releng.usw2.mozilla.com |      8201
 |    9201 |   21 |     55 |       0 | NULL       |
|      241 | bm92-tests1-panda | buildbot-master92.srv.releng.use1.mozilla.com |      8201
 |    9201 |   19 |     55 |       0 | NULL       |
|      243 | bm93-tests1-panda | buildbot-master93.srv.releng.usw2.mozilla.com |      8201
 |    9201 |   21 |     55 |       0 | NULL       |
|      245 | bm94-tests1-panda | buildbot-master94.srv.releng.use1.mozilla.com |      8201
 |    9201 |   19 |     55 |       0 | NULL       |
|      263 | bm89-tests1-panda | buildbot-master89.srv.releng.scl3.mozilla.com |      8201
 |    9201 |   10 |     55 |       0 | bug 892691 |
+----------+-------------------+-----------------------------------------------+----------
-+---------+------+--------+---------+------------+
11 rows in set (0.00 sec)

mysql> select * from datacenters where dcid=10;
+------+------+
| dcid | name |
+------+------+
|   10 | scl3 |
+------+------+
1 row in set (0.00 sec)

# ...................................... masterid, nick, fqdn, http-port, pb-port, dcid, poolid, enabled, notes
# tegras
mysql> INSERT INTO masters VALUES (NULL, "bm88-tests1-tegra", "buildbot-master88.srv.rele
ng.scl3.mozilla.com", 8201, 9201, 10, 53, 0, NULL), (NULL, "bm99-tests1-tegra", "buildbot
-master99.srv.releng.scl3.mozilla.com", 8201, 9201, 10, 53, 0, NULL);
Query OK, 2 rows affected (0.07 sec)
Records: 2  Duplicates: 0  Warnings: 0
# pandas
mysql> INSERT INTO masters VALUES (NULL, "bm100-tests1-panda", "buildbot-master100.srv.re
leng.scl3.mozilla.com", 8201, 9201, 10, 55, 0, NULL),(NULL, "bm101-tests1-panda", "buildb
ot-master101.srv.releng.scl3.mozilla.com", 8201, 9201, 10, 55, 0, NULL),(NULL, "bm102-tes
ts1-panda", "buildbot-master102.srv.releng.scl3.mozilla.com", 8201, 9201, 10, 55, 0, NULL
);
Query OK, 3 rows affected (0.00 sec)
Records: 3  Duplicates: 0  Warnings: 0

Comment 6

[Justin Wood (:Callek)]

Pushed the production-masters.json live with   https://hg.mozilla.org/build/tools/rev/bddd63ce81de (though I forget to mention in checkin that john reviewed it)

I also pushed puppet change   https://hg.mozilla.org/build/puppet/rev/67456c0af054 with r=nodechangeonly

And then merged it live:
  https://hg.mozilla.org/build/puppet/rev/c32b7164dbaa

Comment 7

[Justin Wood (:Callek)]

enabled in slavealloc and landed https://hg.mozilla.org/build/tools/rev/3dc0afdfc5e3

Comment 8

[Justin Wood (:Callek)]

mysql> select * from pools where name LIKE "%tegra%" or name LIKE "%panda%";
+--------+------------------+
| poolid | name             |
+--------+------------------+
|     23 | tests-mtv1-tegra |
|     55 | tests-panda      |
|     29 | tests-scl1-panda |
|     53 | tests-tegra      |
+--------+------------------+
4 rows in set (0.00 sec)

mysql> update slaves SET poolid=53 WHERE name LIKE "%tegra%" and poolid=23;
Query OK, 347 rows affected (0.07 sec)
Rows matched: 347  Changed: 347  Warnings: 0

mysql> update slaves SET poolid=55 WHERE name LIKE "%panda%" and poolid=29;
Query OK, 845 rows affected (0.08 sec)
Rows matched: 845  Changed: 845  Warnings: 0


--- todo is to check on health of the masters and jobs... This may take a few hours or so as things cut over to new masters.

Comment 9

[Justin Wood (:Callek)]

and I caused trees to close due to forgetting one important piece... setting a slave password:

mysql> insert into slave_passwords VALUES (55,NULL,"<password>"),(53,NULL,"<password>");
Query OK, 2 rows affected (0.00 sec)
Records: 2  Duplicates: 0  Warnings: 0

Comment 10

[Justin Wood (:Callek)]

Filed Bug 951209 and 951210 to shut down the KVM based masters.
Disabled the same in slavealloc and production-masters.json with https://hg.mozilla.org/build/tools/rev/ecebc86a7009

Also filed Bug 950997 in order to make sure they have proper CPU count. -- marking resolved