Closed
Bug 942206
Opened 11 years ago
Closed 11 years ago
Setup in-house buildbot masters for mobile testers
Categories
(Infrastructure & Operations Graveyard :: CIDuty, task)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: coop, Assigned: Callek)
References
Details
(Whiteboard: [capacity][buildduty])
Attachments
(1 file)
6.99 KB,
patch
|
jhopkins
:
review+
|
Details | Diff | Splinter Review |
These in-house masters have been provisioned and need to be setup to act as test masters for pandas and tegras. buildbot-master99.srv.releng.scl3.mozilla.com buildbot-master100.srv.releng.scl3.mozilla.com buildbot-master101.srv.releng.scl3.mozilla.com buildbot-master102.srv.releng.scl3.mozilla.com
Comment 1•11 years ago
|
||
The existing downtime on the service checks just expired, so I've set it again until Dec 4th (a week).
Comment 2•11 years ago
|
||
And for another week. What's the ETA on this work ?
Assignee | ||
Comment 3•11 years ago
|
||
We currently have 5 panda masters, and 4 tegra masters. We have 6 total new masters to play with, I'm going to give it a go with 2 tegra and 4 panda to see how things behave (thats ~220 pandas per master, and 125 tegras per master) if things are unstable we can rebalance. We can always check ganglia/graphite for graphs. If there is an objection lets hear it!
Assignee | ||
Comment 4•11 years ago
|
||
add then to production-masters.json
Attachment #8347389 -
Flags: review?(jhopkins)
Updated•11 years ago
|
Attachment #8347389 -
Flags: review?(jhopkins) → review+
Assignee | ||
Comment 5•11 years ago
|
||
Did SQL for these masters just now as well, with r=jhopkins over IRC. [note the actual value-changing stuff is at end]: mysql> describe pools; +--------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +--------+--------------+------+-----+---------+----------------+ | poolid | int(11) | NO | PRI | NULL | auto_increment | | name | varchar(128) | NO | UNI | NULL | | +--------+--------------+------+-----+---------+----------------+ 2 rows in set (0.01 sec) mysql> select * from pools where name like "%tegra%"; +--------+------------------+ | poolid | name | +--------+------------------+ | 23 | tests-mtv1-tegra | | 53 | tests-tegra | +--------+------------------+ 2 rows in set (0.01 sec) mysql> select * from pools where name like "%panda%"; +--------+------------------+ | poolid | name | +--------+------------------+ | 55 | tests-panda | | 29 | tests-scl1-panda | +--------+------------------+ 2 rows in set (0.00 sec) mysql> select * from masters where poolid in (23,53); +----------+-------------------+-----------------------------------------------+---------- -+---------+------+--------+---------+-------+ | masterid | nickname | fqdn | http_port | pb_port | dcid | poolid | enabled | notes | +----------+-------------------+-----------------------------------------------+---------- -+---------+------+--------+---------+-------+ | 93 | bm19-tests1-tegra | buildbot-master19.build.mtv1.mozilla.com | 8201 | 9201 | 5 | 23 | 1 | | | 94 | bm20-tests1-tegra | buildbot-master20.build.mtv1.mozilla.com | 8201 | 9201 | 5 | 23 | 1 | | | 99 | bm22-tests1-tegra | buildbot-master22.build.mtv1.mozilla.com | 8201 | 9201 | 5 | 23 | 1 | | | 126 | bm10-tests1-tegra | buildbot-master10.build.mtv1.mozilla.com | 8201 | 9201 | 5 | 23 | 1 | | | 247 | bm95-tests1-tegra | buildbot-master95.srv.releng.usw2.mozilla.com | 8201 | 9201 | 21 | 53 | 0 | NULL | | 249 | bm96-tests1-tegra | buildbot-master96.srv.releng.use1.mozilla.com | 8201 | 9201 | 19 | 53 | 0 | NULL | | 251 | bm97-tests1-tegra | buildbot-master97.srv.releng.usw2.mozilla.com | 8201 | 9201 | 21 | 53 | 0 | NULL | | 253 | bm98-tests1-tegra | buildbot-master98.srv.releng.use1.mozilla.com | 8201 | 9201 | 19 | 53 | 0 | NULL | +----------+-------------------+-----------------------------------------------+---------- -+---------+------+--------+---------+-------+ 8 rows in set (0.00 sec) mysql> select * from masters where poolid in (29,55); +----------+-------------------+-----------------------------------------------+---------- -+---------+------+--------+---------+------------+ | masterid | nickname | fqdn | http_port | pb_port | dcid | poolid | enabled | notes | +----------+-------------------+-----------------------------------------------+---------- -+---------+------+--------+---------+------------+ | 125 | bm29-tests1-panda | buildbot-master29.build.scl1.mozilla.com | 8201 | 9201 | 7 | 29 | 1 | | | 149 | bm43-tests1-panda | buildbot-master43.build.scl1.mozilla.com | 8201 | 9201 | 7 | 29 | 1 | | | 151 | bm44-tests1-panda | buildbot-master44.build.scl1.mozilla.com | 8201 | 9201 | 7 | 29 | 1 | | | 153 | bm42-tests1-panda | buildbot-master42.build.scl1.mozilla.com | 8201 | 9201 | 7 | 29 | 1 | | | 155 | bm45-tests1-panda | buildbot-master45.build.scl1.mozilla.com | 8201 | 9201 | 7 | 29 | 1 | NULL | | 237 | bm90-tests1-panda | buildbot-master90.srv.releng.use1.mozilla.com | 8201 | 9201 | 19 | 55 | 0 | NULL | | 239 | bm91-tests1-panda | buildbot-master91.srv.releng.usw2.mozilla.com | 8201 | 9201 | 21 | 55 | 0 | NULL | | 241 | bm92-tests1-panda | buildbot-master92.srv.releng.use1.mozilla.com | 8201 | 9201 | 19 | 55 | 0 | NULL | | 243 | bm93-tests1-panda | buildbot-master93.srv.releng.usw2.mozilla.com | 8201 | 9201 | 21 | 55 | 0 | NULL | | 245 | bm94-tests1-panda | buildbot-master94.srv.releng.use1.mozilla.com | 8201 | 9201 | 19 | 55 | 0 | NULL | | 263 | bm89-tests1-panda | buildbot-master89.srv.releng.scl3.mozilla.com | 8201 | 9201 | 10 | 55 | 0 | bug 892691 | +----------+-------------------+-----------------------------------------------+---------- -+---------+------+--------+---------+------------+ 11 rows in set (0.00 sec) mysql> select * from datacenters where dcid=10; +------+------+ | dcid | name | +------+------+ | 10 | scl3 | +------+------+ 1 row in set (0.00 sec) # ...................................... masterid, nick, fqdn, http-port, pb-port, dcid, poolid, enabled, notes # tegras mysql> INSERT INTO masters VALUES (NULL, "bm88-tests1-tegra", "buildbot-master88.srv.rele ng.scl3.mozilla.com", 8201, 9201, 10, 53, 0, NULL), (NULL, "bm99-tests1-tegra", "buildbot -master99.srv.releng.scl3.mozilla.com", 8201, 9201, 10, 53, 0, NULL); Query OK, 2 rows affected (0.07 sec) Records: 2 Duplicates: 0 Warnings: 0 # pandas mysql> INSERT INTO masters VALUES (NULL, "bm100-tests1-panda", "buildbot-master100.srv.re leng.scl3.mozilla.com", 8201, 9201, 10, 55, 0, NULL),(NULL, "bm101-tests1-panda", "buildb ot-master101.srv.releng.scl3.mozilla.com", 8201, 9201, 10, 55, 0, NULL),(NULL, "bm102-tes ts1-panda", "buildbot-master102.srv.releng.scl3.mozilla.com", 8201, 9201, 10, 55, 0, NULL ); Query OK, 3 rows affected (0.00 sec) Records: 3 Duplicates: 0 Warnings: 0
Assignee | ||
Comment 6•11 years ago
|
||
Pushed the production-masters.json live with https://hg.mozilla.org/build/tools/rev/bddd63ce81de (though I forget to mention in checkin that john reviewed it) I also pushed puppet change https://hg.mozilla.org/build/puppet/rev/67456c0af054 with r=nodechangeonly And then merged it live: https://hg.mozilla.org/build/puppet/rev/c32b7164dbaa
Assignee | ||
Comment 7•11 years ago
|
||
enabled in slavealloc and landed https://hg.mozilla.org/build/tools/rev/3dc0afdfc5e3
Assignee | ||
Comment 8•11 years ago
|
||
mysql> select * from pools where name LIKE "%tegra%" or name LIKE "%panda%"; +--------+------------------+ | poolid | name | +--------+------------------+ | 23 | tests-mtv1-tegra | | 55 | tests-panda | | 29 | tests-scl1-panda | | 53 | tests-tegra | +--------+------------------+ 4 rows in set (0.00 sec) mysql> update slaves SET poolid=53 WHERE name LIKE "%tegra%" and poolid=23; Query OK, 347 rows affected (0.07 sec) Rows matched: 347 Changed: 347 Warnings: 0 mysql> update slaves SET poolid=55 WHERE name LIKE "%panda%" and poolid=29; Query OK, 845 rows affected (0.08 sec) Rows matched: 845 Changed: 845 Warnings: 0 --- todo is to check on health of the masters and jobs... This may take a few hours or so as things cut over to new masters.
Assignee | ||
Comment 9•11 years ago
|
||
and I caused trees to close due to forgetting one important piece... setting a slave password: mysql> insert into slave_passwords VALUES (55,NULL,"<password>"),(53,NULL,"<password>"); Query OK, 2 rows affected (0.00 sec) Records: 2 Duplicates: 0 Warnings: 0
Assignee | ||
Comment 10•11 years ago
|
||
Filed Bug 951209 and 951210 to shut down the KVM based masters. Disabled the same in slavealloc and production-masters.json with https://hg.mozilla.org/build/tools/rev/ecebc86a7009 Also filed Bug 950997 in order to make sure they have proper CPU count. -- marking resolved
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Updated•6 years ago
|
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
Updated•4 years ago
|
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•