Closed Bug 971780 Opened 11 years ago Closed 11 years ago

Setup in-house buildbot masters for remaining in-house testers

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

Product:
Infrastructure & Operations Graveyard
Component:
CIDuty
Platform:
x86
Linux
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: Callek, Assigned: Callek)

References

Details

(Whiteboard: [capacity][buildduty])

Attachments

(5 files)

[tools] add masters to production-masters.json disabled
13.76 KB, patch
armenzg
: review+
Callek
: checked-in+
Details | Diff | Splinter Review
[puppet] add master defs to puppet
3.72 KB, patch
armenzg
: review+
Callek
: checked-in+
Details | Diff | Splinter Review
sql - part 1 (adds masters and new pools)
10.34 KB, text/plain
coop
: review+
Callek
: checked-in+
Details
[tools] enable masters
3.85 KB, patch
bhearsum
: review+
Callek
: checked-in+
Details | Diff | Splinter Review
SQL - part 2
33.17 KB, text/plain
bhearsum
: review+
Callek
: checked-in+
Details
+++ This bug was initially created as a clone of Bug #942206 +++
Blocks: 867593
Assignee: nobody → bugspam.Callek
saying ~ 90 hosts per master (ignoring enabled/disabled -- caring only about "prod" not "decomm") talos-r3-fed-% --- 89 talos-r3-fed64-% --- 31 talos-linux32-ix --- 55 talos-linux64-ix --- 119 ------------------->> 294 mysql> select * from slaves where (dcid=5 OR dcid=7 OR dcid=10 or dcid=13) and envid=2 and poolid IN (22,51,47,49) AND name NOT like "panda%" and name NOT like "%ec2%" and name NOT like "%spot%"; (confirms number above) == 3 linux test masters t-mavericks-r5 --- 5 talos-mtnlion-r5 --- 88 talos-r4-snow --- 164 ------------------>> 257 mysql> select * from slaves where (dcid=5 OR dcid=7 OR dcid=10 or dcid=13) and envid=2 and poolid IN (11,28) AND name NOT like "%ec2%" and name NOT like "%spot%"; (confirms number above) === 3 osx masters (current on AWS = 6) t-w732-ix --- 126 t-w864-ix --- 129 t-xp32-ix --- 128 ------------>> 383 mysql> select * from slaves where (dcid=5 OR dcid=7 OR dcid=10 or dcid=13) and envid=2 and poolid IN (19,31) AND name NOT like "%ec2%" and name NOT like "%spot%"; (confirms it) === 4 win masters (current on AWS = 6) which === 10 new masters
Depends on: 971812
breakdown of 3 linux; 3 mac; 4 win; as per earlier comments.
Attachment #8377632 - Flags: review?(armenzg)
This adds them to puppet (so puppet creates the master dirs) It also [as a ride-along] moves the windows node def to the tester section rather than being lumped against the bottom
Attachment #8377635 - Flags: review?(armenzg)
Attachment #8377632 - Flags: review?(armenzg) → review+
Attachment #8377635 - Flags: review?(armenzg) → review+
coop, attached is a full list of what I did/need to do. The review is for the INSERT INTO pools .... and INSERT INTO masters near the end the dcid for slaves is to identify that they are all in multi-dc's and thus why I'm not naming the pools by dc.
Attachment #8378446 - Flags: review?(coop)
Attachment #8378446 - Attachment is patch: false
Attachment #8378446 - Flags: review?(coop) → review+
Comment on attachment 8377632 [details] [diff] [review] [tools] add masters to production-masters.json disabled https://hg.mozilla.org/build/tools/rev/19569ea375a7
Attachment #8377632 - Flags: checked-in+
Comment on attachment 8378446 [details] sql - part 1 (adds masters and new pools) (forgive my console wrap) mysql> insert into pools (name) VALUES ("tests-inhouse-linux"),("tests-inhouse-windows"),("tests-inhouse-macosx"); Query OK, 3 rows affected (0.03 sec) Records: 3 Duplicates: 0 Warnings: 0 mysql> DESCRIBE INSERT INTO masters (nickname, fqdn, http_port, pb_port, dcid, poolid, enabled) VALUES ("bm103-tests1-linux", "buildbot-master103.srv.releng.scl3.mozilla.com", 8201, 9201, (SELECT dcid FROM datacenters where name="scl3"), (SELECT poolid FROM pools WHERE name="tests-inhouse-linux"), 0 ), ("bm104-tests1-linux", "buildbot-master104.srv.releng.scl3.mozil la.com", 8201, 9201, (SELECT dcid FROM datacenters where name="scl3"), (SELECT poolid FROM pools WHERE name="tests-inhou se-linux"), 0 ), ("bm105-tests1-linux", "buildbot-master105.srv.releng.scl3.mozilla.com", 8201, 9201, (SELECT dcid F ROM datacenters where name="scl3"), (SELECT poolid FROM pools WHERE name="tests-inhouse-linux"), 0 ), ("bm106-tests1-macos x", "buildbot-master106.srv.releng.scl3.mozilla.com", 8201, 9201, (SELECT dcid FROM datacenters where name="scl3"), (S ELECT poolid FROM pools WHERE name="tests-inhouse-macosx"), 0 ), ("bm107-tests1-macosx", "buildbot-master107.srv.releng.sc l3.mozilla.com", 8201, 9201, (SELECT dcid FROM datacenters where name="scl3"), (SELECT poolid FROM pools WHERE name="tes ts-inhouse-macosx"), 0 ), ("bm108-tests1-macosx", "buildbot-master108.srv.releng.scl3.mozilla.com", 8201, 9201, (SEL ECT dcid FROM datacenters where name="scl3"), (SELECT poolid FROM pools WHERE name="tests-inhouse-macosx"), 0 ), ("bm109-t ests1-windows", "buildbot-master109.srv.releng.scl3.mozilla.com", 8201, 9201, (SELECT dcid FROM datacenters where name=" scl3"), (SELECT poolid FROM pools WHERE name="tests-inhouse-windows"), 0 ), ("bm110-tests1-windows", "buildbot-master110 .srv.releng.scl3.mozilla.com", 8201, 9201, (SELECT dcid FROM datacenters where name="scl3"), (SELECT poolid FROM pools W HERE name="tests-inhouse-windows"), 0 ), ("bm111-tests1-windows", "buildbot-master111.srv.releng.scl3.mozilla.com", 8201 , 9201, (SELECT dcid FROM datacenters where name="scl3"), (SELECT poolid FROM pools WHERE name="tests-inhouse-windows"), 0 ), ("bm112-tests1-windows", "buildbot-master112.srv.releng.scl3.mozilla.com", 8201, 9201, (SELECT dcid FROM datacen ters where name="scl3"), (SELECT poolid FROM pools WHERE name="tests-inhouse-windows"), 0 ); +----+-------------+-------------+-------+---------------+------+---------+-------+------+----------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------------+-------+---------------+------+---------+-------+------+----------------+ | 1 | PRIMARY | NULL | NULL | NULL | NULL | NULL | NULL | NULL | No tables used | | 21 | SUBQUERY | pools | const | name | name | 386 | const | 1 | NULL | | 20 | SUBQUERY | datacenters | const | name | name | 386 | const | 1 | NULL | | 19 | SUBQUERY | pools | const | name | name | 386 | const | 1 | NULL | | 18 | SUBQUERY | datacenters | const | name | name | 386 | const | 1 | NULL | | 17 | SUBQUERY | pools | const | name | name | 386 | const | 1 | NULL | | 16 | SUBQUERY | datacenters | const | name | name | 386 | const | 1 | NULL | | 15 | SUBQUERY | pools | const | name | name | 386 | const | 1 | NULL | | 14 | SUBQUERY | datacenters | const | name | name | 386 | const | 1 | NULL | | 13 | SUBQUERY | pools | const | name | name | 386 | const | 1 | NULL | | 12 | SUBQUERY | datacenters | const | name | name | 386 | const | 1 | NULL | | 11 | SUBQUERY | pools | const | name | name | 386 | const | 1 | NULL | | 10 | SUBQUERY | datacenters | const | name | name | 386 | const | 1 | NULL | | 9 | SUBQUERY | pools | const | name | name | 386 | const | 1 | NULL | | 8 | SUBQUERY | datacenters | const | name | name | 386 | const | 1 | NULL | | 7 | SUBQUERY | pools | const | name | name | 386 | const | 1 | NULL | | 6 | SUBQUERY | datacenters | const | name | name | 386 | const | 1 | NULL | | 5 | SUBQUERY | pools | const | name | name | 386 | const | 1 | NULL | | 4 | SUBQUERY | datacenters | const | name | name | 386 | const | 1 | NULL | | 3 | SUBQUERY | pools | const | name | name | 386 | const | 1 | NULL | | 2 | SUBQUERY | datacenters | const | name | name | 386 | const | 1 | NULL | +----+-------------+-------------+-------+---------------+------+---------+-------+------+----------------+ 21 rows in set (0.00 sec) mysql> INSERT INTO masters (nickname, fqdn, http_port, pb_port, dcid, poolid, enabled) VALUES ("bm103-tests1-linux", "buildbot -master103.srv.releng.scl3.mozilla.com", 8201, 9201, (SELECT dcid FROM datacenters where name="scl3"), (SELECT poolid FR OM pools WHERE name="tests-inhouse-linux"), 0 ), ("bm104-tests1-linux", "buildbot-master104.srv.releng.scl3.mozilla.com", 8201, 9201, (SELECT dcid FROM datacenters where name="scl3"), (SELECT poolid FROM pools WHERE name="tests-inhouse-linux" ), 0 ), ("bm105-tests1-linux", "buildbot-master105.srv.releng.scl3.mozilla.com", 8201, 9201, (SELECT dcid FROM datac enters where name="scl3"), (SELECT poolid FROM pools WHERE name="tests-inhouse-linux"), 0 ), ("bm106-tests1-macosx", "bu ildbot-master106.srv.releng.scl3.mozilla.com", 8201, 9201, (SELECT dcid FROM datacenters where name="scl3"), (SELECT poo lid FROM pools WHERE name="tests-inhouse-macosx"), 0 ), ("bm107-tests1-macosx", "buildbot-master107.srv.releng.scl3.mozill a.com", 8201, 9201, (SELECT dcid FROM datacenters where name="scl3"), (SELECT poolid FROM pools WHERE name="tests-inhous e-macosx"), 0 ), ("bm108-tests1-macosx", "buildbot-master108.srv.releng.scl3.mozilla.com", 8201, 9201, (SELECT dcid FROM datacenters where name="scl3"), (SELECT poolid FROM pools WHERE name="tests-inhouse-macosx"), 0 ), ("bm109-tests1-win dows", "buildbot-master109.srv.releng.scl3.mozilla.com", 8201, 9201, (SELECT dcid FROM datacenters where name="scl3"), (SELECT poolid FROM pools WHERE name="tests-inhouse-windows"), 0 ), ("bm110-tests1-windows", "buildbot-master110.srv.rele ng.scl3.mozilla.com", 8201, 9201, (SELECT dcid FROM datacenters where name="scl3"), (SELECT poolid FROM pools WHERE name ="tests-inhouse-windows"), 0 ), ("bm111-tests1-windows", "buildbot-master111.srv.releng.scl3.mozilla.com", 8201, 9201, (SELECT dcid FROM datacenters where name="scl3"), (SELECT poolid FROM pools WHERE name="tests-inhouse-windows"), 0 ), ( "bm112-tests1-windows", "buildbot-master112.srv.releng.scl3.mozilla.com", 8201, 9201, (SELECT dcid FROM datacenters wher e name="scl3"), (SELECT poolid FROM pools WHERE name="tests-inhouse-windows"), 0 ); Query OK, 10 rows affected, 1 warning (0.01 sec) Records: 10 Duplicates: 0 Warnings: 1
Attachment #8378446 - Flags: checked-in+
Attachment #8379063 - Flags: review?(bhearsum)
Attachment #8379063 - Flags: review?(bhearsum) → review+
Comment on attachment 8379063 [details] [diff] [review] [tools] enable masters https://hg.mozilla.org/build/tools/rev/740c5deeae94 Also enabled them in slavealloc
Attachment #8379063 - Flags: checked-in+
Attached file SQL - part 2 —
review is for the last 3 statements (DESCRIBE) This enables all the inhouse slaves to the new pools, and documents the prior pool state incase of rollback need
Attachment #8379077 - Flags: review?(bhearsum)
Attachment #8379077 - Flags: review?(bhearsum) → review+
Comment on attachment 8379077 [details] SQL - part 2 mysql> UPDATE slaves SET poolid=61 where (dcid=5 OR dcid=7 OR dcid=10 or dcid=13) and envid=2 and poolid IN (22,51,47,49) AND name NOT like "panda%" and name NOT like "%ec2%" and name NOT like "%spot%"; Query OK, 292 rows affected (0.05 sec) Rows matched: 292 Changed: 292 Warnings: 0 mysql> UPDATE slaves SET poolid=63 where (dcid=5 OR dcid=7 OR dcid=10 or dcid=13) and envid=2 and poolid IN (19,31) AND name NOT like "%ec2%" and name NOT like "%spot%"; Query OK, 383 rows affected (0.02 sec) Rows matched: 383 Changed: 383 Warnings: 0 mysql> UPDATE slaves SET poolid=65 where (dcid=5 OR dcid=7 OR dcid=10 or dcid=13) and envid=2 and poolid IN (11,28) AND name NOT like "%ec2%" and name NOT like "%spot%"; Query OK, 257 rows affected (0.02 sec) Rows matched: 257 Changed: 257 Warnings: 0
Attachment #8379077 - Flags: checked-in+
mysql> SELECT COUNT(*), p.name, s.poolid FROM slaves as s, pools as p WHERE s.poolid=p.poolid GROUP BY p.name ORDER BY p.name; +----------+---------------------------+--------+ | COUNT(*) | name | poolid | +----------+---------------------------+--------+ | 308 | build-aws-us-east-1 | 37 | | 550 | build-aws-us-west-2 | 41 | | 343 | build-scl3 | 24 | | 79 | preprod | 8 | | 75 | preprod-tests | 16 | | 7 | servo | 52 | | 1211 | tests-aws-us-east-1-linux | 47 | | 1247 | tests-aws-us-west-2-linux | 49 | | 292 | tests-inhouse-linux | 61 | | 257 | tests-inhouse-macosx | 65 | | 383 | tests-inhouse-windows | 63 | | 845 | tests-panda | 55 | | 29 | tests-scl1-linux | 22 | | 69 | tests-scl1-macosx | 11 | | 16 | tests-scl1-windows | 19 | | 1 | tests-scl3-macosx | 28 | | 347 | tests-tegra | 53 | | 498 | try-aws-us-east-1 | 39 | | 800 | try-aws-us-west-2 | 43 | | 168 | try-scl3 | 25 | +----------+---------------------------+--------+ 20 rows in set (0.01 sec) Looks like I should track down the outliers on the tests-scl* and why
Status Update -- flows are/were busted, reverting slavealloc pool changes...
(In reply to Justin Wood (:Callek) from comment #13) > Status Update -- flows are/were busted, reverting slavealloc pool changes... correction, flows are *now* back in place, but turns out I forgot to add passwords for the new pools. with :bhearsum, :catlee, and :jlunds help we're rebooting some ~900 slaves to get them back into production. Including the painful windows ones
re: c#14 all clear re: c#12 looks like mostly decom slaves, but a few non: mysql> SELECT s.name as slavename, -> d.name as distro, -> b.name as bits, -> purpose.name as type, -> dc.name as DC, -> t.name as trust, -> e.name as enviro, -> p.name as pool, -> CASE WHEN s.locked_masterid IS NULL THEN "True" ELSE "False" END as locked, -> s.enabled as enabled -> FROM slaves as s, pools as p, bitlengths as b, distros as d, purposes as purpose, datacenters as dc, trustlevels as t, e vironments as e -> WHERE s.poolid=p.poolid AND s.distroid=d.distroid AND b.bitsid=s.bitsid -> AND purpose.purposeid=s.purposeid AND dc.dcid=s.dcid AND t.trustid=s.trustid -> AND e.envid=s.envid -> AND p.name LIKE "tests-scl%" -> ; +----------------------+----------+------+-------+---------------+-------+--------+--------------------+--------+---------+ | slavename | distro | bits | type | DC | trust | enviro | pool | locked | enabled | +----------------------+----------+------+-------+---------------+-------+--------+--------------------+--------+---------+ | talos-r3-fed64-002 | fedora12 | 64 | tests | scl1 | try | decomm | tests-scl1-linux | True | 0 | | talos-r3-fed64-003 | fedora12 | 64 | tests | scl1 | try | decomm | tests-scl1-linux | True | 0 | | talos-r3-fed64-004 | fedora12 | 64 | tests | scl1 | try | decomm | tests-scl1-linux | True | 0 | | talos-r3-fed64-007 | fedora12 | 64 | tests | scl1 | try | decomm | tests-scl1-linux | True | 0 | | talos-r3-fed64-010 | fedora12 | 64 | tests | scl1 | try | decomm | tests-scl1-linux | True | 0 | | talos-r3-fed64-005 | fedora12 | 64 | tests | scl1 | try | decomm | tests-scl1-linux | True | 0 | | talos-r3-fed64-006 | fedora12 | 64 | tests | scl1 | try | decomm | tests-scl1-linux | True | 0 | | talos-r3-fed64-008 | fedora12 | 64 | tests | scl1 | try | decomm | tests-scl1-linux | True | 0 | | talos-r3-fed64-009 | fedora12 | 64 | tests | scl1 | try | decomm | tests-scl1-linux | True | 0 | | talos-r3-fed64-011 | fedora12 | 64 | tests | scl1 | try | decomm | tests-scl1-linux | True | 0 | | talos-r3-fed64-012 | fedora12 | 64 | tests | scl1 | try | decomm | tests-scl1-linux | True | 0 | | talos-r3-fed64-013 | fedora12 | 64 | tests | scl1 | try | decomm | tests-scl1-linux | True | 0 | | talos-r3-fed64-015 | fedora12 | 64 | tests | scl1 | try | decomm | tests-scl1-linux | True | 0 | | talos-r3-fed64-016 | fedora12 | 64 | tests | scl1 | try | decomm | tests-scl1-linux | True | 0 | | talos-r3-fed64-017 | fedora12 | 64 | tests | scl1 | try | decomm | tests-scl1-linux | True | 0 | | talos-r3-fed64-018 | fedora12 | 64 | tests | scl1 | try | decomm | tests-scl1-linux | True | 0 | | talos-r3-fed64-019 | fedora12 | 64 | tests | scl1 | try | decomm | tests-scl1-linux | True | 0 | | talos-r3-fed64-020 | fedora12 | 64 | tests | scl1 | try | decomm | tests-scl1-linux | True | 0 | | talos-r3-fed64-021 | fedora12 | 64 | tests | scl1 | try | decomm | tests-scl1-linux | True | 0 | | talos-r3-fed64-022 | fedora12 | 64 | tests | scl1 | try | decomm | tests-scl1-linux | True | 0 | | talos-r3-fed64-023 | fedora12 | 64 | tests | scl1 | try | decomm | tests-scl1-linux | True | 0 | | talos-r3-fed64-024 | fedora12 | 64 | tests | scl1 | try | decomm | tests-scl1-linux | True | 0 | | talos-r3-fed64-025 | fedora12 | 64 | tests | scl1 | try | decomm | tests-scl1-linux | True | 0 | | talos-r3-fed64-026 | fedora12 | 64 | tests | scl1 | try | decomm | tests-scl1-linux | True | 0 | | talos-r3-fed64-027 | fedora12 | 64 | tests | scl1 | try | decomm | tests-scl1-linux | True | 0 | | talos-r3-fed64-028 | fedora12 | 64 | tests | scl1 | try | decomm | tests-scl1-linux | True | 0 | | talos-r3-fed64-029 | fedora12 | 64 | tests | scl1 | try | decomm | tests-scl1-linux | True | 0 | | talos-r3-fed64-001 | fedora12 | 64 | tests | scl1 | try | decomm | tests-scl1-linux | True | 0 | | talos-r3-fed64-014 | fedora12 | 64 | tests | scl1 | try | decomm | tests-scl1-linux | True | 0 | | talos-r3-leopard-001 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-003 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-004 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-013 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-005 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-006 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-008 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-009 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-011 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-012 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-014 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-015 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-016 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-017 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-018 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-019 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-020 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-021 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-022 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-023 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-024 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-025 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-026 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-027 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-028 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-029 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-030 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-031 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-032 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-033 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-034 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-035 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-036 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-037 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-038 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-039 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-040 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-041 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-042 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-043 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-044 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-045 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-046 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-047 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-048 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-049 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-050 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-051 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-052 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-053 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-054 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-055 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-056 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-057 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-058 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-059 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r4-snow-003 | darwin10 | 64 | tests | scl1 | try | dev/pp | tests-scl1-macosx | True | 0 | | talos-r4-snow-030 | darwin10 | 64 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r4-snow-041 | darwin10 | 64 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r4-snow-084 | darwin10 | 64 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-060 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-061 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-062 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-063 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-065 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r3-leopard-066 | darwin9 | 32 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r4-snow-171 | darwin10 | 64 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r4-snow-172 | darwin10 | 64 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | talos-r4-snow-173 | darwin10 | 64 | tests | scl1 | try | decomm | tests-scl1-macosx | True | 0 | | t-r3-w764-001 | win7 | 64 | tests | scl1 | try | decomm | tests-scl1-windows | True | 0 | | t-r3-w764-002 | win7 | 64 | tests | scl1 | try | decomm | tests-scl1-windows | True | 0 | | t-r3-w764-003 | win7 | 64 | tests | scl1 | try | decomm | tests-scl1-windows | True | 0 | | t-r3-w764-004 | win7 | 64 | tests | scl1 | try | decomm | tests-scl1-windows | True | 0 | | t-r3-w764-005 | win7 | 64 | tests | scl1 | try | decomm | tests-scl1-windows | True | 0 | | t-xp32-ix-099 | winxp | 32 | tests | scl1 | try | dev/pp | tests-scl1-windows | True | 0 | | tst-w64-ec2-001 | win64 | 64 | tests | aws-us-east-1 | try | prod | tests-scl1-windows | True | 1 | | tst-w64-ec2-002 | win64 | 64 | tests | aws-us-east-1 | try | prod | tests-scl1-windows | True | 1 | | tst-w64-ec2-003 | win64 | 64 | tests | aws-us-east-1 | try | prod | tests-scl1-windows | True | 1 | | tst-w64-ec2-004 | win64 | 64 | tests | aws-us-east-1 | try | prod | tests-scl1-windows | True | 1 | | tst-w64-ec2-005 | win64 | 64 | tests | aws-us-east-1 | try | prod | tests-scl1-windows | True | 1 | | tst-w64-ec2-006 | win64 | 64 | tests | aws-us-east-1 | try | prod | tests-scl1-windows | True | 1 | | tst-w64-ec2-007 | win64 | 64 | tests | aws-us-east-1 | try | prod | tests-scl1-windows | True | 1 | | tst-w64-ec2-008 | win64 | 64 | tests | aws-us-east-1 | try | prod | tests-scl1-windows | True | 1 | | tst-w64-ec2-009 | win64 | 64 | tests | aws-us-east-1 | try | prod | tests-scl1-windows | True | 1 | | tst-w64-ec2-010 | win64 | 64 | tests | aws-us-east-1 | try | prod | tests-scl1-windows | True | 1 | | talos-mtnlion-r5-087 | darwin12 | 64 | tests | scl3 | try | decomm | tests-scl3-macosx | True | 0 | +----------------------+----------+------+-------+---------------+-------+--------+--------------------+--------+---------+ 115 rows in set (0.02 sec)
We disabled bm70-80 (all of the windows/mac test masters in AWS except for bm69). I disabled them in production-masters.json, stopped the master processes, disabled nagios notifications.
These masters all look healthy now! c#16 being spun to a new bug for tracking/dealing with
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Blocks: 976281
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: