Closed Bug 612288 Opened 14 years ago Closed 14 years ago

slave roundup

Categories

(Release Engineering :: General, defect, P2)

Product:
Release Engineering
Component:
General
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jhford, Assigned: jhford)

References

Details

(Whiteboard: [buildduty][buildslaves]) 

There are a bunch of slaves that aren't talking to their masters.  We need to figure out why and fix it

These slaves are not talking to their masters (according to missing-slaves.py) but are responding to pings on their build.mozilla.org address

Host linux-ix-slave08 is up on build.mozilla.org
Host linux-ix-slave16 is up on build.mozilla.org
Host linux-ix-slave17 is up on build.mozilla.org
Host linux-ix-slave31 is up on build.mozilla.org
Host linux-ix-slave32 is up on build.mozilla.org
Host linux-ix-slave34 is up on build.mozilla.org
Host linux-ix-slave35 is up on build.mozilla.org
Host moz2-darwin10-slave10 is up on build.mozilla.org
Host moz2-darwin10-slave23 is up on build.mozilla.org
Host moz2-darwin10-slave25 is up on build.mozilla.org
Host moz2-darwin9-slave10 is up on build.mozilla.org
Host moz2-linux-slave10 is up on build.mozilla.org
Host moz2-linux64-slave10 is up on build.mozilla.org
Host mv-moz2-linux-ix-slave17 is up on build.mozilla.org
Host mv-moz2-linux-ix-slave21 is up on build.mozilla.org
Host mw32-ix-slave12 is up on build.mozilla.org
Host t-r3-w764-003 is up on build.mozilla.org
Host t-r3-w764-004 is up on build.mozilla.org
Host t-r3-w764-005 is up on build.mozilla.org
Host t-r3-w764-006 is up on build.mozilla.org
Host t-r3-w764-007 is up on build.mozilla.org
Host t-r3-w764-008 is up on build.mozilla.org
Host t-r3-w764-009 is up on build.mozilla.org
Host t-r3-w764-011 is up on build.mozilla.org
Host t-r3-w764-012 is up on build.mozilla.org
Host t-r3-w764-013 is up on build.mozilla.org
Host t-r3-w764-014 is up on build.mozilla.org
Host t-r3-w764-015 is up on build.mozilla.org
Host t-r3-w764-016 is up on build.mozilla.org
Host t-r3-w764-017 is up on build.mozilla.org
Host t-r3-w764-019 is up on build.mozilla.org
Host t-r3-w764-021 is up on build.mozilla.org
Host t-r3-w764-022 is up on build.mozilla.org
Host t-r3-w764-024 is up on build.mozilla.org
Host t-r3-w764-025 is up on build.mozilla.org
Host t-r3-w764-026 is up on build.mozilla.org
Host t-r3-w764-029 is up on build.mozilla.org
Host t-r3-w764-030 is up on build.mozilla.org
Host t-r3-w764-031 is up on build.mozilla.org
Host t-r3-w764-032 is up on build.mozilla.org
Host t-r3-w764-033 is up on build.mozilla.org
Host t-r3-w764-034 is up on build.mozilla.org
Host t-r3-w764-035 is up on build.mozilla.org
Host t-r3-w764-036 is up on build.mozilla.org
Host t-r3-w764-037 is up on build.mozilla.org
Host t-r3-w764-043 is up on build.mozilla.org
Host t-r3-w764-044 is up on build.mozilla.org
Host t-r3-w764-045 is up on build.mozilla.org
Host t-r3-w764-046 is up on build.mozilla.org
Host t-r3-w764-047 is up on build.mozilla.org
Host t-r3-w764-048 is up on build.mozilla.org
Host t-r3-w764-049 is up on build.mozilla.org
Host t-r3-w764-050 is up on build.mozilla.org
Host talos-r3-fed-004 is up on build.mozilla.org
Host talos-r3-fed-014 is up on build.mozilla.org
Host talos-r3-fed-017 is up on build.mozilla.org
Host talos-r3-fed-023 is up on build.mozilla.org
Host talos-r3-fed-029 is up on build.mozilla.org
Host talos-r3-fed-030 is up on build.mozilla.org
Host talos-r3-fed-032 is up on build.mozilla.org
Host talos-r3-fed-036 is up on build.mozilla.org
Host talos-r3-fed-039 is up on build.mozilla.org
Host talos-r3-fed-043 is up on build.mozilla.org
Host talos-r3-fed-051 is up on build.mozilla.org
Host talos-r3-fed64-006 is up on build.mozilla.org
Host talos-r3-fed64-007 is up on build.mozilla.org
Host talos-r3-fed64-017 is up on build.mozilla.org
Host talos-r3-fed64-018 is up on build.mozilla.org
Host talos-r3-fed64-019 is up on build.mozilla.org
Host talos-r3-fed64-023 is up on build.mozilla.org
Host talos-r3-fed64-028 is up on build.mozilla.org
Host talos-r3-fed64-037 is up on build.mozilla.org
Host talos-r3-fed64-040 is up on build.mozilla.org
Host talos-r3-fed64-042 is up on build.mozilla.org
Host talos-r3-leopard-005 is up on build.mozilla.org
Host talos-r3-leopard-006 is up on build.mozilla.org
Host talos-r3-leopard-007 is up on build.mozilla.org
Host talos-r3-leopard-008 is up on build.mozilla.org
Host talos-r3-leopard-009 is up on build.mozilla.org
Host talos-r3-leopard-014 is up on build.mozilla.org
Host talos-r3-leopard-015 is up on build.mozilla.org
Host talos-r3-leopard-017 is up on build.mozilla.org
Host talos-r3-leopard-018 is up on build.mozilla.org
Host talos-r3-leopard-019 is up on build.mozilla.org
Host talos-r3-leopard-021 is up on build.mozilla.org
Host talos-r3-leopard-032 is up on build.mozilla.org
Host talos-r3-leopard-034 is up on build.mozilla.org
Host talos-r3-leopard-045 is up on build.mozilla.org
Host talos-r3-snow-011 is up on build.mozilla.org
Host talos-r3-snow-015 is up on build.mozilla.org
Host talos-r3-snow-017 is up on build.mozilla.org
Host talos-r3-snow-018 is up on build.mozilla.org
Host talos-r3-snow-019 is up on build.mozilla.org
Host talos-r3-snow-020 is up on build.mozilla.org
Host talos-r3-snow-021 is up on build.mozilla.org
Host talos-r3-snow-022 is up on build.mozilla.org
Host talos-r3-snow-025 is up on build.mozilla.org
Host talos-r3-snow-026 is up on build.mozilla.org
Host talos-r3-snow-032 is up on build.mozilla.org
Host talos-r3-snow-033 is up on build.mozilla.org
Host talos-r3-snow-038 is up on build.mozilla.org
Host talos-r3-snow-040 is up on build.mozilla.org
Host talos-r3-snow-041 is up on build.mozilla.org
Host talos-r3-snow-042 is up on build.mozilla.org
Host talos-r3-snow-049 is up on build.mozilla.org
Host talos-r3-snow-053 is up on build.mozilla.org
Host talos-r3-snow-055 is up on build.mozilla.org
Host talos-r3-w7-004 is up on build.mozilla.org
Host talos-r3-w7-006 is up on build.mozilla.org
Host talos-r3-w7-007 is up on build.mozilla.org
Host talos-r3-w7-008 is up on build.mozilla.org
Host talos-r3-w7-009 is up on build.mozilla.org
Host talos-r3-w7-012 is up on build.mozilla.org
Host talos-r3-w7-013 is up on build.mozilla.org
Host talos-r3-w7-017 is up on build.mozilla.org
Host talos-r3-w7-024 is up on build.mozilla.org
Host talos-r3-xp-014 is up on build.mozilla.org
Host talos-r3-xp-018 is up on build.mozilla.org
Host talos-r3-xp-047 is up on build.mozilla.org
Host try-mac-slave28 is up on build.mozilla.org
Host w32-ix-slave08 is up on build.mozilla.org
Host w32-ix-slave34 is up on build.mozilla.org
Host w32-ix-slave36 is up on build.mozilla.org
Host win32-slave10 is up on build.mozilla.org
Assignee: nobody → jhford
Severity: normal → major
OS: Mac OS X → All
Priority: -- → P2
Hardware: x86 → All
Whiteboard: [buildduty][buildslaves]
This would explain why our wait times for test pool are degrading recently; we're seeing worse wait times, even with a smaller number of jobs! :-(
See Also: → 611846
Targetted hit list for fedora

talos-r3-fed-012
talos-r3-fed-017
talos-r3-fed-018
talos-r3-fed-019
talos-r3-fed-022
talos-r3-fed-024
talos-r3-fed-029
talos-r3-fed-035
talos-r3-fed-036
talos-r3-fed-039
talos-r3-fed-040
I tackled these:

talos-r3-leopard-003
talos-r3-leopard-005
talos-r3-leopard-006
talos-r3-leopard-007
talos-r3-leopard-008
talos-r3-leopard-009
talos-r3-leopard-012
talos-r3-leopard-013
talos-r3-leopard-014
talos-r3-leopard-015
talos-r3-leopard-016
talos-r3-leopard-017
talos-r3-leopard-018
talos-r3-leopard-019
talos-r3-leopard-031
talos-r3-leopard-032
talos-r3-leopard-034
talos-r3-leopard-038
talos-r3-leopard-040
talos-r3-leopard-042
talos-r3-leopard-045

For those that were actually moved to SCL I had to:
- verify that they were set to talk with scl-production-puppet.build.scl1.mozilla.com in:
> /Library/LaunchDaemons/com.reductivelabs.puppet.plist
- sudo rm -rf /etc/puppet/ssl/

For slaves talos-r3-leopard-0{14,17,19,32,34} (which are actually still on MV) I did the following:
- change them back to talk with mv-production-puppet.build.mozilla.org
- sudo rm -rf /etc/puppet/ssl/
- I also had to change their port back to 9012 as they were pointing to 9011
The following are talking with scl-production-puppet.build since they seem to be at scl (talos-r3-snow-025.build.scl1.mozilla.com has address 10.12.50.78)
talos-r3-snow-011
talos-r3-snow-017
talos-r3-snow-019
talos-r3-snow-021
talos-r3-snow-022
talos-r3-snow-025
talos-r3-snow-026
talos-r3-snow-031
talos-r3-snow-032
talos-r3-snow-033
talos-r3-snow-034
talos-r3-snow-038
talos-r3-snow-040
talos-r3-snow-042
talos-r3-snow-049
talos-r3-snow-053
talos-r3-snow-055

These are still at MV:
talos-r3-snow-015
talos-r3-snow-016
talos-r3-snow-018
I think they just need buildbot.tac.off to be moved but I would rather clean up any remaining slaves tomorrow.
(In reply to comment #2)
> Targetted hit list for fedora
> 
> talos-r3-fed-012

down, re-added to reboot bug

> talos-r3-fed-017

had to puppetca --clean, online and doing jobs

> talos-r3-fed-018

down, added to reboot bug

> talos-r3-fed-019

down, added to reboot bug

> talos-r3-fed-022

down, added to reboot bug

> talos-r3-fed-024

down, added to reboot bug

> talos-r3-fed-029

was turned off but didn't make it to santa clara.  turned it back on but it is in a weird state, not running in production

> talos-r3-fed-035
> talos-r3-fed-036
> talos-r3-fed-039
> talos-r3-fed-040
(In reply to comment #3)
> I tackled these:
> 
> talos-r3-leopard-003
> talos-r3-leopard-005
> talos-r3-leopard-006
> talos-r3-leopard-007
> talos-r3-leopard-008
> talos-r3-leopard-009
> talos-r3-leopard-012
> talos-r3-leopard-013
> talos-r3-leopard-014
> talos-r3-leopard-015
> talos-r3-leopard-016
> talos-r3-leopard-017
> talos-r3-leopard-018
> talos-r3-leopard-019
> talos-r3-leopard-031
> talos-r3-leopard-032
> talos-r3-leopard-034
> talos-r3-leopard-038
> talos-r3-leopard-040
> talos-r3-leopard-042
> talos-r3-leopard-045

These slaves weren't connecting to buildbot.  The logs suggested clearing the puppet certificate on the master.  I cleared the cert with puppetca for all of these leopard slaves and am rebooting/verifying connection.  So far I have verified to be connected to their master:
> talos-r3-leopard-008
> talos-r3-leopard-009
Verified to be online and connected to buildbot:

> talos-r3-leopard-012
> talos-r3-leopard-014
> talos-r3-leopard-013 - had wrong port number
> talos-r3-leopard-015
> talos-r3-leopard-017
> talos-r3-leopard-018
> talos-r3-leopard-019
> talos-r3-leopard-031
> talos-r3-leopard-032
> talos-r3-leopard-038
> talos-r3-leopard-040
> talos-r3-leopard-042
> talos-r3-leopard-045

Down:
> talos-r3-leopard-016

Worked on and need to verify the following:
> talos-r3-leopard-003
> talos-r3-leopard-005
> talos-r3-leopard-006
> talos-r3-leopard-007
> talos-r3-leopard-034
Verified to be online and connected to buildbot:
> talos-r3-leopard-003
> talos-r3-leopard-005
> talos-r3-leopard-006

I haven't a clue what is wrong with:

> talos-r3-leopard-007
> talos-r3-leopard-034
Verified on buildbot:
> talos-r3-snow-022
> talos-r3-snow-042
> talos-r3-snow-049
> talos-r3-snow-033
> talos-r3-snow-034
> talos-r3-snow-040

Need to debug:
> talos-r3-snow-011
> talos-r3-snow-017
> talos-r3-snow-019
> talos-r3-snow-021
> talos-r3-snow-025
> talos-r3-snow-026
> talos-r3-snow-031
> talos-r3-snow-032
> talos-r3-snow-038
> talos-r3-snow-053
> talos-r3-snow-055
> talos-r3-snow-015 (mv)
> talos-r3-snow-016 (mv)
> talos-r3-snow-018 (mv)
Verified:
> talos-r3-snow-031
> talos-r3-leopard-007
> talos-r3-snow-019 puppet was pointing to build.m.o not build.scl1.m.c
 puppet was pointing to build.m.o not build.scl1.m.c
> talos-r3-snow-015 (mv)
> talos-r3-snow-016 (mv)
> talos-r3-snow-018 (mv)

Down:
> talos-r3-snow-011 offline since Oct 29, 2010
> talos-r3-snow-017 offline since Oct 16, 2010
> talos-r3-snow-021 offline since Oct 27, 2010
> talos-r3-snow-025 offline since Nov 02, 2010
> talos-r3-snow-026 offline since Oct 19, 2010
> talos-r3-snow-032 offline since Oct 21, 2010
> talos-r3-snow-038 offline since Oct 26, 2010
> talos-r3-snow-053 offline since Nov 04, 2010
> talos-r3-snow-055 offline since Nov 04, 2010
Comment 10 should read as below (there as an accidental paste in comment 10)

Verified:
> talos-r3-snow-031
> talos-r3-leopard-007
> talos-r3-snow-019 puppet was pointing to build.m.o not build.scl1.m.c
> talos-r3-snow-015 (mv)
> talos-r3-snow-016 (mv)
> talos-r3-snow-018 (mv)

Down:
> talos-r3-snow-011 offline since Oct 29, 2010
> talos-r3-snow-017 offline since Oct 16, 2010
> talos-r3-snow-021 offline since Oct 27, 2010
> talos-r3-snow-025 offline since Nov 02, 2010
> talos-r3-snow-026 offline since Oct 19, 2010
> talos-r3-snow-032 offline since Oct 21, 2010
> talos-r3-snow-038 offline since Oct 26, 2010
> talos-r3-snow-053 offline since Nov 04, 2010
> talos-r3-snow-055 offline since Nov 04, 2010
Verified:
> talos-r3-fed-029


Down:
> talos-r3-fed-012 not responding to pings
> talos-r3-fed-018 not responding to pings
> talos-r3-fed-019 not responding to pings
> talos-r3-fed-022 not responding to pings
> talos-r3-fed-024 not responding to pings
> talos-r3-fed-035 not responding to pings
> talos-r3-fed-036 os is broken
> talos-r3-fed-039 not syncing with puppet
> talos-r3-fed-040 not responding to pings
I'm not sure what the current state is here.  Would it be worth using a google spreadsheet for this sort of "roundup", to facilitate tracking the changing state and coordinating multiple engineers?  The final state could be attached to the ticket for historical posterity, if necessary.
(In reply to comment #8)
> Verified to be online and connected to buildbot:
> > talos-r3-leopard-003

This box is currently not running buildbot - (buildbot.tac.off).  Is there a reason?
(In reply to comment #14)
> I'm not sure what the current state is here.  Would it be worth using a google
> spreadsheet for this sort of "roundup", to facilitate tracking the changing
> state and coordinating multiple engineers?  The final state could be attached
> to the ticket for historical posterity, if necessary.

Good call. Bear, Rail, and I had been using an etherpad this week, but a spreadsheet probably makes more sense. I'll get that setup today and move our etherpad data into it.
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.