586443 - check up on / fix-up slaves after bug 552058

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[[e-mail] (:bhearsum)]

Some slaves weren't brought up to date and/or onto their new master after the last part of bug 552058 landed. In most cases this was because they were offline at the time. There is at least the following to do:
- Sync up/reboot the following slaves to get them onto mv-production-puppet:
-- linux-ix-ref, talos-r3-fed-{014,024,036}, talos-r3-fed64-{011,020,037,043,ref}, talos-r3-snow-ref, try-mac-slave{23,28}
- Grep /var/log/messages on mv-production-puppet/mpt-production-puppet for unauthenticated clients and other errors
- Compare accepted keys vs. site files for each masters
- Make sure all staging slaves are still syncing to staging-puppet.b.m.o.

Comment 1

[[e-mail] (:bhearsum)]

To the first point of the original comment: I surveyed all of the production slaves today. All of them were up to date and syncing to the correct master, except:
Waiting on a re-image:
moz2-darwin10-slave45,46, try-mac64-slave17

Offline:
linux-ix-ref
mv-moz2-linux-ix-slave01
talos-r3-fed-014,36,38
talos-r3-fed64-ref,037
talos-r3-snow-ref
linux-ref-platform
linux64-ref-platform

Some of these I can bring back up myself, some are waiting on IT. Either way, I'll be watching for these and dealing with them as they come back up.

Working on the other points, too.

Comment 2

[[e-mail] (:bhearsum)]

(In reply to comment #0)
> Some slaves weren't brought up to date and/or onto their new master after the
> last part of bug 552058 landed. In most cases this was because they were
> offline at the time. There is at least the following to do:
> - Sync up/reboot the following slaves to get them onto mv-production-puppet:
> -- linux-ix-ref, talos-r3-fed-{014,024,036},
> talos-r3-fed64-{011,020,037,043,ref}, talos-r3-snow-ref, try-mac-slave{23,28}
> - Grep /var/log/messages on mv-production-puppet/mpt-production-puppet for
> unauthenticated clients and other errors

> - Compare accepted keys vs. site files for each masters

Done on production-puppet and mv-production-puppet. There was some fixing to be done -- removing now-useless keys production-puppet and fixing up strange cases where a slave had no key on the server, yet was still syncing. Those are all ok now.

Comment 3

[[e-mail] (:bhearsum)]

> - Grep /var/log/messages on mv-production-puppet/mpt-production-puppet for
> unauthenticated clients and other errors

This is done, too.


Left to do:
sync up/reboot the following slaves:
- linux-ix-ref
- mv-moz2-linux-ix-slave01
- talos-r3-fed-0{14,36,38}
- talos-r3-fed64-{ref,037}
- talos-r3-snow-ref

And:
> - Make sure all staging slaves are still syncing to staging-puppet.b.m.o.

Comment 4

[[e-mail] (:bhearsum)]

(In reply to comment #3)
> > - Grep /var/log/messages on mv-production-puppet/mpt-production-puppet for
> > unauthenticated clients and other errors
> 
> This is done, too.
> 
> 
> Left to do:
> sync up/reboot the following slaves:
> - linux-ix-ref
> - mv-moz2-linux-ix-slave01
> - talos-r3-fed-0{14,36,38}
> - talos-r3-fed64-{ref,037}
> - talos-r3-snow-ref

...as well as moz2-darwin9-slave04 (can't access by ssh/vnc)
and talos-r3-fed-001 (currently loaned to neil deakin)

This is done:
> > - Make sure all staging slaves are still syncing to staging-puppet.b.m.o.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #4)
> ...as well as moz2-darwin9-slave04 (can't access by ssh/vnc)

This is dead, bug 583015. We should remove it from the puppet/buildbot/pony configs.

Comment 6

[[e-mail] (:bhearsum)]

Done:
linux-ix-ref
mv-moz2-linux-ix-slave01
talos-r3-snow-ref
talos-r3-fed64-ref

Still to do:
> > - talos-r3-fed-0{14,36,38}
> > - talos-r3-fed64-037

Comment 7

[[e-mail] (:bhearsum)]

Done
> > > - talos-r3-fed-0{14,36,38}
> > > - talos-r3-fed64-037



DONE!