749252 - load issues with scl-production-puppet server

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Amy Rich [:arr] [:arich]]

We've now hit load issues (and resulting puppet failures) two days in a row on scl-production-puppet because too many hosts are trying to hit the puppet server at once.

Yesterday we used iptables to limit the hosts that could connect and found that this impacted a bunch of machines in sjc1 as well as those in scl1.  sjc1 has it's own puppet master and shouldn't be talking to the one in scl1.

There was also an issue discovered with the new linux hosts in scl3, also talking to the puppet master in scl1.  I'm not sure if that issue is resolved yet or not, but all of the hosts in scl3 should be checked and verified that they are talking to the scl3 puppet server, not the one in scl1.

Additionally this morning there was a bad configuration push by releng that made all of the linux talos machines fail to puppetize (my understanding is that the necessary files form the pus did not get copied to all servers).

Since we are not allowed to monitor the puppet process on the talos machines (and it doesn't appear that anyone in releng reads the error mail generated by puppet on the clients), these all just failed silently.

People didn't notice until the non-talos machines started to fail.  This was due to all of the talos machines hitting the puppet master at once (because they rely on puppet succeeding and reboot and try again otherwise) and driving up the load so that more hosts started to fail puppetization.

In order to bring things under control, I upped the puppet master from 1 to 3 CPUs (at which point it was still pegged as all the talos machines tried to connect) and 4G of RAM (from 2G since it started swapping itself to death).  The patch was also backed out.

I have also documented the problem and possible server-side corrective measures at:
https://mana.mozilla.org/wiki/display/SpecOps/Buildbot

So, what's left to be done here for releng:

* verify that all of the hosts in scl3 are connecting to the correct puppet master.  Checking what's in the startup script may not be sufficient as we saw the r5s first connect to scl3 and then be redirected to the staging puppet server based on their manifests back when we brought them online.  I don't bother to say check sjc1 hosts since they will all be decommissioned in the next three weeks.

* add/refresh any documentation on how to check that a host is using the proper puppet server and add/refresh troubleshooting documentation on how to diagnose a failing puppet connection on the client end

Comment 1

[Chris Cooper [:coop] (he/him)]

(In reply to Amy Rich [:arich] [:arr] from comment #0) 
> * verify that all of the hosts in scl3 are connecting to the correct puppet
> master.  Checking what's in the startup script may not be sufficient as we
> saw the r5s first connect to scl3 and then be redirected to the staging
> puppet server based on their manifests back when we brought them online.  I
> don't bother to say check sjc1 hosts since they will all be decommissioned
> in the next three weeks.

I confess that I don't know enough about how this is actually setup, but isn't this expected? Didn't we at some point in time have a system where (at least new) slaves would try to connect to a generic production-puppet master and then get shunted to the correct master for them? Or is that not what's happening here?
 
> * add/refresh any documentation on how to check that a host is using the
> proper puppet server and add/refresh troubleshooting documentation on how to
> diagnose a failing puppet connection on the client end

I don't know if such documentation exists yet.

Comment 2

[hwine]

This seems to fit well with my scl3 new master & slave bringup and associated wiki doc update, so grabbing and adding.

Coop: my understanding from bhearsum is that the puppet master redirect mechanism no longer works, and we now set the puppet-server-to-use as part of the bringup process.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

(In reply to Hal Wine [:hwine] from comment #2)
> Coop: my understanding from bhearsum is that the puppet master redirect
> mechanism no longer works, and we now set the puppet-server-to-use as part
> of the bringup process.

Agreed.  I'm not entirely convinced that there are sjc1 hosts hitting the scl1 master.  A grep for '10\.2\.' and 'sjc1' in the bouncer error logs and /var/log/messages doesn't show anything.  Ditto for 10.26 and scl3.

A quick check of a random r5 builder mini in scl3 shows the correct:

bld-lion-r5-066:~ root# cat /Library/LaunchDaemons/com.reductivelabs.puppet.plist 
..
                <string>scl3-production-puppet.srv.releng.scl3.mozilla.com</string>

"Redirecting" to another master consists of rewriting the relevant configuration files (/etc/sysconfig/puppet, com.reductivelabs.puppet.plist, etc.) and rebooting the host, so the config files actually *are* authoritative there.

Anyway, Hal, please do have a look through to make sure things are in order beyond my quick peek here.

Comment 4

[hwine]

completed quite a while back, afaik. Reopen if any current issues