828657 - Perform a reboot of web3 in stage to check that daemontools starts things correctly

Status: RESOLVED FIXED

(Cloud Services :: Operations: Miscellaneous, task)

Description

[John Morrison [:jrgm]]

Perform a reboot of web3 in stage to check that daemontools starts things correctly

There's evidence it starts browserid-webhead-loadtest twice.

Comment 1

[Gene Wood [:gene]]

This machine just died again. I suspect a hardware problem. I'm going to have the hardware diagnosed and rebuild the machine.

Comment 2

[John Morrison [:jrgm]]

Well, I got what I needed anyways from this reboot.

1. At boot time, the browserid daemontools services are not started.

2. They get started by a puppet run some time after boot:

> 11:08:38 web3 puppet-agent[6823]: (/Stage[main]/Daemontools::Base/Exec[start-daemontools]/returns) executed successfully

3. When they are started, they are all started so both browserid-webhead and
   browserid-webhead-loadtest race to win the LISTEN. The loser continues to
   run but has nothing to do.

4. About a half-hour later, puppet appears to shut down browserid-webhead

> Jan 16 11:39:10 web3 puppet-agent[6823]: (/Stage[main]/Browserid::Webhead/Exec[svc -d /service/browserid-webhead]/returns) executed successfully

Now, if browserid-webhead was the one that won the race at step 3, that means
that we're left with browserid-webhead-loadtest running, but not listening on
its assigned port. That's exactly what must have happened the previous
time. This time, -loadtest won the race, so things appear to be fine.

So, the way puppet and daemontools are set up seems wrong (or maybe just
daemontools).

1. Daemontools is supposed to start its services at init time. Not sure why it
   is not doing so.

2. /var/services/browserid-webhead should have a `down` file in place. (We
   never want that one to run in stage).

But I'll put the above in a followup bug.