864823 - Chief for Socorro

Status: RESOLVED FIXED

(Socorro :: Infra, task, P3)

Description

[Lonnen :lonnen]

We'd like to set up Chief to handle the main Socorro push. For non-trivial features we'll still need to involve IT, but most of the time we shouldn't have to interrupt ops.

Comment 1

[Lonnen :lonnen]

The setup is a little different than your standard playdoh project, but very similar to the socorro-crashstats chief setup that JakeM set up in our new cluster in Q1.

Comment 2

[Fred Wenzel [:wenzel]]

Is this a webdev bug or an ops bug?

Comment 3

[Lonnen :lonnen]

reassigning to web operations.

Comment 4

[Jake Maul [:jakem]]

We're not too comfortable with putting this in Chief just yet, because it seems to frequently fail. Socorro 44 had trouble with 3 nodes... it feels like the update mechanism isn't robust enough yet. It's not to the point where you can push a button and reasonably expect that it will work every time.

I'm also afraid it might take too long- Chief has a tendency to time out waiting for response. Would take some finagling.

What about shell access (sudo even, if necessary) to the relevant admin node, such that you could run the update script yourself? Would that be an acceptable alternative?

Comment 5

[Robert Helmer [:rhelmer]]

(In reply to Jake Maul [:jakem] from comment #4)
> We're not too comfortable with putting this in Chief just yet, because it
> seems to frequently fail. Socorro 44 had trouble with 3 nodes... it feels
> like the update mechanism isn't robust enough yet. It's not to the point
> where you can push a button and reasonably expect that it will work every
> time.
> 
> I'm also afraid it might take too long- Chief has a tendency to time out
> waiting for response. Would take some finagling.
> 
> What about shell access (sudo even, if necessary) to the relevant admin
> node, such that you could run the update script yourself? Would that be an
> acceptable alternative?

lonnen and I were just looking at this, and I suspect this is what's going on:

* apps are started with daemonize, which handles the lock https://github.com/mozilla/socorro/blob/master/scripts/init.d/socorro-processor#L33
* killproc gives a 15s window for the app to shut down after a normal kill is sent, then sends a -9 (see /etc/init.d/functions)

I suspect at this point, daemonize doesn't clear the lockfile properly... I'll test this and put in a check if that's the case.

Are there problems other than with these init scripts?

Comment 6

[Robert Helmer [:rhelmer]]

(In reply to Robert Helmer [:rhelmer] from comment #5)
> (In reply to Jake Maul [:jakem] from comment #4)
> > We're not too comfortable with putting this in Chief just yet, because it
> > seems to frequently fail. Socorro 44 had trouble with 3 nodes... it feels
> > like the update mechanism isn't robust enough yet. It's not to the point
> > where you can push a button and reasonably expect that it will work every
> > time.
> > 
> > I'm also afraid it might take too long- Chief has a tendency to time out
> > waiting for response. Would take some finagling.
> > 
> > What about shell access (sudo even, if necessary) to the relevant admin
> > node, such that you could run the update script yourself? Would that be an
> > acceptable alternative?
> 
> lonnen and I were just looking at this, and I suspect this is what's going
> on:
> 
> * apps are started with daemonize, which handles the lock
> https://github.com/mozilla/socorro/blob/master/scripts/init.d/socorro-
> processor#L33
> * killproc gives a 15s window for the app to shut down after a normal kill
> is sent, then sends a -9 (see /etc/init.d/functions)
> 
> I suspect at this point, daemonize doesn't clear the lockfile properly...
> I'll test this and put in a check if that's the case.

OK I have just tested this and it does clear the lockfile properly (checked the pidfile to make sure it's not actually there anymore).

The only option I see is that perhaps the app is in uninterruptible sleep, so it doesn't respond to the kill -9 before we turn around and try to start the app.

Our timeout is 15s right now, we may need to either extend that or change the design of the apps.

Comment 7

[Robert Helmer [:rhelmer]]

Filed bug 868512 to track down fixing this on the Socorro side.

Comment 8

[Lonnen :lonnen]

@JakeM -- I think we'd rather have chief, and I hope that after 868512 is resolved we can set it up. As an interim solution, though, giving rhelmer and myself the means to push code ourselves would be appreciated.

Comment 9

[Lonnen :lonnen]

What are the next steps on this?

Comment 10

[Lonnen :lonnen]

ping. whats up with this?

Comment 11

[Laura Thomson :laura]

ping?

Comment 12

[Lonnen :lonnen]

This is highly desirable as part of a q3 goal to sync the release processes of socorro + the django frontend. How can we move forward with this?

Comment 13

[Brandon Burton [:solarce]]

:lonnen and I discussed this on Vidyo and we've settled on a phase 1 for Chief and Socorro.

Phase 1:

* Install Chief on socorroadm.private.phx1 via Puppet
* Setup Chief for socorro staging
 * Initial commander compatible python script will just call the existing shell scripts
 * cron job will start calling Chief instead of the script directly
* Chief + cronjob takes over the staging auto-deploy

Comment 14

[Brandon Burton [:solarce]]

bburton@macbookair-00886541dab0 [06:16:31] [~/code/mozilla/sysadmins/puppet/trunk]
-> % svn ci -m "setting up chief for socorro cluster, bug 864823" modules/webapp/manifests/chief/socorro.pp
Adding         modules/webapp/manifests/chief/socorro.pp
Transmitting file data .
Committed revision 71035.

Comment 15

[Brandon Burton [:solarce]]

bburton@macbookair-00886541dab0 [06:17:23] [~/code/mozilla/sysadmins/puppet/trunk]
-> % svn ci -m "setting up chief for socorro cluster, bug 864823" manifests/nodes/socorro.pp
Sending        manifests/nodes/socorro.pp
Transmitting file data .
Committed revision 71036.

Comment 16

[:Atoll]

FYI, recent socorro stage commits from sysadmins r71551..71558 seem to have broken puppet on certain stage hosts:

socorro-collector1.stage.webapp.phx1
socorro-mware1.stage.webapp.phx1
socorro-web1.stage.webapp.phx1
etc.

The error is a missing dependency for /var/log/xyz, where xyz varies based on the host. For instance:

Failed to apply catalog: Could not find dependency Package[httpd] for File[/var/log/httpd/crash-reports.mozilla.com]

Comment 17

[Brandon Burton [:solarce]]

(In reply to Richard Soderberg [:atoll] from comment #16)
> FYI, recent socorro stage commits from sysadmins r71551..71558 seem to have
> broken puppet on certain stage hosts:
> 
> socorro-collector1.stage.webapp.phx1
> socorro-mware1.stage.webapp.phx1
> socorro-web1.stage.webapp.phx1
> etc.
> 
> The error is a missing dependency for /var/log/xyz, where xyz varies based
> on the host. For instance:
> 
> Failed to apply catalog: Could not find dependency Package[httpd] for
> File[/var/log/httpd/crash-reports.mozilla.com]

Thanks, I got lost in a maze of edits trying to track down a conflicting httpd service resource that ended up being in ganglia::server

I guess I failed to completely clean up my mess

I shall fix it shortly

Comment 18

[Lonnen :lonnen]

Kind of got it, but then we moved out.