608024 - create RHEL init scripts for socorro

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Robert Helmer [:rhelmer]]

It would make both automated and manual Socorro deployment easier if all the daemon(ish) Socorro processes had RHEL init scripts:

* monitor (scripts/startMonitor.py)
* middleware (scripts/webservices.py)
* processor (scripts/startProcessor.py)

Collector and webapp-php should be handled by their respective Apache httpd restart.

I *think* everything else in scripts/ is either one-time or run by cron.

Comment 1

[Robert Helmer [:rhelmer]]

Attachment: proposed init script for socorro

Here is a working init script for the 1.7 "monitor" service. This is basically a template that could be used to wrap any socorro start scripts; all needed configuration (python, pythonpath, socorro dir, script name, etc.) is at the top of the file.

Note that this requires the "daemonize" package to be installed (http://bmc.github.com/daemonize/), which is in the standard repo for RHEL5. Normally one would use the "daemon" function here, but fortunately this approach is compatible with what the daemon function does, so we can use the rest of the standard init functions as you'd expect.

daemonize handles pid/lock files, logging, and all the usual things you want with daemonizing (closing file descriptors, forking, etc.), so it's pretty handy for wrapping scripts that were not written to be daemons.

If we want to reduce the number of layers here, we could make the init scripts themselves python daemons and completely replace the script/start*.py scripts, but I opted for the wrapper approach since it's 0 code change on the socorro side, so is faster to implement and feels less risky pre-PHX.

Let me know if this looks acceptable and I'll get the others ready and generate a patch. I think this makes sense to land in the googlecode repo.

Comment 2

[Robert Helmer [:rhelmer]]

Comment on attachment 486800 [details]
proposed init script for socorro

Note that I have not addressed the idea of using the init scripts to pass configuration (while leaving the config files with sane defaults).

I suggest doing overrides using environment variables, we could have a common set (which would contain appdir, python, etc.) in /etc/default/socorro and individual overrides could be done in e.g. /etc/socorro-monitor.conf

For simplicity, I'd just have the configs files be "source"able by the init script such as:

/etc/socorro-monitor.conf:
stderrErrorLoggingLevel=10

These would be the only config files we'd manage through puppet, and they would only contain overrides to the scripts/config/*.py-dist files. Part of the install process would need to be copying the *.py-dist files to *.py, but they would never be directly modified by puppet beyond that.

Comment 3

[Justin Dow [:jabba]]

I think the idea in comment 2 is perfect and the init script looks good to me (I've read it over, but haven't tried running it yet). I think this is a good time to define some things so that we have standards going forward:

the svn checkout directory should be /data/socorro for all servers and services, so the appdir in the init script should use that directory.

Lets also make /etc/socorro a directory managed by puppet and the init script should source files in that directory, if they exist. This is where we'll override defaults, so instead of /etc/socorro-monitor.conf, let's do /etc/socorro/monitor.conf.

Also, I've never done the review thing with bugzilla, what's the full process there?

Comment 4

[Justin Dow [:jabba]]

Also, yes, the init script should live in the svn checkout, perhaps under scripts/rhel-init.sh or something like that and then one could make a symlink from /etc/init.d/socorro-monitor to /data/socorro/scripts/rhel-init.sh. If proper variables are used, this should make it possible to have one init script for all the various daemons and should also be valid for other users of socorro outside of Mozilla.

Does that sound reasonable?

Comment 5

[Robert Helmer [:rhelmer]]

Comment on attachment 486800 [details]
proposed init script for socorro

Sorry really meant feedback (which you gave), it's not really ready for review until it's a patch for checkin.

Comment 6

[Robert Helmer [:rhelmer]]

(In reply to comment #4)
> Also, yes, the init script should live in the svn checkout, perhaps under
> scripts/rhel-init.sh or something like that and then one could make a symlink
> from /etc/init.d/socorro-monitor to /data/socorro/scripts/rhel-init.sh. If
> proper variables are used, this should make it possible to have one init script
> for all the various daemons and should also be valid for other users of socorro
> outside of Mozilla.
> 
> Does that sound reasonable?

Hmm yeah we could do that; I was imagining having different scripts but I don't like the duplication, since they'd basically be identical copies except for the comments and a few variables at the top.

How would it know the difference, by looking at how it was called (arg $0) ?

I think my criteria would be the principle of least surprise - if you don't think it would be too disorienting for other sysadmins (in the middle of the night while the system is on fire), it's fine with me.

Comment 7

[Robert Helmer [:rhelmer]]

Attachment: init scripts for socorro

Init scripts for processor, middleware, monitor.

Config files in /etc/socorro/{processor,middleware,monitor}.conf will be sourced, if they exist. chkconfig and /sbin/service should work, restarts should be safe due to pidfile usage, and lockfiles prevent multiple instances of services from running. All services are run as "socorro" user. Log files /var/log/socorro-{processor,middleware,monitor}.log must be writable by the "socorro" user.

I've tested this in development and it seems to work as expected. Note that the config files are simply shell include files, so variables must be exported. Here is my local monitor.conf:

export databaseUserName=test
export databaseName=socorro
export databaseHost=postgres123
export hbaseHost=hadoop123

jabba and I discussed in irc, seeing as how this is only 3 scripts right now I just did copies (instead of having one rhel-init.sh that looks at it's argv[0]).

We should probably also have a common config file, since all three scripts want the same database config at least in most configurations. I could make these also source a /etc/socorro/common.conf or something along those lines, suggestions welcome.

Comment 8

[Justin Dow [:jabba]]

Yes to the common.conf. I think in general, no local changes should need to happen to anything in the /data/socorro directory (svn checkout), so any configuration that happens should happen in /etc/socorro/* . Also, I thought that the middleware layer runs under apache as a mod_wsgi application. Is this not the case?

Comment 9

[Robert Helmer [:rhelmer]]

(In reply to comment #8)
> Yes to the common.conf. I think in general, no local changes should need to
> happen to anything in the /data/socorro directory (svn checkout), so any
> configuration that happens should happen in /etc/socorro/* 

Agreed, I'll attach a new patch with common.conf

> Also, I thought
> that the middleware layer runs under apache as a mod_wsgi application. Is this
> not the case?

That's a good point, I don't know :) If it's run as a mod_wsgi app in staging/prod, then I'll go ahead and nix the script in the next patch.

Comment 10

[Robert Helmer [:rhelmer]]

Attachment: init scripts for socorro

Same as attachment 487659 [details] [diff] [review] except:

* remove middleware script; only processor and monitor now
* first source /etc/socorro/common.conf, then individual .conf

I did some basic tests and the override seems to work as expected.

Comment 11

[Justin Dow [:jabba]]

Looking good. We should change the $prefix from /data/www/crash-stats.mozilla.com to /data/socorro . I think we should standardize all components to use this new directory going forward. Otherwise it looks great and I'll be ready to test after we do the 1.7.5 release. After that is out the door, then we should blow away stage and rebuild using the puppet configs and these scripts, such that the next release will include them.

Comment 12

[Robert Helmer [:rhelmer]]

Attachment: [as landed] init scripts for socorro

Patch as-landed. Same as attachment 487688 [details] [diff] [review] except for prefix=/data/socorro:

Adding         scripts/init.d
Adding         scripts/init.d/socorro-monitor
Adding         scripts/init.d/socorro-processor
Transmitting file data ..
Committed revision 2684.

Comment 13

[Robert Helmer [:rhelmer]]

Found a typo in testing; wrong config filename for processor init script:

Sending        scripts/init.d/socorro-processor
Transmitting file data .
Committed revision 2685.

Comment 14

[Robert Helmer [:rhelmer]]

One thing that Jabba pointed out that I didn't address with this patch is that we may need to have more than one processor running per machine (we do this now in production, hopefully will not have to in PHX).

I'll have a patch for this momentarily.

Comment 15

[Robert Helmer [:rhelmer]]

Attachment: use arg 0 to determine lock/log/pid file name

This allows one to do the following:

ln -s (...)/socorro-processor /etc/init.d/socorro-processor1
ln -s (...)/socorro-processor /etc/init.d/socorro-processor2

These will be restartable separately, and create separate log files.

Technically we could merge both socorro-monitor and socorro-processor into one script using this technique, but I think having them both there (with separate comments and chkconfig args) would be less surprising for other Socorro users.

Comment 16

[Robert Helmer [:rhelmer]]

Comment on attachment 491309 [details] [diff] [review]
use arg 0 to determine lock/log/pid file name

Sending        scripts/init.d/socorro-monitor
Sending        scripts/init.d/socorro-processor
Transmitting file data ..
Committed revision 2783.

Comment 17

[Robert Helmer [:rhelmer]]

Attachment: init script for newCrashMover service

The socorro-{processor,monitor} init scripts seem to be working fine.

Add one more, for the newCrashMover service introduced for bug 608734.

Comment 18

[Robert Helmer [:rhelmer]]

Comment on attachment 493739 [details] [diff] [review]
init script for newCrashMover service

Adding         socorro-crashmover
Transmitting file data .
Committed revision 2792.

Comment 19

[Robert Helmer [:rhelmer]]

Set executable:

Sending        socorro-crashmover

Committed revision 2793.