1158825 - increase process timeout for cron monitoring jobs and add in file age checks on log files

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[Amy Rich [:arr] [:arich]]

After discussion at the architecture meeting and in irc, it was decided that it's better to try to let the cron jobs aws_watch_pending and aws_stop_idle run to completion rather than killing them while they're still actively processing data. Because both of these save us money, we still want to kill them off if they've been running too long, though.

Currently both cron jobs have secondary jobs that monitor them and kill off the original process if it reaches a defined timeout. At the moment that's 1200 seconds for both. In order to give the jobs time to finish, we'll bump that up to 3600 (60 minutes vs 20) to see if we're still killing the job off prematurely.

In addition we're adding a file age check to both log files to make sure that data is still being written to them.

Comment 1

[Amy Rich [:arr] [:arich]]

Attachment: file-age-checks.diff

Nagios additions for the file age checks on both log files.

Comment 2

[Amy Rich [:arr] [:arich]]

Attachment: aws-manager-cron-timeouts.diff

Bumped the timeouts to 3600 and modified the termination message of the monitor job to include more searchable data for log file analysis.

Comment 3

[Rail Aliiev [:rail]]

Comment on attachment 8598033 [details] [diff] [review]
aws-manager-cron-timeouts.diff

Let's try. watch pending is serial, so if things are slow it may slower respond to pending jobs.

Comment 4

[Amy Rich [:arr] [:arich]]

remote:   https://hg.mozilla.org/build/puppet/rev/3d682538fd9d

(and currently in testing as aws-manager2 is pinned to my env)

Comment 5

[Amy Rich [:arr] [:arich]]

Checked in and host unpinned.

Comment 6

[Amy Rich [:arr] [:arich]]

It turns out that aws_stop_idle.py hangs at least once a day and doesn't keep processing hosts. This means we actually want/need to kill it off more frequently. After discussion with :rail, I'm going to change the time for stop_idle back to 1200s to address this and bug 1160314 to have debug the hanging issues in the script and get them fixed.

change landed in https://hg.mozilla.org/build/puppet/rev/4a012a269c33
needs to be merged to production.