826449 - [socorro stage] auto-update script running too often

Status: RESOLVED INCOMPLETE

(Infrastructure & Operations Graveyard :: WebOps: Other, task, P4)

Description

[Robert Helmer [:rhelmer]]

We noticed that the auto-update script on socorroadm (invoked by /etc/cron.d/auto-update) is running every 15m which is way too often.

There's actually a newer version of this which uses a better test to see if there is a new build or not (I assume jenkins is being wonky but haven't checked), this is what we use on dev:

/data/socorro/application/scripts/update-socorro.sh

Might be worth just switching to that.

Comment 1

[Jake Maul [:jakem]]

Just to be sure here... you want us to change /etc/cron.d/auto-update on socorroadm.stage.private.phx1 from:

*/15 * * * * root /root/bin/auto-update.sh

to something like:

*/5 * * * * root /data/socorro/application/scripts/update-socorro.sh


My understanding based on your description is this newer script can/should be run *more* often, because it will abort if there's nothing to do. Thus we should either run it instead (and more frequently), or we should run the old one less frequently (hourly maybe?)

Comment 2

[Robert Helmer [:rhelmer]]

(In reply to Jake Maul [:jakem] from comment #1)
> Just to be sure here... you want us to change /etc/cron.d/auto-update on
> socorroadm.stage.private.phx1 from:
> 
> */15 * * * * root /root/bin/auto-update.sh
> 
> to something like:
> 
> */5 * * * * root /data/socorro/application/scripts/update-socorro.sh
> 
> 
> My understanding based on your description is this newer script can/should
> be run *more* often, because it will abort if there's nothing to do. Thus we
> should either run it instead (and more frequently), or we should run the old
> one less frequently (hourly maybe?)

Right on all counts.

The old script *should* not run if there isn't a newer version available, but obviously that's not working correctly and it's probably not worth debugging since there's a newer version of the script checked into socorro anyway.

Comment 3

[Daniel Maher [:phrawzty]]

(In reply to Jake Maul [:jakem] from comment #1)
> */5 * * * * root /data/socorro/application/scripts/update-socorro.sh

This has been committed to Puppet and should be active within the next 60 minutes.

Comment 4

[Robert Helmer [:rhelmer]]

Cron started emailing this to me recently:

Syntax: socorro-install.sh <url-to-socorro_tar_gz>

Comment 5

[Daniel Maher [:phrawzty]]

I extracted the URL[1] that is hardcoded into /root/bin/auto-update.sh and tacked it on to the end of the cronjob.

Please confirm / verify that this is your desired solution.

[1] https://ci.mozilla.org/job/socorro-release/lastSuccessfulBuild/artifact/socorro.tar.gz

Comment 6

[Robert Helmer [:rhelmer]]

(In reply to Daniel Maher [:phrawzty] from comment #5)
> I extracted the URL[1] that is hardcoded into /root/bin/auto-update.sh and
> tacked it on to the end of the cronjob.
> 
> Please confirm / verify that this is your desired solution.
> 
> [1]
> https://ci.mozilla.org/job/socorro-release/lastSuccessfulBuild/artifact/
> socorro.tar.gz

Should be but isn't working :) Probably my fault, can you back this change out for now and I'll figure out the problem?

Comment 7

[Robert Helmer [:rhelmer]]

Oh actually sorry :( I just realized that this script isn't going to do the right thing on stage, it doesn't know how to ssh out to all the other servers and update those too :( 

So this definitely needs to be backed out.

Comment 8

[Brandon Burton [:solarce]]

We're rolling back, tracking work in bug 853102