777544 - [socorro prod] crontabber needs to be reset

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Robert Helmer [:rhelmer]]

crontabber seems to be trying to run SignaturesCronApp on each run. Let's go ahead and reset the state file (and the crontabber_state table in the DB) and see if that helps.

Also, I see that crontabber doesn't do any locking so we get multiple instances of it if it runs long - this is going to cause problems. I will file a separate bug for that.

Comment 1

[Robert Helmer [:rhelmer]]

So it looks like crontabber will attempt to run everything if there's an empty crontabber.json state file, rather than waiting until the next time things should run (which is ok behavior I suppose I just wasn't expecting it :) )

I think the combination of 
1) old crontabber.json state file (now fixed) and 
2) no locking for crontabber (bug 777549, PR pending)

were combining to cause the problem here... still running tests w/ solarce

Comment 2

[Robert Helmer [:rhelmer]]

OK to correct this it looks like we'll need to modify ~socorro/persistent/crontabbers.json so that crontabber doesn't wait 24h to retry, and runs at 10 AM (UTC) instead.

Comment 3

[Robert Helmer [:rhelmer]]

Attachment: current crontabbers.json on prod

Comment 4

[Peter Bengtsson [:peterbe]]

Attachment: Doctored so that all the jobs run at 02:00 Pacific time tomorrow Thursday 26 July

Comment 5

[Robert Helmer [:rhelmer]]

(In reply to Peter Bengtsson [:peterbe] from comment #4)
> Created attachment 645983 [details]
> Doctored so that all the jobs run at 02:00 Pacific time tomorrow Thursday 26
> July

jakem can you please put that crontabbers.json in ~socorro/persistent/crontabbers.json ?

Comment 6

[Jake Maul [:jakem]]

Done. Additional work in bug 777573.

Comment 7

[Robert Helmer [:rhelmer]]

We found another bootstrap issue that we missed before, so we need to make another adjustment, sorry for the noise.

Comment 8

[Peter Bengtsson [:peterbe]]

Attachment: last_success all equal to the last_run

Comment 9

[Jake Maul [:jakem]]

Done.