[socorro prod] crontabber needs to be reset

RESOLVED FIXED

Status

mozilla.org Graveyard
Server Operations
RESOLVED FIXED
6 years ago
3 years ago

People

(Reporter: rhelmer, Assigned: jakem)

Tracking

Details

Attachments

(1 attachment, 2 obsolete attachments)

(Reporter)

Description

6 years ago
crontabber seems to be trying to run SignaturesCronApp on each run. Let's go ahead and reset the state file (and the crontabber_state table in the DB) and see if that helps.

Also, I see that crontabber doesn't do any locking so we get multiple instances of it if it runs long - this is going to cause problems. I will file a separate bug for that.
(Reporter)

Comment 1

6 years ago
So it looks like crontabber will attempt to run everything if there's an empty crontabber.json state file, rather than waiting until the next time things should run (which is ok behavior I suppose I just wasn't expecting it :) )

I think the combination of 
1) old crontabber.json state file (now fixed) and 
2) no locking for crontabber (bug 777549, PR pending)

were combining to cause the problem here... still running tests w/ solarce
(Reporter)

Updated

6 years ago
Assignee: server-ops → bburton
(Reporter)

Comment 2

6 years ago
OK to correct this it looks like we'll need to modify ~socorro/persistent/crontabbers.json so that crontabber doesn't wait 24h to retry, and runs at 10 AM (UTC) instead.
Assignee: bburton → nmaul
(Reporter)

Comment 3

6 years ago
Created attachment 645981 [details]
current crontabbers.json on prod
Created attachment 645983 [details]
Doctored so that all the jobs run at 02:00 Pacific time tomorrow Thursday 26 July
Attachment #645981 - Attachment is obsolete: true
(Reporter)

Comment 5

6 years ago
(In reply to Peter Bengtsson [:peterbe] from comment #4)
> Created attachment 645983 [details]
> Doctored so that all the jobs run at 02:00 Pacific time tomorrow Thursday 26
> July

jakem can you please put that crontabbers.json in ~socorro/persistent/crontabbers.json ?
(Assignee)

Comment 6

6 years ago
Done. Additional work in bug 777573.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
(Reporter)

Updated

6 years ago
Blocks: 777573
(Reporter)

Comment 7

6 years ago
We found another bootstrap issue that we missed before, so we need to make another adjustment, sorry for the noise.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Created attachment 646731 [details]
last_success all equal to the last_run
Attachment #645983 - Attachment is obsolete: true
Attachment #646731 - Flags: review?(rhelmer)
(Assignee)

Comment 9

6 years ago
Done.
Status: REOPENED → RESOLVED
Last Resolved: 6 years ago6 years ago
Resolution: --- → FIXED
(Reporter)

Updated

6 years ago
Attachment #646731 - Flags: review?(rhelmer) → review+
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.