715780 - fix locking on cron submitter

Status: RESOLVED FIXED

(Socorro :: Backend, task)

Description

[Robert Helmer [:rhelmer]]

We have a script on prod which submits crashes to staging in /data/bin/submit_dumps_to_staging.sh (also on stage, which submits to dev)

This script creates a lock in /tmp which frequently gets stuck and needs to be removed manually.

We should figure out why this is happening, and just switch it to use flock or something. Also it should use /var/lock not /tmp

Comment 1

[Jason Thomas [:jason]]

Attachment: submit dumps

New submit dumps script attached. rhelmer could you please review?

Comment 2

[Jason Thomas [:jason]]

Attachment: submit dumps

Uploaded an old version, here is the latest.

Comment 3

[Robert Helmer [:rhelmer]]

Attachment: submit dumps v2

Hey this looks way better! The "if $(flock ...)" bit doesn't seem to work for me on stage - why not do it like this?

I also cleaned up the psql line some, I don't think both of those (very complex) sed expressions are necessary just to remove the leading space, replaced it with a much smaller "tr -d"

Comment 4

[Robert Helmer [:rhelmer]]

(In reply to Robert Helmer [:rhelmer] from comment #3)
> Created attachment 601841 [details]
>
> I also cleaned up the psql line some, I don't think both of those (very
> complex) sed expressions are necessary just to remove the leading space,
> replaced it with a much smaller "tr -d"

BTW I realize it was like this in the original script, not your fault :)

I also find the fact that this uses the jobs table really odd, but we can look at that later.

Comment 5

[Jason Thomas [:jason]]

(In reply to Robert Helmer [:rhelmer] from comment #3)
> Created attachment 601841 [details]
> submit dumps v2
> 
> Hey this looks way better! The "if $(flock ...)" bit doesn't seem to work
> for me on stage - why not do it like this?
> 

I think we need to capture the exit code of flock -n, without it the any additional instances started continues to execute even if there is already an exclusive lock held by another instance.

When tested with if statement, when the script is running and another instance is executed the second instance fails immediately. Without it any additional instances started will execute.

Comment 6

[Jason Thomas [:jason]]

ping?

Comment 7

[Jason Thomas [:jason]]

Attachment: submit dumps v3

updated to include if statement.

Comment 8

[Robert Helmer [:rhelmer]]

Ready for inclusion to Socorro - https://github.com/mozilla/socorro/pull/437

Comment 9

[[github robot]]

Commit pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/558f7f14b4a1d911a6a4713b276fcf7fa55b29b4
Merge pull request #437 from rhelmer/bug715780-cron-submitter

bug 605429 - bring cron test crash submitter into socorro, with locking ...

Comment 10

[Robert Helmer [:rhelmer]]

This will go out next Wednesday, and will be installed to:

/data/socorro/application/scripts/crons/cron_submitter.sh

Logs will be written to /var/log/socorro/cron_submitter-${CRASH_REPORTS_HOST}.log