887945 - release-runner.py should use a lockfile to make sure only one instance is ever running

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Comment 1

[Rail Aliiev [:rail]]

Attachment: use twisted.python.lockfile

I haven't tested this yet.

Comment 2

[Rail Aliiev [:rail]]

Attachment: use twisted.python.lockfile v2

When we switch to puppet managed wrapper (either shell or python) we should explicitly pass --lockfile to not pollute tools checkout with untracked files and keep the lock file in /builds/releaserunner.

Comment 3

[[e-mail] (:bhearsum)]

Comment on attachment 771415 [details] [diff] [review]
use twisted.python.lockfile v2

Review of attachment 771415 [details] [diff] [review]:
-----------------------------------------------------------------

::: buildfarm/release/release-runner.py
@@ +220,2 @@
>      # Shorthand
> +    sendchange_master = config.get('release-runner', 'sendchange_master')

Not worried about the "if not sendchange_master" check anymore?

Comment 4

[Rail Aliiev [:rail]]

(In reply to Ben Hearsum [:bhearsum] from comment #3)
> Comment on attachment 771415 [details] [diff] [review]
> use twisted.python.lockfile v2
> 
> Review of attachment 771415 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: buildfarm/release/release-runner.py
> @@ +220,2 @@
> >      # Shorthand
> > +    sendchange_master = config.get('release-runner', 'sendchange_master')
> 
> Not worried about the "if not sendchange_master" check anymore?

No. config.get raises an exception in that case, what is kind of the same. parser.error is not really what we want there.

Comment 5

[Rail Aliiev [:rail]]

Comment on attachment 771415 [details] [diff] [review]
use twisted.python.lockfile v2

https://hg.mozilla.org/build/tools/rev/49eef4de11a6 and deployed on bm81

[[e-mail] tools]$ 

$ pwd 
/builds/releaserunner/tools
[[e-mail] tools]$ hg st
? buildfarm/release/.release-runner.lock
[[e-mail] tools]$ ls -l buildfarm/release/.release-runner.lock
lrwxrwxrwx 1 cltbld cltbld 5 Jul  4 11:51 buildfarm/release/.release-runner.lock -> 28187

The library uses a (broken) symlink to PID.

Comment 6

[Rail Aliiev [:rail]]

$ ps -p `readlink buildfarm/release/.release-runner.lock` uw
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
cltbld   28187  0.0  0.3 218088 20196 ?        S    11:51   0:00 python release-runner.py -c /builds/releaserunner/release-runner.ini