1369300 - cron job to run /data/genericrhel6/deploy is locked

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Edmund Wong (:ewong)]

Every 10 minutes, seamonkey-release@m.o is getting the following email:

Couldn't get build lock; waiting another 15 seconds...
Couldn't get build lock; waiting another 15 seconds...
Couldn't get build lock; waiting another 15 seconds...
Couldn't get build lock. at bin/build-exclusive.pl line 51.

Started @ around 12:50 PDT on 31st May.  

I'm guessing there's a hung job or a crashed job that left a lock..
Not too sure.

Filing this bug.

Comment 1

[Ryan C [:ryanc] (UTC-4)]

Assisting Edmund, here is the log which fixed the problem,

[[e-mail]1 rchilds]# grep -nr "build-exclusive.pl" /etc/cron.d/*
/etc/cron.d/www.bugzilla.org:9:#*/10          0-7,9-23 * * * root (cd /data/genericrhel6/build/bugzilla-org ; bin/build-exclusive.pl; /data/genericrhel6/deploy -nq www.bugzilla.org)
/etc/cron.d/www.bugzilla.org:10:#0-10,40-59/10 8        * * * root (cd /data/genericrhel6/build/bugzilla-org ; bin/build-exclusive.pl; /data/genericrhel6/deploy -nq www.bugzilla.org)
/etc/cron.d/www.seamonkeyproject.org:3:*/10          0-7,9-23 * * * root (cd /data/genericrhel6/build/seamonkeyproject-org; bin/build-exclusive.pl; /data/genericrhel6/deploy -nq www.seamonkeyproject.org)
/etc/cron.d/www.seamonkeyproject.org:4:0-10,30-59/10 8        * * * root (cd /data/genericrhel6/build/seamonkeyproject-org; bin/build-exclusive.pl; /data/genericrhel6/deploy -nq www.seamonkeyproject.org)

[[e-mail]1 rchilds]# ps aux | grep sea
rchilds  16789  0.0  0.0 103328   848 pts/3    S+   07:25   0:00 grep sea
root     28952  0.0  0.0 106120  1132 ?        Ss   May31   0:00 /bin/sh -c (cd /data/genericrhel6/build/seamonkeyproject-org; bin/build-exclusive.pl; /data/genericrhel6/deploy -nq www.seamonkeyproject.org)
root     28981  0.0  0.0 106120   528 ?        S    May31   0:00 /bin/sh -c (cd /data/genericrhel6/build/seamonkeyproject-org; bin/build-exclusive.pl; /data/genericrhel6/deploy -nq www.seamonkeyproject.org)

[[e-mail]1 rchilds]# ls -alh /tmp/smprojorg-build.pid
-rw-r--r-- 1 root root 6 May 31 19:40 /tmp/smprojorg-build.pid

[[e-mail]1 ~]$ sudo cat /tmp/smprojorg-build.pid
29022

[[e-mail]1 ~]$ ps aux | grep 29022
root     29022  0.0  0.0 125568  3664 ?        S    May31   0:00 /usr/bin/perl -w bin/build-exclusive.pl
rchilds  29791  0.0  0.0 103328   852 pts/3    S+   07:33   0:00 grep 29022

[[e-mail]1 ~]$ sudo strace -p 29022
Process 29022 attached
wait4(29108, ^CProcess 29022 detached
 <detached ...>

[[e-mail]1 ~]$ sudo kill -9 29022

[[e-mail]1 rchilds]# sudo rm -rf /tmp/smprojorg-build.pid

[[e-mail]1 ~]$ ps aux | grep sea
rchilds   4811  0.0  0.0 103328   848 pts/3    S+   07:42   0:00 grep sea
root     31857  0.0  0.0 106120  1132 ?        Ss   07:40   0:00 /bin/sh -c (cd /data/genericrhel6/build/seamonkeyproject-org; bin/build-exclusive.pl; /data/genericrhel6/deploy -nq www.seamonkeyproject.org)
root     31896  0.0  0.0 106120   528 ?        S    07:40   0:00 /bin/sh -c (cd /data/genericrhel6/build/seamonkeyproject-org; bin/build-exclusive.pl; /data/genericrhel6/deploy -nq www.seamonkeyproject.org)

Comment 2

[Edmund Wong (:ewong)]

Unfortunately, I thought it was fixed, but now we're getting:

waiting for lock on repository /data/genericrhel6/build/seamonkeyproject-org held by 'genericadm.private.phx1.mozilla.com:29119'
abort: repository /data/genericrhel6/build/seamonkeyproject-org: timed out waiting for lock held by genericadm.private.phx1.mozilla.com:29119

so something is definitely not working well..

Callek, any ideas?

Comment 3

[Edmund Wong (:ewong)]

(In reply to Edmund Wong (:ewong) from comment #2)
> Unfortunately, I thought it was fixed, but now we're getting:
> 
> waiting for lock on repository /data/genericrhel6/build/seamonkeyproject-org
> held by 'genericadm.private.phx1.mozilla.com:29119'
> abort: repository /data/genericrhel6/build/seamonkeyproject-org: timed out
> waiting for lock held by genericadm.private.phx1.mozilla.com:29119
> 
> so something is definitely not working well..
> 
> Callek, any ideas?

Apparently :ryanc killing off that process allowed another process to run which
pointed out that 29119 was still running (from 31st May).  Got :ryanc to kill
off that and so far, (touch wood), haven't gotten anything from it.  

That said, on the record, that host has hg 3.7.1 and while I don't think
it is the problem, I won't discount the possibility that we'll need to
upgrade that.  However, according to :ryanc, we'll need to go through
the CAB (Change Approval Board) in order to get this host' hg upgraded.

Comment 4

[Edmund Wong (:ewong)]

That said, I'll resolve this bug.

Thanks :ryanc!  And thanks :ashlee for pointing me to the right person.

Comment 5

[Edmund Wong (:ewong)]

It's doing this again. :(

Couldn't get build lock; waiting another 15 seconds...
Couldn't get build lock; waiting another 15 seconds...
Couldn't get build lock; waiting another 15 seconds...
Couldn't get build lock. at bin/build-exclusive.pl line 51.

So I am guessing there is still one pid running and got hung up again and another instance
tried to run as well.

Comment 6

[Edmund Wong (:ewong)]

(In reply to Edmund Wong (:ewong) from comment #5)
> It's doing this again. :(
> 
> Couldn't get build lock; waiting another 15 seconds...
> Couldn't get build lock; waiting another 15 seconds...
> Couldn't get build lock; waiting another 15 seconds...
> Couldn't get build lock. at bin/build-exclusive.pl line 51.
> 
> So I am guessing there is still one pid running and got hung up again and
> another instance
> tried to run as well.

probably caused by https://hg.mozilla.org/SeaMonkey/seamonkey-project-org/rev/0d7aa4fa97871fe6b248a8ed95fc3ec7b79f0a95

Will need to have that fixed before this is fixed.

Comment 7

[Justin Lazaro [:jlaz] (use needinfo)]

Killed locked processes that involved build-exclusive.pl.  ewong confirms over IRC that all is well:

23:35 < ewong> jlaz  I think that did it!  Thanks!

Comment 8

[Edmund Wong (:ewong)]

it's back to being locked again..   so I must've missed something else.

If someone can do the following, it would be most appreciated:

1) find the process that's got a lock on /tmp/smprojorg-build
2) find the arguments for the command line
3) kill all the processes
4) go  to the root of this account (where it has bin, etc, lib, src, README..)
5) run "bin/build <argument list>"
6) pastebin the contents here.

it eventually runs ttree etc/ttree.cfg *argument list*

I'm just curious as to what is busting it.. if anything at all.

Comment 9

[Edmund Wong (:ewong)]

actually, looking at the email subject, it does:

cd /data/static/build/seamonkeyproject-org; bin/build-exclusive.pl; /data/static/deploy -nq www.seamonkeyproject.org


So I am guessing there are no @ARGV in

https://hg.mozilla.org/SeaMonkey/seamonkey-project-org/file/tip/bin/build-exclusive.pl#l59

and so, it'd just be:

ttree etc/ttree.cfg

Comment 10

[Edmund Wong (:ewong)]

here's an addendum... (to add to the confusion)

On the 5th July (and previously) we were getting email from genericadm.private.phx1.mozilla.com

On the 8th July (and forwards), we're getting email from  staticadm.private.phx1.mozilla.com 

Anyone got any ideas of what's going on?

Comment 11

[Edmund Wong (:ewong)]

as an addendum after talking to :jlaz,  it was an hg process that was holding up the lock.  Why? No clue.

Though..  /usr/bin/hg is 3.7.3  so this needs updating.

So, as a summary, I'm wondering if someone can clarify the following:

1) Should the deploy process be running from genericadm, or from staticadm?
2) either way, /usr/bin/hg needs to be updated to at least 3.9.1., but preference is to 4.2.2
3) not sure if it's possible to find out right now, but how come we changed from using genericadm to staticadm? 
   (5th.. using genericadm...)  7th,  using staticadm. (since the hung hg process was from 7th July).

Anyone know who I can ping?

Thanks!

Comment 12

[Ryan C [:ryanc] (UTC-4)]

I believe this is a WebOps owned system - Moving in that direction to see if they can assist.

Comment 13

[Edmund Wong (:ewong)]

After :jlaz unhorked it on the 29th August, it's now doing the same thing again starting at 6am on 6th Sept.

On the 2nd Sept, an email from root@genericadm:

Cron <root@genericadm> (cd /data/genericrhel6/build/seamonkeyproject-org; bin/build-exclusive.pl; /data/genericrhel6/deploy -nq www.seamonkeyproject.org)

  with an "abort: HTTP Error 500: Internal Server Error"

Then today at 6:10am, seamonkey-release started getting:

Cron <root@staticadm> (cd /data/static/build/seamonkeyproject-org; bin/build-exclusive.pl; /data/static/deploy -nq www.seamonkeyproject.org)

  with the contents:

Couldn't get build lock; waiting another 15 seconds...
Couldn't get build lock; waiting another 15 seconds...
Couldn't get build lock; waiting another 15 seconds...
Couldn't get build lock. at bin/build-exclusive.pl line 51.

is anyone available to clarify comment #11?

I don't understand why I sometimes get email from root@genericadm and other times (when something
has gone wrong) from root@staticadm.   I doubt they are the same system.

Comment 14

[Edmund Wong (:ewong)]

Thanks for any help.

(clicked save changes too early.. )

Comment 15

[Joey Krejci [:joeyk]]

Removed cron job from puppet repo:

modules/webapp/files/static/admin/etc-cron.d/www.seamonkeyproject.org

Also removed cron from static-adm:

/etc/cron.d/www.seamonkeyproject.org


In response to comment 13, static and generic are different and static is actually EOL so Im hoping removing this cron will fix the strange issues you're seeing. Let me know if it re-appears and I'll leave the bug open for now.

Comment 16

[Edmund Wong (:ewong)]

(In reply to Joey Krejci [:joeyk] from comment #15)
> Removed cron job from puppet repo:
> 
> modules/webapp/files/static/admin/etc-cron.d/www.seamonkeyproject.org
> 
> Also removed cron from static-adm:
> 
> /etc/cron.d/www.seamonkeyproject.org
> 
> 
> In response to comment 13, static and generic are different and static is
> actually EOL so Im hoping removing this cron will fix the strange issues
> you're seeing. Let me know if it re-appears and I'll leave the bug open for
> now.

It looks like this fixed it!  

Thanks Joey!

Comment 17

[Edmund Wong (:ewong)]

it's rearing it's head again

Comment 18

[Joey Krejci [:joeyk]]

Hey Edmund,

Can you send me some console output? 

Thanks!

Comment 19

[Joey Krejci [:joeyk]]

So we found the hung process, and 'hg' is also still at 3.7.3. I feel like that might be the root cause of the issue, however we don't want to update it without knowing what else is using that and if it will break other builds, etc if we upgrade. 

Here was the hung process which is spawned by /data/genericrhel6/build/seamonkeyproject-org/bin/build-exclusive.pl Line 44:

print STDERR "Couldn't get build lock; waiting another $WAIT seconds...\n";

root     32090  0.0  0.1 189932 12604 ?        S    00:00   0:00 /usr/bin/python /usr/bin/hg pull -q -u

After I killed 32090 the cron job completed. 

So for now if it gets hung just ping #webops on Slack, refer to this bug and someone will handle it. Im also going to add the WebOps team to that cron email so we can be notified as well.

Comment 20

[Edmund Wong (:ewong)]

(In reply to Joey Krejci [:joeyk] from comment #18)
> Hey Edmund,
> 
> Can you send me some console output? 
> 
> Thanks!

Sorry, I'm not sure what you mean by console output.  I only get cron email jobs. 

I think I asked earlier whether genericadm can be upgraded to a newer hg (i.e. comment #3).
Not sure who I need to ping to get it upgraded.  Or can an updated hg be installed elsewhere
(i.e. /opt/mercurial) and the cron job point to it? (yes.. a bit handwavey.. don't know
the setup on that box so just speculating).

Comment 21

[Edmund Wong (:ewong)]

The hg process is horked again.

Anyone know who we can ping to possibly update Mercurial on this machine?

Comment 22

[Edmund Wong (:ewong)]

An option could also be to install a new mercurial version in /opt/mercurial
and have the cron job use that mercurial.  (of course..it's probably
not that simple.. )

Comment 23

[Joey Krejci [:joeyk]]

Hey Edmund, 

I created a new Bug 1421371, to review the upgrade process. 

Let's move the discussion into there for now and keep this bug closed out. 

Thanks!

- Joey K

Comment 24

[Edmund Wong (:ewong)]

(In reply to Joey Krejci [:joeyk] from comment #23)
> Hey Edmund, 
> 
> I created a new Bug 1421371, to review the upgrade process. 
> 
> Let's move the discussion into there for now and keep this bug closed out. 
> 
> Thanks!
> 
> - Joey K

Thanks Joey.

Can you cc me to that bug?

Comment 25

[Edmund Wong (:ewong)]

this bug is rearing it's head again.

can someone kill the locked process?

Comment 26

[Edmund Wong (:ewong)]

This is the current msg:

Couldn't get build lock; waiting another 15 seconds...
Got build lock after 15 seconds.
waiting for lock on repository /data/genericrhel6/build/seamonkeyproject-org held by 'genericadm.private.phx1.mozilla.com:9918'
abort: repository /data/genericrhel6/build/seamonkeyproject-org: timed out waiting for lock held by genericadm.private.phx1.mozilla.com:9918

Comment 27

[Edmund Wong (:ewong)]

fwiw..  the hung process isn't exclusive.pl.. but mercurial.

Comment 28

[Edmund Wong (:ewong)]

after :vinh killed the errant mercurial process,  I haven't received a single spam from genericadm.
So going to resolve this as fixed.