713271 - talos-master complains about buildbot because several update_from_files.py are blocked

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Armen [:armenzg]]

It is probably going to go away after the DB issues are solved but just in case I'm filing it.

It seems the process is supposed to run for a little bit and then go away but it seems it gets hung because of the DB issues.

Probably getting rid of "buildbotcustom" out of the path would prevent this.

[62] talos-master:buildbot is CRITICAL: PROCS CRITICAL: 4 processes with args buildbot

[cltbld@talos-master ~]$ tail -F ~cltbld/update.log 
    self.connection = self.__connect()
  File "/tools/python-2.5.1/lib/python2.5/site-packages/SQLAlchemy-0.5.5-py2.5.egg/sqlalchemy/pool.py", line 261, in __connect
    connection = self.__pool._creator()
  File "/tools/python-2.5.1/lib/python2.5/site-packages/SQLAlchemy-0.5.5-py2.5.egg/sqlalchemy/engine/strategies.py", line 80, in connect
    raise exc.DBAPIError.instance(None, None, e)
sqlalchemy.exc.OperationalError: (OperationalError) (2013, "Lost connection to MySQL server at 'reading initial communication packet', system error: 104") None None

real    121m51.383s
user    0m0.217s
sys     0m0.093s

[cltbld@talos-master ~]$ ps aux | grep buildbot 
cltbld   19787  0.0  0.0   2400   960 ?        Ss   09:20   0:00 /bin/sh -c lockfile -60 -r 3 $HOME/lockfile.bbdb 2>/dev/null && (source /home/cltbld/.bash_profile && cd /builds/buildbot/talos-pool; time nice python /tools/buildbotcustom/buildbotcustom/bin/update_from_files.py -c /home/cltbld/.bbcustom_update_from_files-pool.conf .; rm -f $HOME/lockfile.bbdb) >> update.log 2>&1
cltbld   19789  0.0  0.0   4480  1048 ?        S    09:20   0:00 /bin/sh -c lockfile -60 -r 3 $HOME/lockfile.bbdb 2>/dev/null && (source /home/cltbld/.bash_profile && cd /builds/buildbot/talos-pool; time nice python /tools/buildbotcustom/buildbotcustom/bin/update_from_files.py -c /home/cltbld/.bbcustom_update_from_files-pool.conf .; rm -f $HOME/lockfile.bbdb) >> update.log 2>&1
cltbld   19812  0.0  0.3  30240 13352 ?        SN   09:20   0:00 python /tools/buildbotcustom/buildbotcustom/bin/update_from_files.py -c /home/cltbld/.bbcustom_update_from_files-pool.conf .
cltbld   22516  0.0  0.0   4032   696 pts/0    R+   11:21   0:00 grep buildbot
cltbld   27226  0.1  2.3 119168 91792 ?        Sl    2010 1088:58 python /tools/buildbot/bin/buildbot start /builds/buildbot/talos-pool

This is all started by this cronjob:
*/10 * * * *    lockfile -60 -r 3 $HOME/lockfile.bbdb 2>/dev/null && (source /home/cltbld/.bash_profile && cd /builds/buildbot/talos-pool; time nice python /tools/buildbotcustom/buildbotcustom/bin/update_from_files.py -c /home/cltbld/.bbcustom_update_from_files-pool.conf .; rm -f $HOME/lockfile.bbdb) >> update.log 2>&1

Comment 1

[Armen [:armenzg]]

talos-master:buildbot is OK: PROCS OK: 1 process with args buildbot

It seems it is all good now.