1120473 - DB cleanup is losing connection to database

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Chris AtLee [:catlee]]

Attachment: add pool_recycle

Our DB maintenance script has been dying over the past few weeks because it loses its connection to the database:

Traceback (most recent call last):
  File "/builds/buildbot/db_maint/tools/buildfarm/maintenance/cleanup_db.py", line 284, in <module>
    cleanup_statusdb_builds(meta, options.cutoff)
  File "/builds/buildbot/db_maint/tools/buildfarm/maintenance/cleanup_db.py", line 109, in cleanup_statusdb_builds
    deleter(t_builds.c.id),
  File "/builds/buildbot/db_maint/tools/buildfarm/maintenance/cleanup_db.py", line 71, in cleaner_upper
    q(ids[i:i + chunk_size])
  File "/builds/buildbot/db_maint/tools/buildfarm/maintenance/cleanup_db.py", line 86, in delete_func
    n = q.execute().rowcount
  File "/builds/buildbot/db_maint/lib/python2.7/site-packages/sqlalchemy/sql/expression.py", line 2841, in execute
    return e._execute_clauseelement(self, multiparams, params)
  File "/builds/buildbot/db_maint/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2453, in _execute_clauseelement
    return connection._execute_clauseelement(elem, multiparams, params)
  File "/builds/buildbot/db_maint/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/builds/buildbot/db_maint/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1698, in _execute_context
    context)
  File "/builds/buildbot/db_maint/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1691, in _execute_context
    context)
  File "/builds/buildbot/db_maint/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 331, in do_execute
    cursor.execute(statement, parameters)
  File "/builds/buildbot/db_maint/lib/python2.7/site-packages/MySQLdb/cursors.py", line 174, in execute
    self.errorhandler(self, exc, value)
  File "/builds/buildbot/db_maint/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
sqlalchemy.exc.OperationalError: (OperationalError) (2006, 'MySQL server has gone away') 'DELETE FROM steps WHERE steps.build_id IN (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)' (32324066L, ...

going to try adding a pool recycle timeout to workaround this.

Comment 1

[[e-mail] (:bhearsum)]

Comment on attachment 8547632 [details] [diff] [review]
add pool_recycle

Reduce, reuse, recycle!