DB cleanup is losing connection to database

RESOLVED FIXED

Status

Release Engineering
General
RESOLVED FIXED
4 years ago
2 months ago

People

(Reporter: catlee, Assigned: catlee)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Assignee)

Description

4 years ago
Created attachment 8547632 [details] [diff] [review]
add pool_recycle

Our DB maintenance script has been dying over the past few weeks because it loses its connection to the database:

Traceback (most recent call last):
  File "/builds/buildbot/db_maint/tools/buildfarm/maintenance/cleanup_db.py", line 284, in <module>
    cleanup_statusdb_builds(meta, options.cutoff)
  File "/builds/buildbot/db_maint/tools/buildfarm/maintenance/cleanup_db.py", line 109, in cleanup_statusdb_builds
    deleter(t_builds.c.id),
  File "/builds/buildbot/db_maint/tools/buildfarm/maintenance/cleanup_db.py", line 71, in cleaner_upper
    q(ids[i:i + chunk_size])
  File "/builds/buildbot/db_maint/tools/buildfarm/maintenance/cleanup_db.py", line 86, in delete_func
    n = q.execute().rowcount
  File "/builds/buildbot/db_maint/lib/python2.7/site-packages/sqlalchemy/sql/expression.py", line 2841, in execute
    return e._execute_clauseelement(self, multiparams, params)
  File "/builds/buildbot/db_maint/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2453, in _execute_clauseelement
    return connection._execute_clauseelement(elem, multiparams, params)
  File "/builds/buildbot/db_maint/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/builds/buildbot/db_maint/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1698, in _execute_context
    context)
  File "/builds/buildbot/db_maint/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1691, in _execute_context
    context)
  File "/builds/buildbot/db_maint/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 331, in do_execute
    cursor.execute(statement, parameters)
  File "/builds/buildbot/db_maint/lib/python2.7/site-packages/MySQLdb/cursors.py", line 174, in execute
    self.errorhandler(self, exc, value)
  File "/builds/buildbot/db_maint/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
sqlalchemy.exc.OperationalError: (OperationalError) (2006, 'MySQL server has gone away') 'DELETE FROM steps WHERE steps.build_id IN (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)' (32324066L, ...

going to try adding a pool recycle timeout to workaround this.
Attachment #8547632 - Flags: review?(bhearsum)
Comment on attachment 8547632 [details] [diff] [review]
add pool_recycle

Reduce, reuse, recycle!
Attachment #8547632 - Flags: review?(bhearsum) → review+
(Assignee)

Updated

4 years ago
Depends on: 1120481
(Assignee)

Updated

4 years ago
Attachment #8547632 - Flags: checked-in+
(Assignee)

Updated

4 years ago
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
Component: General Automation → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.