1273081 - [mozillians.org][prod][celery] Database gave error: OperationalError(2006, 'MySQL server has gone away')

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Engagement, task)

Description

[John Giannelos [:nemo-yiannis]]

We are getting these errors triggered by celery:

Database gave error: OperationalError(2006, 'MySQL server has gone away')
Database error while sync: OperationalError(2006, 'MySQL server has gone away')

It looks like this was triggered by some DB work during the weekend. Possible relevant bug from IRC discussion (which I don't have access) is 1264322.

We fixed this issue on stage by forcing a chief push that restarts celery.
Can you restart celery on prod too?

Comment 1

[John Giannelos [:nemo-yiannis]]

Here are the sentry errors:

https://sentry.prod.mozaws.net/operations/mozilliansorg-prod/issues/332189/
https://sentry.prod.mozaws.net/operations/mozilliansorg-prod/issues/332194/

Comment 2

[Peter Radcliffe [:pir]]

Per request in #moc ran:
  /etc/init.d/celeryd-mozillians-prod restart
  /etc/init.d/celeryd-mozillians-prod-beat restart

on python[1-4].webapp.phx1.mozilla.com

Presumably this still needs to be looked into as to why it failed in the first place from a db failover, leaving bug open.

Comment 3

[Peter Radcliffe [:pir]]

Also ran by request:
  /etc/init.d/celeryd-mozillians-dev restart
on python1.dev.webapp.phx1

Comment 4

[John Giannelos [:nemo-yiannis]]

Things look OK with mozillians.org celery services. Errors stopped triggering. Let's leave this one open to investigate what went wrong.

Comment 5

[Shyam Mani [:fox2mike]]

Sorry, we're pretty swamped at the moment...let's revisit this if it happens again.

Comment 6

[John Giannelos [:nemo-yiannis]]

We are getting the same errors. Can you restart the celery services in all envs as described in comments #2 and #3.

Comment 7

[Ludovic Hirlimann [:Usul]]

lowering before it ping me, on it.

Comment 8

[Ludovic Hirlimann [:Usul]]

Per request in #moc ran:
  /etc/init.d/celeryd-mozillians-prod restart
  /etc/init.d/celeryd-mozillians-prod-beat restart

on python[1-4].webapp.phx1.mozilla.com

ran :
/etc/init.d/celeryd-mozillians-stage restart
/etc/init.d/celeryd-mozillians-stage-beat restart
on python[12].stage.webapp.phx1.mozilla.com

Comment 9

[Ludovic Hirlimann [:Usul]]

Did dev also as per comment #3

Comment 10

[John Giannelos [:nemo-yiannis]]

Awesome! I'll keep an eye on it to check for any errors. If everything is OK i will close the bug. Thanks for the fast response!