1185507 - relengapi: use MySQL as the celery results backend

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

This should be a minor config change, but will need to be tested.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

OK, I have this configured on the staging relengapi.  I've tested it out with badpenny tasks (which ordinarily ignore their results).

I also added a `run-celery.sh` that you can run on a celery node (from /data/www/relengapi) to run the celery command line.

Jordan, can you do a bit of stress-testing of the staging site, and we can take a look at the DB tables to see what they look like?

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

BTW, all I changed was RELENGAPI_BACKEND="db+mysql://...".

It took me a while to figure out that celery tasks with ignore_results=True wouldn't even create the results table.  I had to hack in a task without that parameter set to get the results table to appear.

Comment 3

[Jordan Lund (:jlund)]

(In reply to Dustin J. Mitchell [:dustin] from comment #2)
> BTW, all I changed was RELENGAPI_BACKEND="db+mysql://...".
> 
> It took me a while to figure out that celery tasks with ignore_results=True
> wouldn't even create the results table.  I had to hack in a task without
> that parameter set to get the results table to appear.

archiver shouldn't need such a hack so we should get a clear idea of how well this works with some stress tests. I'll write a bash script to fire off a bunch of archiver_client.py calls with various branches/revs

Comment 4

[Jordan Lund (:jlund)]

staging results (pun intended) looked good.

however we discovered that the automagic cleanup (deletion) of result 'stored task tombstones' do not happen for database backends unless we:

1) flip on 'celery beat'
2) implement our own 'cleanup' via badpenny job
3) use a different backend like redis

context: http://docs.celeryproject.org/en/latest/configuration.html#std:setting-CELERY_TASK_RESULT_EXPIRES

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

I just landed this, in hopes it will decrease the frequency of bug 1185098, where the Flask app is failing to connect to amqp to fetch results from the backend.  The MySQL backend may do a better job of reconnecting.

To revert: set CELERY_BACKEND='amqp' in settings.py.

We still need to set up the cleanup task using one of the approaches jordan gave in comment 4.

Comment 6

[Jordan Lund (:jlund)]

Attachment: Screen Shot 2015-07-24 at 1.22.06 PM.png

\o/ you can see the effect once this landed around 0930 PT

this has drastically reduced the number of errors like the following that we were hitting https://bugzilla.mozilla.org/show_bug.cgi?id=1185098#c0

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

351 archiver tasks over the weekend, about 68% of which were try jobs.  And I can confirm there's no cleanup task running!

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

I think we can just call the task directly from a badpenny task -- which will give better tracking, while still maintaining Celery's default behavior.  I'm asking to verify here:

https://github.com/celery/celery/pull/2728

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

https://github.com/mozilla/build-relengapi/pull/312

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

Pull req is in staging now and will soon be in prod.