Closed
Bug 1185507
Opened 9 years ago
Closed 9 years ago
relengapi: use MySQL as the celery results backend
Categories
(Infrastructure & Operations :: RelOps: General, task)
Infrastructure & Operations
RelOps: General
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: dustin, Assigned: dustin)
References
Details
Attachments
(1 file)
164.51 KB,
image/png
|
Details |
This should be a minor config change, but will need to be tested.
Assignee | ||
Updated•9 years ago
|
Assignee: relops → dustin
Assignee | ||
Comment 1•9 years ago
|
||
OK, I have this configured on the staging relengapi. I've tested it out with badpenny tasks (which ordinarily ignore their results).
I also added a `run-celery.sh` that you can run on a celery node (from /data/www/relengapi) to run the celery command line.
Jordan, can you do a bit of stress-testing of the staging site, and we can take a look at the DB tables to see what they look like?
Flags: needinfo?(jlund)
Assignee | ||
Comment 2•9 years ago
|
||
BTW, all I changed was RELENGAPI_BACKEND="db+mysql://...".
It took me a while to figure out that celery tasks with ignore_results=True wouldn't even create the results table. I had to hack in a task without that parameter set to get the results table to appear.
Comment 3•9 years ago
|
||
(In reply to Dustin J. Mitchell [:dustin] from comment #2)
> BTW, all I changed was RELENGAPI_BACKEND="db+mysql://...".
>
> It took me a while to figure out that celery tasks with ignore_results=True
> wouldn't even create the results table. I had to hack in a task without
> that parameter set to get the results table to appear.
archiver shouldn't need such a hack so we should get a clear idea of how well this works with some stress tests. I'll write a bash script to fire off a bunch of archiver_client.py calls with various branches/revs
Flags: needinfo?(jlund)
Comment 4•9 years ago
|
||
staging results (pun intended) looked good.
however we discovered that the automagic cleanup (deletion) of result 'stored task tombstones' do not happen for database backends unless we:
1) flip on 'celery beat'
2) implement our own 'cleanup' via badpenny job
3) use a different backend like redis
context: http://docs.celeryproject.org/en/latest/configuration.html#std:setting-CELERY_TASK_RESULT_EXPIRES
Assignee | ||
Comment 5•9 years ago
|
||
I just landed this, in hopes it will decrease the frequency of bug 1185098, where the Flask app is failing to connect to amqp to fetch results from the backend. The MySQL backend may do a better job of reconnecting.
To revert: set CELERY_BACKEND='amqp' in settings.py.
We still need to set up the cleanup task using one of the approaches jordan gave in comment 4.
Comment 6•9 years ago
|
||
\o/ you can see the effect once this landed around 0930 PT
this has drastically reduced the number of errors like the following that we were hitting https://bugzilla.mozilla.org/show_bug.cgi?id=1185098#c0
Assignee | ||
Comment 7•9 years ago
|
||
351 archiver tasks over the weekend, about 68% of which were try jobs. And I can confirm there's no cleanup task running!
Assignee | ||
Comment 8•9 years ago
|
||
I think we can just call the task directly from a badpenny task -- which will give better tracking, while still maintaining Celery's default behavior. I'm asking to verify here:
https://github.com/celery/celery/pull/2728
Assignee | ||
Comment 9•9 years ago
|
||
Assignee | ||
Comment 10•9 years ago
|
||
Pull req is in staging now and will soon be in prod.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•