Celery is not relesing App, Even queue.purge
Categories
(Webtools Graveyard :: Pontoon, defect)
Tracking
(Not tracked)
People
(Reporter: bjh13579, Unassigned)
Details
User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:87.0) Gecko/20100101 Firefox/87.0
Steps to reproduce:
Pontoon is good tool. so my pontoon intance is very heavily load. so Many Sigterm occur. Problem is R15 Sigterm after Cerely is holding that project, even i trying this - (not celery 5, just relesing version)
Replace my-app-name with your Heroku app's name.
celery amqp --broker=heroku config:get RABBITMQ_URL --app=my-app-name
Replace my-queue-name with your queue's name (e.g. celery).
1> queue.purge my-queue-name
queue name - serial_task.pontoon.sync.tasks.sync_project[project=11] is not releaing celery. what can i do? i wasting my money for sky.
Actual results:
2021-04-15T05:58:14.143309+00:00 app[worker.1]: [2021-04-15 05:58:14,142: ERROR/ForkPoolWorker-5] Task pontoon.sync.tasks.sync_project[14e332f3-d6d4-42b1-beef-28a4773f14b1] raised unexpected: RuntimeError("Can't execute task 'serial_task.pontoon.sync.tasks.sync_project[project=11]' because the previously called task is still running.")
2021-04-15T05:58:14.143318+00:00 app[worker.1]: Traceback (most recent call last):
2021-04-15T05:58:14.143319+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.8/site-packages/celery/app/trace.py", line 385, in trace_task
2021-04-15T05:58:14.143320+00:00 app[worker.1]: R = retval = fun(*args, **kwargs)
2021-04-15T05:58:14.143321+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.8/site-packages/newrelic/hooks/application_celery.py", line 84, in wrapper
2021-04-15T05:58:14.143321+00:00 app[worker.1]: return wrapped(*args, **kwargs)
2021-04-15T05:58:14.143321+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.8/site-packages/celery/app/trace.py", line 648, in protected_call
2021-04-15T05:58:14.143322+00:00 app[worker.1]: return self.run(*args, **kwargs)
2021-04-15T05:58:14.143322+00:00 app[worker.1]: File "/app/pontoon/sync/core.py", line 66, in wrapped_func
2021-04-15T05:58:14.143323+00:00 app[worker.1]: raise error
Expected results:
Celery is automatic stop when R15 Sigterm, and Restarting for this jobs.
Reporter | ||
Comment 1•3 years ago
|
||
ugh, sorry for my english. anyway, problem is simple. in my cronjob(dyno restart) is not disturbance celery's working. but Occur R15's sigterm is very annoying and not stopping and maintain the last working queue. so just wait is only answer.
trying readthedoc's queue releasing(or AMQP's Web interface's Queue purge), it's not working, so i just waiting for time and resend the sync_projects
Comment 2•3 years ago
•
|
||
From the SYNC_TASK_TIMEOUT docs:
Multiple sync tasks for the same project cannot run concurrently to prevent potential DB and VCS inconsistencies. We store the information about the running task in cache and clear it after the task completes. In case of an error, we might never clear the cache, so we use SYNC_TASK_TIMEOUT as the longest possible period after which the cache is cleared and the subsequent task can run. The value should exceed the longest sync task of the instance. The default value is 3600 seconds (1 hour).
If you want to manually empty the cache and run subsequent sync task before SYNC_TASK_TIMEOUT expires, you can run the following commands from the CLI:
from django.core.cache import cache
cache.delete('serial_task.pontoon.sync.tasks.sync_project[project=11]')
If Heroku is restarting your dynos, because they run out of memory, you need to upgrade the plan.
Updated•3 years ago
|
Description
•