Closed Bug 1705305 Opened 3 years ago Closed 3 years ago

Celery is not relesing App, Even queue.purge

Categories

(Webtools Graveyard :: Pontoon, defect)

defect

Tracking

(Not tracked)

RESOLVED WORKSFORME

People

(Reporter: bjh13579, Unassigned)

Details

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:87.0) Gecko/20100101 Firefox/87.0

Steps to reproduce:

Pontoon is good tool. so my pontoon intance is very heavily load. so Many Sigterm occur. Problem is R15 Sigterm after Cerely is holding that project, even i trying this - (not celery 5, just relesing version)

Replace my-app-name with your Heroku app's name.

celery amqp --broker=heroku config:get RABBITMQ_URL --app=my-app-name

Replace my-queue-name with your queue's name (e.g. celery).

1> queue.purge my-queue-name

queue name - serial_task.pontoon.sync.tasks.sync_project[project=11] is not releaing celery. what can i do? i wasting my money for sky.

Actual results:

2021-04-15T05:58:14.143309+00:00 app[worker.1]: [2021-04-15 05:58:14,142: ERROR/ForkPoolWorker-5] Task pontoon.sync.tasks.sync_project[14e332f3-d6d4-42b1-beef-28a4773f14b1] raised unexpected: RuntimeError("Can't execute task 'serial_task.pontoon.sync.tasks.sync_project[project=11]' because the previously called task is still running.")

2021-04-15T05:58:14.143318+00:00 app[worker.1]: Traceback (most recent call last):

2021-04-15T05:58:14.143319+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.8/site-packages/celery/app/trace.py", line 385, in trace_task

2021-04-15T05:58:14.143320+00:00 app[worker.1]: R = retval = fun(*args, **kwargs)

2021-04-15T05:58:14.143321+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.8/site-packages/newrelic/hooks/application_celery.py", line 84, in wrapper

2021-04-15T05:58:14.143321+00:00 app[worker.1]: return wrapped(*args, **kwargs)

2021-04-15T05:58:14.143321+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.8/site-packages/celery/app/trace.py", line 648, in protected_call

2021-04-15T05:58:14.143322+00:00 app[worker.1]: return self.run(*args, **kwargs)

2021-04-15T05:58:14.143322+00:00 app[worker.1]: File "/app/pontoon/sync/core.py", line 66, in wrapped_func

2021-04-15T05:58:14.143323+00:00 app[worker.1]: raise error

Expected results:

Celery is automatic stop when R15 Sigterm, and Restarting for this jobs.

ugh, sorry for my english. anyway, problem is simple. in my cronjob(dyno restart) is not disturbance celery's working. but Occur R15's sigterm is very annoying and not stopping and maintain the last working queue. so just wait is only answer.

trying readthedoc's queue releasing(or AMQP's Web interface's Queue purge), it's not working, so i just waiting for time and resend the sync_projects

From the SYNC_TASK_TIMEOUT docs:

Multiple sync tasks for the same project cannot run concurrently to prevent potential DB and VCS inconsistencies. We store the information about the running task in cache and clear it after the task completes. In case of an error, we might never clear the cache, so we use SYNC_TASK_TIMEOUT as the longest possible period after which the cache is cleared and the subsequent task can run. The value should exceed the longest sync task of the instance. The default value is 3600 seconds (1 hour).

If you want to manually empty the cache and run subsequent sync task before SYNC_TASK_TIMEOUT expires, you can run the following commands from the CLI:

from django.core.cache import cache
cache.delete('serial_task.pontoon.sync.tasks.sync_project[project=11]')

If Heroku is restarting your dynos, because they run out of memory, you need to upgrade the plan.

Status: UNCONFIRMED → RESOLVED
Closed: 3 years ago
Resolution: --- → WORKSFORME
Product: Webtools → Webtools Graveyard
You need to log in before you can comment on or make changes to this bug.