If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

Limit tasks that validator worker can execute

RESOLVED FIXED

Status

Cloud Services
Operations: Marketplace
RESOLVED FIXED
6 years ago
3 years ago

People

(Reporter: kumar, Assigned: jason)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

It looks like celery's task dispatcher is introducing some recursion and thus, over time, we are running out of stacks (Python's limit is 1000). See bug 735473

As a quick fix, perhaps we can just limit the number of tasks a validator worker executes before it is replaced by a new process.

We can set:
CELERYD_MAX_TASKS_PER_CHILD
or use --maxtasksperchild when starting the worker

http://celery.readthedocs.org/en/latest/userguide/workers.html#max-tasks-per-child-setting
(Assignee)

Updated

6 years ago
Component: Server Operations: Web Operations → Server Operations: AMO Operations
QA Contact: cshields → oremj
(Assignee)

Updated

6 years ago
Assignee: server-ops → jthomas
(Assignee)

Comment 1

6 years ago
Added --maxtasksperchild=50 to devhub workers. Hopefully this will help.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Component: Server Operations: AMO Operations → Operations: Marketplace
Product: mozilla.org → Mozilla Services
You need to log in before you can comment on or make changes to this bug.