Closed Bug 1248004 Opened 8 years ago Closed 8 years ago

High memory usage in Heroku async task workers

Categories

(developer.mozilla.org Graveyard :: BrowserCompat, defect)

Product:
developer.mozilla.org Graveyard
Component:
BrowserCompat
Platform:
All
Other
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jwhitlock, Unassigned, Mentored)

References

Details

(Keywords: in-triage, Whiteboard: [specification][type:bug][bc:infra][bc:milestone=scooter]) 

What did you do?
================
Ran the import_mdn.py tool, which provisions many async tasks

What happened?
==============
In the logs, warnings of the memory quota exceeded, such as:

2016-02-12T17:53:06.909996+00:00 heroku[worker.1]: Process running mem=619M(120.9%)
2016-02-12T17:53:06.909996+00:00 heroku[worker.1]: Error R14 (Memory quota exceeded)

The quota appears to be 512 MB. Memory usage remained high until the worker was restarted.

What should have happened?
==========================
Memory usage should stay within quota


Is there anything else we should know?
======================================
An online article from 2013 (and my previous celery experience) says that this is known Python 2 behavior, mitigated by restarting task workers periodically, or possibly by switching to Python 3:

https://dzone.com/articles/diagnosing-memory-leaks-python
Blocks: 1240757
Mentor: jwhitlock
Whiteboard: [specification][type:bug] → [specification][type:bug][bc:infra][bc:milestone=scooter]
Blocks: 1243225
No longer blocks: 1240757
The work on bug 1247942 included switching Heroku to Python 3 and limiting the worker concurrency, which also fixed this issue.

https://github.com/mdn/browsercompat/pull/106
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Product: developer.mozilla.org → developer.mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.