test-linux1804-64-asan-qr/opt-mochitest-remote jobs frequently run out of memory
Categories
(Remote Protocol :: Agent, defect, P3)
Tracking
(firefox128 fixed, firefox129 fixed)
People
(Reporter: jcristau, Assigned: jcristau)
References
(Blocks 1 open bug)
Details
(Keywords: intermittent-failure, Whiteboard: [webdriver:m11][webdriver:external])
Attachments
(1 file)
Per https://bugzilla.mozilla.org/show_bug.cgi?id=1759288#c209, there are frequent failures of the mochitest-remote tests on linux ASAN, where the workers drop off the net part way through the run. Looking at one that did run to completion (https://share.firefox.dev/3VCRxG1, from https://treeherder.mozilla.org/jobs?repo=autoland&revision=ff34de8a49b079ff98eb45e3243575e2a4d5646b&selectedTaskRun=XztS7P3kTKCc_UxO6IEjdg.3), it seems likely the worker is running OOM and sometimes crashing.
Can something be changed to use less memory here? If not, these tests should probably run on bigger instances.
Assignee | ||
Updated•4 months ago
|
Assignee | ||
Comment 1•4 months ago
|
||
Match the instance type used by the mochitest-devtools-chrome.
Comment 2•4 months ago
|
||
Interesting. As it looks like the messagehandler browser chrome tests specifically cause such an increased memory usage. Julian, is that only visible on Linux or do Windows ASAN builds show similar behavior?
Before we bump the instance type we probably should indeed check what's causing the high memory usage. Maybe running these tests locally with the gecko profiler active could give some insights.
Assignee | ||
Comment 3•4 months ago
|
||
https://profiler.firefox.com/from-url/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FeI77sO4MRTihi63mLLOZMQ%2Fruns%2F0%2Fartifacts%2Fpublic%2Ftest_info%2Fprofile_resource-usage.json/marker-chart/?globalTrackOrder=0&thread=0&timelineType=stack&v=10 is from https://treeherder.mozilla.org/jobs?repo=autoland&revision=ff34de8a49b079ff98eb45e3243575e2a4d5646b&searchStr=asan%2Cremote&selectedTaskRun=eI77sO4MRTihi63mLLOZMQ.0
Looks like it peaks at 7GB memory usage.
Assignee | ||
Comment 4•4 months ago
|
||
I guess worth noting the windows job is running with 16GB ram already.
Comment 5•4 months ago
|
||
I see. So then lets go ahead and update the instance type so it matches other browser chrome test suites.
Updated•4 months ago
|
Updated•4 months ago
|
Updated•4 months ago
|
Comment 8•4 months ago
|
||
bugherder |
Comment hidden (Intermittent Failures Robot) |
Updated•3 months ago
|
Description
•