Code Coverage aggregation task is OOMing
Categories
(Testing :: Code Coverage, defect)
Tracking
(Not tracked)
People
(Reporter: nicolas.guichard, Assigned: nicolas.guichard)
References
(Blocks 1 open bug, Regression)
Details
(Keywords: regression)
Attachments
(2 external files)
Code coverage reports stopped flowing to the GCP bucket and Searchfox around two weeks ago (last reports Searchfox shows is for git 17e93ad5d3261e20104c7f6f2ec867ecc138ca1a ie 2026-08-20).
Logs of the coverage aggregation task suggests it gets killed for using too much memory:
- “Peak System Memory Used: 14.76 GiB” vs “Total System Memory: 15.61 GiB”
- “
grcovfailed with code: -9” where -9 == -SIGKILL
It would probably be good to try and reduce the grcov memory usage, but I think the simple short-term fix is to increase the worker's memory.
| Assignee | ||
Comment 1•2 months ago
|
||
:marco Do you agree that increasing the available memory is the right call here (at least for now)?
Also, do you know where that knob is?
Comment 2•2 months ago
|
||
Yes, sounds OK to me.
The config can be changed in https://github.com/mozilla-releng/fxci-config/blob/bb7b82a367d9e7d972a33113431e57d0eb3b5f6a/worker-pools.yml#L3619-L3643. However, we only want to do it for code-coverage and not for the others. I'm not sure if we can do that in the same yaml entry or if we have to split the code coverage entry out.
Comment 3•2 months ago
|
||
Should be possible to key machine_type by-pool-group.
What are the chances it's something similar to bug 1925873 where it'd OOM no matter how much ram we gave it though?
Comment 4•2 months ago
|
||
Comment 5•2 months ago
|
||
Authored by https://github.com/jcristau
https://github.com/mozilla-releng/fxci-config/commit/b209fbb3aa7809a7ae6e805ac895d15e655377d9
[main] Bug 2070106 - bump code-coverage machine type to c2-standard-8 (#1205)
Comment 6•2 months ago
|
||
Should have set leave-open here.
Comment 7•2 months ago
|
||
Looks like things are back to green, now using 26GiB instead of <16.
Updated•2 months ago
|
| Assignee | ||
Comment 8•4 days ago
|
||
The new Android coverage data from bug 2077159 appears to have bumped the memory usage again, so we're OOMing again.
- 4211a1ce1da294805573e629e591118237a5e55b peaked at 28.65 GiB
- 1137460458364499c6788a3f8ac9836a086bb04b peaked at 30.12 GiB and was killed
The amount of data hasn't changed, we only have less 0s.
I'll look into profiling and reducing grcov's memory usage now.
Comment 9•17 hours ago
|
||
(via GitHub webhook, authenticated as shared-secret)
Description
•