Open Bug 2070106 Opened 2 months ago Updated 17 hours ago

Code Coverage aggregation task is OOMing

Categories

(Testing :: Code Coverage, defect)

defect

Tracking

(Not tracked)

REOPENED

People

(Reporter: nicolas.guichard, Assigned: nicolas.guichard)

References

(Blocks 1 open bug, Regression)

Details

(Keywords: regression)

Attachments

(2 external files)

Code coverage reports stopped flowing to the GCP bucket and Searchfox around two weeks ago (last reports Searchfox shows is for git 17e93ad5d3261e20104c7f6f2ec867ecc138ca1a ie 2026-08-20).

Logs of the coverage aggregation task suggests it gets killed for using too much memory:

  • “Peak System Memory Used: 14.76 GiB” vs “Total System Memory: 15.61 GiB”
  • “grcov failed with code: -9” where -9 == -SIGKILL

It would probably be good to try and reduce the grcov memory usage, but I think the simple short-term fix is to increase the worker's memory.

:marco Do you agree that increasing the available memory is the right call here (at least for now)?
Also, do you know where that knob is?

Flags: needinfo?(mcastelluccio)

Yes, sounds OK to me.
The config can be changed in https://github.com/mozilla-releng/fxci-config/blob/bb7b82a367d9e7d972a33113431e57d0eb3b5f6a/worker-pools.yml#L3619-L3643. However, we only want to do it for code-coverage and not for the others. I'm not sure if we can do that in the same yaml entry or if we have to split the code coverage entry out.

Flags: needinfo?(mcastelluccio) → needinfo?(jcristau)

Should be possible to key machine_type by-pool-group.

What are the chances it's something similar to bug 1925873 where it'd OOM no matter how much ram we gave it though?

Flags: needinfo?(jcristau) → needinfo?(mcastelluccio)
See Also: → 1925873
Status: NEW → RESOLVED
Closed: 2 months ago
Flags: qe-verify+
Resolution: --- → FIXED

Should have set leave-open here.

Status: RESOLVED → REOPENED
Resolution: FIXED → ---

Looks like things are back to green, now using 26GiB instead of <16.

Status: REOPENED → RESOLVED
Closed: 2 months ago → 2 months ago
Flags: needinfo?(mcastelluccio)
Resolution: --- → FIXED
Assignee: nobody → jcristau

The new Android coverage data from bug 2077159 appears to have bumped the memory usage again, so we're OOMing again.

The amount of data hasn't changed, we only have less 0s.

I'll look into profiling and reducing grcov's memory usage now.

Assignee: jcristau → nicolas.guichard
Status: RESOLVED → REOPENED
Keywords: regression
Regressed by: 2077159
Resolution: FIXED → ---
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: