Status

enhancement
RESOLVED FIXED
6 months ago
6 months ago

People

(Reporter: glandium, Assigned: glandium)

Tracking

unspecified
mozilla66

Firefox Tracking Flags

(firefox66 fixed)

Details

Attachments

(1 attachment)

Assignee

Description

6 months ago
No description provided.
Assignee

Comment 1

6 months ago
Firefox uses multiple processes. It has intentional leaks, and when
running with ASAN, we have suppressions to eliminate those. When running
ASAN builds through CI tests, when Firefox exits, each of the processes
(parent and child) exits and goes through its leaks and when there are
(which is a given), the ASAN runtime runs llvm-symbolizer to symbolicate
and match against suppressions. So each process runs llvm-symbolizer. At
the same time.

Some of the addresses to symbolicate are in libxul. Which contains all
DWARF info, making it a ~1GB monster. Oh, and because you're lucky,
things align perfectly such that libxul size is a multiple of the page
size. That makes llvm-symbolizer pread() the file instead of mmap()ing
it. Did I say there are multiple processes? So suddenly you have n
processes simultaneously allocating and filling 1GB of memory each, on
CI machines that have enough memory for the job they usually run, but
not enough for a sudden rush of n GB.

And things go awry. When you're lucky and the OOM killer didn't take
care of killing the CI entirely, symbolication couldn't happen and the
suppressions are not matched, and leaks are reported.

This all turns out it originates in how llvm-symbolicate chooses between
pread() and mmap(), which turns out is just defaults not being made for
binary files.
Assignee

Updated

6 months ago
No longer blocks: 1516374

That's quite the edge case!

Comment 4

6 months ago
bugherder
Status: NEW → RESOLVED
Closed: 6 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla66
You need to log in before you can comment on or make changes to this bug.