1615066 - Symbolication of local Android and Linux builds does not work due to out-of-memory errors

Status: RESOLVED FIXED

(Core :: Gecko Profiler, defect, P2)

Description

[Markus Stange [:mstange]]

When profiling local builds for Android or Linux, symbolication of libxul.so often fails. That's because that file is really really big and we try to extract symbols from it using a piece of WebAssembly.
My libxul.so for Android is 1.85GB big. And a libxul.so for Linux is 2.2GB big. The WebAssembly heap is limited to 2GB, so we run out of wasm memory during symbolication.

90% of this size is due to debug information. Stripping out the debug information can work around the problem but needs to be done manually. For example with the following command:

~/.mozbuild/clang/bin/llvm-strip --no-strip-all --strip-debug /Users/mstange/code/obj-m-android-opt/toolkit/library/build/libxul.so

Comment 1

[Markus Stange [:mstange]]

Bug 1392234 might increase this limitation to 4GB soon. I'll wait for a bit more before I decide what to do here.

Comment 2

[Markus Stange [:mstange]]

Julian Seward convinced me that I really want to implement a different approach, where we only read the parts of the files that we actually need. That should both be faster and use less memory.

Comment 3

[Lee Salzman [:lsalzman]]

A potential mozconfig workaround that was suggested by Mike Hommey:

export LDFLAGS=-Wl,--compress-debug-sections=zlib

Comment 4

[Markus Stange [:mstange]]

(In reply to Markus Stange [:mstange] from comment #2)

where we only read the parts of the files that we actually need.

Implementing this doesn't seem straightforward because the object crate wants a slice to the entire file. This is in contrast to the DWARF information reader which has a Reader abstraction.

Comment 5

[Lars T Hansen [:lth]]

Not really advocating using lots of RAM, and that said: heaps > 2GB are coming but will realistically be operational 2021Q2 at the earliest.

Comment 6

[Markus Stange [:mstange]]

(In reply to Markus Stange [:mstange] from comment #4)

(In reply to Markus Stange [:mstange] from comment #2)

where we only read the parts of the files that we actually need.

Implementing this doesn't seem straightforward because the object crate wants a slice to the entire file. This is in contrast to the DWARF information reader which has a Reader abstraction.

I've filed gimli-rs/object#269 to discuss changing object's API.

Comment 7

[Markus Stange [:mstange]]

Update: Philip Craig has started implementing the proposed API in the object crate. I have a prototype which makes use of the new API and it looks really promising.

Comment 9

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

Bug 1711843 shows that this issue may also produce totally wrong garbage.

Comment 10

[Lars T Hansen [:lth]]

Update on our side: 4GB heaps are now a thing in Wasm on 64-bit systems (FF89), so the bug should probably be retitled, but see comment 5.

Comment 11

[Markus Stange [:mstange]]

So yeah, people are still seeing this despite the 4GB heap size, and I'm not sure why. Anyway, I have the partial-read approach working locally and I'm planning to finish this work soon.

Comment 12

[Julien Wajsberg [:julienw]]

(In reply to Markus Stange [:mstange] from comment #11)

So yeah, people are still seeing this despite the 4GB heap size, and I'm not sure why. Anyway, I have the partial-read approach working locally and I'm planning to finish this work soon.

Last time we looked, the unstripped libxul is much bigger than 4GB actually :/

Comment 13

[Florian Quèze [:florian]]

(In reply to Julien Wajsberg [:julienw] from comment #12)

(In reply to Markus Stange [:mstange] from comment #11)

So yeah, people are still seeing this despite the 4GB heap size, and I'm not sure why. Anyway, I have the partial-read approach working locally and I'm planning to finish this work soon.

Last time we looked, the unstripped libxul is much bigger than 4GB actually :/

On my local linux build it's 2.1GB unstripped, and 227MB stripped. We might be doing multiple string copies.

Comment 14

[Markus Stange [:mstange]]

Attachment: WIP: Bug 1615066 - WIP: Move path resolution into the worker. Should be split up more.

Comment 15

[Markus Stange [:mstange]]

Attachment: Bug 1615066 - Add support for new profiler-get-symbols version which supports partial file reading. r=canaltinova

Depends on D117937

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3c2ca00ff665
Add support for new profiler-get-symbols version which supports partial file reading. r=canaltinova

Comment 17

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/3c2ca00ff665

Comment 18

[Julien Wajsberg [:julienw]]

🥳🎉