1634205 - Support Gecko Profiler on FreeBSD

Status: RESOLVED FIXED

(Core :: Gecko Profiler, enhancement, P5)

Description

[Val Packett]

I'm working on this, WIP patch coming in a minute.

The profiling seems to work fine, but symbolication is weird, some functions are recognized as unsigned long and __float128, sometimes there's a lot of stuff that's very wrong (stacks of full of DumpBidiLine or even gtk_xtbin_new (??) in WebRender display list building).

Profile example: https://profiler.firefox.com/public/abec15c99b6aa57e1f42fe7392bd0466a3c4c990/

Comment 1

[Val Packett]

Attachment: Bug 1634205 - Support Gecko Profiler and Base Profiler on FreeBSD

Comment 2

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

Thank you!
P5 because we can't officially support FreeBSD, but patches are certainly welcome.

Comment 4

[Markus Stange [:mstange]]

Nice work!

(In reply to greg v [:myfreeweb] from comment #0)

but symbolication is weird,

ProfilerGetSymbols.jsm tried to obtain symbols from /usr/home/greg/src/hg.mozilla.org/mozilla-unified/obj-x86_64-unknown-freebsd13.0/dist/bin/libxul.so. If you list the symbols using nm, does that see all the symbols? The symbol count should be in the 6 digits.

With the following code executed from the browser console, you can check how many symbols ProfilerGetSymbols.jsm sees:

Cu.import("resource://gre/modules/ProfilerGetSymbols.jsm");
libxul = Services.profiler.sharedLibraries.find(l => l.name.includes("libxul.so"));
symbolTableAsTuple = await ProfilerGetSymbols.getSymbolTable(libxul.path, libxul.debugPath, libxul.breakpadId);
symbolTableAsTuple[0].length

Comment 5

[Markus Stange [:mstange]]

Oh, I see, your SharedLibraries code is emitting empty breakpad IDs. Then I'm not quite sure why you're getting any symbols at all. Unless you're using MOZ_PROFILER_SYMBOLICATE - that path doesn't check breakpad IDs.

Comment 6

[Val Packett]

hm, I might not have been using framepointers (?) — after rolling back mozglue/misc/StackWalk.cpp changes (to make sure HAVE__UNWIND_BACKTRACE is what's used) the profiles became much flatter o_0 (but I don't know which is more suspicious, the very deep or the rather flat stacks. also might've been caused by a different change?)

e.g. https://profiler.firefox.com/public/7a43d96b0efa79f3447e127897b59024aa407816/

Unless you're using MOZ_PROFILER_SYMBOLICATE

I am using it, yes.

Comment 7

[Val Packett]

So, turns out it's dladdr being bad — I've added debug prints to trace it:

MDCA dladdr 0x80b10e0b8 in '/usr/home/greg/src/hg.mozilla.org/mozilla-unified/obj-x86_64-unknown-freebsd13.0/dist/bin/libxul.so': 'gtk_xtbin_new'
MDCA dladdr 0x80c39e133 in '/usr/home/greg/src/hg.mozilla.org/mozilla-unified/obj-x86_64-unknown-freebsd13.0/dist/bin/libxul.so': 'XRE_GetBootstrap'
MDCA dladdr 0x80c36c7a6 in '/usr/home/greg/src/hg.mozilla.org/mozilla-unified/obj-x86_64-unknown-freebsd13.0/dist/bin/libxul.so': 'XRE_GetBootstrap'
MDCA dladdr 0x809641941 in '/usr/home/greg/src/hg.mozilla.org/mozilla-unified/obj-x86_64-unknown-freebsd13.0/dist/bin/libxul.so': 'mozilla_dump_image'

and compared to lldb:

(lldb) image lookup --address 0x80b10e0b8
      Address: libxul.so[0x000000000710e0b8] (libxul.so.PT_LOAD[1]..text + 45884664)
      Summary: libxul.so`nsFrame::GetChildLists(nsTArray<mozilla::layout::FrameChildList>*) const + 136 at nsFrame.cpp:2112:1
(lldb) image lookup --address 0x80c39e133
      Address: libxul.so[0x000000000839e133] (libxul.so.PT_LOAD[1]..text + 65348979)
      Summary: libxul.so`js::ObjectGroup::Property** js::TypeHashSet::InsertTry<JS::PropertyKey, js::ObjectGroup::Property, js::ObjectGroup::Property>(js::LifoAlloc&, js::ObjectGroup::Property**&, unsigned int&, JS::PropertyKey) + 483 at TypeInference-inl.h:941:28
(lldb) image lookup --address 0x80c36c7a6
      Address: libxul.so[0x000000000836c7a6] (libxul.so.PT_LOAD[1]..text + 65145830)
      Summary: libxul.so`js::ObjectGroup::sweep(js::AutoSweepObjectGroup const&) + 70 [inlined] js::ObjectGroup::setGeneration(unsigned int) + 3 at TypeInference.cpp:4377
               libxul.so`js::ObjectGroup::sweep(js::AutoSweepObjectGroup const&) + 67 at TypeInference.cpp:4377
(lldb) image lookup --address 0x809641941
      Address: libxul.so[0x0000000005641941] (libxul.so.PT_LOAD[1]..text + 17784193)
      Summary: libxul.so`nsContentUtils::GetDocShellForEventTarget(mozilla::dom::EventTarget*) + 1 at nsContentUtils.cpp:8927

So always ignoring dli_sname in favor of goblin-powered ELF lookup works much better:

MDCA dladdr 0x80b17d778 in '…/dist/bin/libxul.so': 'gtk_xtbin_new'
GetFunction symbolicated 0x80b17d778 as 'virtual function non-virtual override offset : -152 nsHTMLScrollFrame::SetScrollableByAPZ(bool)'

Comment 8

[Val Packett]

Now using LUL for stack walking, everything looks pretty good in general https://profiler.firefox.com/public/96459414cdf5a9acef2f9eb9a743139a387fcd03/

Still some types in the goblin-based symbolication (wchar_t here) :D

core::intrinsics::copy_nonoverlapping
core::ptr::write_unaligned
peek_poke::write_verbatim
peek_poke::poke_into_vec
webrender_api::display_list::DisplayListBuilder::push_item_to_section
webrender_api::display_list::DisplayListBuilder::push_text
wchar_t
mozilla::wr::GlyphOptions const* mozilla::wr::DisplayListBuilder::PushText(mozilla::wr::Rect<float, mozilla::wr::LayoutPixel> const&, mozilla::wr::Rect<float, mozilla::wr::LayoutPixel> const, bool, mozilla::wr::ColorF const&, mozilla::wr::FontInstanceKey, mozilla::Range<mozilla::wr::GlyphInstance const>()
…

Comment 9

[Markus Stange [:mstange]]

It would be good if you could do the BaseProfiler implementation at the same time, or at least soon afterwards. Otherwise this might delay bug 1634784, or rather, bug 1634784 would break the profiler on BSD again.

As for the empty breakpad IDs from the BSD SharedLibraries implementation, there's good news: The BaseProfiler version of shared-libraries-linux.cc is probably much easier to get to build on BSD than the GeckoProfiler version, because the BaseProfiler version does not depend on crash reporter code. Bug 1634784 will allow us to use the BaseProfiler SharedLibraries code everywhere (bug 1634785).

Comment 10

[Val Packett]

Cool, how do I test the baseprofiler?

Comment 11

[Markus Stange [:mstange]]

Uncomment this line, and see if it still builds: https://searchfox.org/mozilla-central/rev/710d6e1015d03343b067b92e6f1f775a0b1cad6f/mozglue/baseprofiler/public/BaseProfiler.h#34

If it builds, you can test it by doing startup profiling with the following env vars:
MOZ_BASE_PROFILER_STARTUP=1 MOZ_BASE_PROFILER_STARTUP_INTERVAL=5 MOZ_BASE_PROFILER_STARTUP_FEATURES=threads,stackwalk,leaf MOZ_BASE_PROFILER_STARTUP_FILTERS=GeckoMain MOZ_PROFILER_STARTUP=1 MOZ_PROFILER_STARTUP_INTERVAL=5 MOZ_PROFILER_STARTUP_FEATURES=threads,js,stackwalk,leaf MOZ_PROFILER_STARTUP_FILTERS=GeckoMain,Compositor,Renderer

Once you capture a profile, you should see a "GeckoMain (pre-XUL)" thread in the track list.

Comment 12

[Jan Beich]

Comment on attachment 9144505 [details]
Bug 1634205 - Support Gecko Profiler and Base Profiler on FreeBSD

// no breakpad on FreeBSD, only pre-symbolication

Idea for separate bug: maybe check if /usr/ports/devel/breakpad/files/patch-* can be applied to the Firefox version.

Comment 13

[Markus Stange [:mstange]]

(In reply to Markus Stange [:mstange] from comment #11)

Uncomment this line, and see if it still builds: https://searchfox.org/mozilla-central/rev/710d6e1015d03343b067b92e6f1f775a0b1cad6f/mozglue/baseprofiler/public/BaseProfiler.h#34

Actually I'm just about to land that change, bug 1634782 was more straightforward than anticipated.

Comment 14

[Val Packett]

(In reply to Jan Beich from comment #12)

Idea for separate bug: maybe check if /usr/ports/devel/breakpad/files/patch-* can be applied to the Firefox version.

huh, turns out that's very easy.

So I have dump_syms built (but not anything else under crashreporter) and the UUID code working. And the base profiler seems to work.

Trying to build the symbols file without MOZ_CRASHREPORTER..

Comment 15

[Val Packett]

Huh, wait, looking at how this symbolication works, it doesn't need crashreporter style symbols, it actually processes binaries themselves using WASM, and the only "breakpad" thing it needs are build IDs??

Well I'm hitting an unreachable in that wasm file now..

Comment 16

[Markus Stange [:mstange]]

(In reply to greg v [:myfreeweb] from comment #15)

Huh, wait, looking at how this symbolication works, it doesn't need crashreporter style symbols, it actually processes binaries themselves using WASM, and the only "breakpad" thing it needs are build IDs??

That's right!

Well I'm hitting an unreachable in that wasm file now..

That's bug 1615066 :-(

Comment 17

[Val Packett]

Okay, pushed an update with all the things. Example: https://perfht.ml/2SrHnqZ

Comment 18

[Jan Beich]

Comment on attachment 9144505 [details]
Bug 1634205 - Support Gecko Profiler and Base Profiler on FreeBSD

EDIT: Nevermind, FreeBSD 11.4 has https://svnweb.freebsd.org/changeset/base/351766, so below error is specific to 11.3.

+# if defined(GP_OS_freebsd)
+# define ElfW(type) Elf_##type
+# endif

NT_GNU_BUILD_ID is also missing on FreeBSD < 12 (11.* is supported until 2021-09-30):

mozglue/baseprofiler/core/shared-libraries-linux.cc:378:34: error: use of undeclared identifier 'NT_GNU_BUILD_ID'
      if (note_header->n_type == NT_GNU_BUILD_ID) break;
                                 ^
In file included from Unified_cpp_mozglue_baseprofiler0.cpp:137:
mozglue/baseprofiler/lul/LulElf.cpp:769:32: error: use of undeclared identifier
      'NT_GNU_BUILD_ID'
    if (note_header->n_type == NT_GNU_BUILD_ID) break;
                               ^
In file included from Unified_cpp_tools_profiler1.cpp:38:
tools/profiler/lul/LulElf.cpp:780:32: error: use of undeclared identifier 'NT_GNU_BUILD_ID'
    if (note_header->n_type == NT_GNU_BUILD_ID) break;
                               ^

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/74707e8ecf38
Support Gecko Profiler and Base Profiler on FreeBSD r=mstange

Comment 20

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/74707e8ecf38

Comment 21

[Landry Breuil (:gaston)]

jan, does this mean only openbsd will now see random build breakage due to assuming everything builds with SPS ?

Comment 22

[Jan Beich]

Solaris, DragonFly, NetBSD and FreeBSD i386/powerpc64 also lack GeckoProfiler. Assuming bug 1635350 doesn't break those and doesn't gate #include lines (to avoid bootlegging) then occasional Tier3 bustage is unlikely. Since bug 1403868 there's not much bustage, anyway.

Comment 23

[Graham Perrin]

https://svnweb.freebsd.org/ports?view=revision&revision=546279 noted with thanks,

www/firefox: document GeckoProfiler is no longer missing

– extremely cool.

Thanks, everyone!