Closed Bug 1634205 Opened 2 years ago Closed 2 years ago

Support Gecko Profiler on FreeBSD

Categories

(Core :: Gecko Profiler, enhancement, P5)

Desktop
FreeBSD
enhancement

Tracking

()

RESOLVED FIXED
mozilla78
Tracking Status
firefox78 --- fixed

People

(Reporter: unrelentingtech, Assigned: unrelentingtech)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

I'm working on this, WIP patch coming in a minute.

The profiling seems to work fine, but symbolication is weird, some functions are recognized as unsigned long and __float128, sometimes there's a lot of stuff that's very wrong (stacks of full of DumpBidiLine or even gtk_xtbin_new (??) in WebRender display list building).

Profile example: https://profiler.firefox.com/public/abec15c99b6aa57e1f42fe7392bd0466a3c4c990/

Thank you!
P5 because we can't officially support FreeBSD, but patches are certainly welcome.

Priority: -- → P5
Duplicate of this bug: 808257

Nice work!

(In reply to greg v [:myfreeweb] from comment #0)

but symbolication is weird,

ProfilerGetSymbols.jsm tried to obtain symbols from /usr/home/greg/src/hg.mozilla.org/mozilla-unified/obj-x86_64-unknown-freebsd13.0/dist/bin/libxul.so. If you list the symbols using nm, does that see all the symbols? The symbol count should be in the 6 digits.

With the following code executed from the browser console, you can check how many symbols ProfilerGetSymbols.jsm sees:

Cu.import("resource://gre/modules/ProfilerGetSymbols.jsm");
libxul = Services.profiler.sharedLibraries.find(l => l.name.includes("libxul.so"));
symbolTableAsTuple = await ProfilerGetSymbols.getSymbolTable(libxul.path, libxul.debugPath, libxul.breakpadId);
symbolTableAsTuple[0].length

Oh, I see, your SharedLibraries code is emitting empty breakpad IDs. Then I'm not quite sure why you're getting any symbols at all. Unless you're using MOZ_PROFILER_SYMBOLICATE - that path doesn't check breakpad IDs.

hm, I might not have been using framepointers (?) — after rolling back mozglue/misc/StackWalk.cpp changes (to make sure HAVE__UNWIND_BACKTRACE is what's used) the profiles became much flatter o_0 (but I don't know which is more suspicious, the very deep or the rather flat stacks. also might've been caused by a different change?)

e.g. https://profiler.firefox.com/public/7a43d96b0efa79f3447e127897b59024aa407816/

Unless you're using MOZ_PROFILER_SYMBOLICATE

I am using it, yes.

So, turns out it's dladdr being bad — I've added debug prints to trace it:

MDCA dladdr 0x80b10e0b8 in '/usr/home/greg/src/hg.mozilla.org/mozilla-unified/obj-x86_64-unknown-freebsd13.0/dist/bin/libxul.so': 'gtk_xtbin_new'
MDCA dladdr 0x80c39e133 in '/usr/home/greg/src/hg.mozilla.org/mozilla-unified/obj-x86_64-unknown-freebsd13.0/dist/bin/libxul.so': 'XRE_GetBootstrap'
MDCA dladdr 0x80c36c7a6 in '/usr/home/greg/src/hg.mozilla.org/mozilla-unified/obj-x86_64-unknown-freebsd13.0/dist/bin/libxul.so': 'XRE_GetBootstrap'
MDCA dladdr 0x809641941 in '/usr/home/greg/src/hg.mozilla.org/mozilla-unified/obj-x86_64-unknown-freebsd13.0/dist/bin/libxul.so': 'mozilla_dump_image'

and compared to lldb:

(lldb) image lookup --address 0x80b10e0b8
      Address: libxul.so[0x000000000710e0b8] (libxul.so.PT_LOAD[1]..text + 45884664)
      Summary: libxul.so`nsFrame::GetChildLists(nsTArray<mozilla::layout::FrameChildList>*) const + 136 at nsFrame.cpp:2112:1
(lldb) image lookup --address 0x80c39e133
      Address: libxul.so[0x000000000839e133] (libxul.so.PT_LOAD[1]..text + 65348979)
      Summary: libxul.so`js::ObjectGroup::Property** js::TypeHashSet::InsertTry<JS::PropertyKey, js::ObjectGroup::Property, js::ObjectGroup::Property>(js::LifoAlloc&, js::ObjectGroup::Property**&, unsigned int&, JS::PropertyKey) + 483 at TypeInference-inl.h:941:28
(lldb) image lookup --address 0x80c36c7a6
      Address: libxul.so[0x000000000836c7a6] (libxul.so.PT_LOAD[1]..text + 65145830)
      Summary: libxul.so`js::ObjectGroup::sweep(js::AutoSweepObjectGroup const&) + 70 [inlined] js::ObjectGroup::setGeneration(unsigned int) + 3 at TypeInference.cpp:4377
               libxul.so`js::ObjectGroup::sweep(js::AutoSweepObjectGroup const&) + 67 at TypeInference.cpp:4377
(lldb) image lookup --address 0x809641941
      Address: libxul.so[0x0000000005641941] (libxul.so.PT_LOAD[1]..text + 17784193)
      Summary: libxul.so`nsContentUtils::GetDocShellForEventTarget(mozilla::dom::EventTarget*) + 1 at nsContentUtils.cpp:8927

So always ignoring dli_sname in favor of goblin-powered ELF lookup works much better:

MDCA dladdr 0x80b17d778 in '…/dist/bin/libxul.so': 'gtk_xtbin_new'
GetFunction symbolicated 0x80b17d778 as 'virtual function non-virtual override offset : -152 nsHTMLScrollFrame::SetScrollableByAPZ(bool)'

Now using LUL for stack walking, everything looks pretty good in general https://profiler.firefox.com/public/96459414cdf5a9acef2f9eb9a743139a387fcd03/

Still some types in the goblin-based symbolication (wchar_t here) :D

core::intrinsics::copy_nonoverlapping
core::ptr::write_unaligned
peek_poke::write_verbatim
peek_poke::poke_into_vec
webrender_api::display_list::DisplayListBuilder::push_item_to_section
webrender_api::display_list::DisplayListBuilder::push_text
wchar_t
mozilla::wr::GlyphOptions const* mozilla::wr::DisplayListBuilder::PushText(mozilla::wr::Rect<float, mozilla::wr::LayoutPixel> const&, mozilla::wr::Rect<float, mozilla::wr::LayoutPixel> const, bool, mozilla::wr::ColorF const&, mozilla::wr::FontInstanceKey, mozilla::Range<mozilla::wr::GlyphInstance const>()
…
Assignee: nobody → greg
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true

It would be good if you could do the BaseProfiler implementation at the same time, or at least soon afterwards. Otherwise this might delay bug 1634784, or rather, bug 1634784 would break the profiler on BSD again.

As for the empty breakpad IDs from the BSD SharedLibraries implementation, there's good news: The BaseProfiler version of shared-libraries-linux.cc is probably much easier to get to build on BSD than the GeckoProfiler version, because the BaseProfiler version does not depend on crash reporter code. Bug 1634784 will allow us to use the BaseProfiler SharedLibraries code everywhere (bug 1634785).

Cool, how do I test the baseprofiler?

Uncomment this line, and see if it still builds: https://searchfox.org/mozilla-central/rev/710d6e1015d03343b067b92e6f1f775a0b1cad6f/mozglue/baseprofiler/public/BaseProfiler.h#34

If it builds, you can test it by doing startup profiling with the following env vars:
MOZ_BASE_PROFILER_STARTUP=1 MOZ_BASE_PROFILER_STARTUP_INTERVAL=5 MOZ_BASE_PROFILER_STARTUP_FEATURES=threads,stackwalk,leaf MOZ_BASE_PROFILER_STARTUP_FILTERS=GeckoMain MOZ_PROFILER_STARTUP=1 MOZ_PROFILER_STARTUP_INTERVAL=5 MOZ_PROFILER_STARTUP_FEATURES=threads,js,stackwalk,leaf MOZ_PROFILER_STARTUP_FILTERS=GeckoMain,Compositor,Renderer

Once you capture a profile, you should see a "GeckoMain (pre-XUL)" thread in the track list.

Comment on attachment 9144505 [details]
Bug 1634205 - Support Gecko Profiler and Base Profiler on FreeBSD

// no breakpad on FreeBSD, only pre-symbolication

Idea for separate bug: maybe check if /usr/ports/devel/breakpad/files/patch-* can be applied to the Firefox version.

(In reply to Markus Stange [:mstange] from comment #11)

Uncomment this line, and see if it still builds: https://searchfox.org/mozilla-central/rev/710d6e1015d03343b067b92e6f1f775a0b1cad6f/mozglue/baseprofiler/public/BaseProfiler.h#34

Actually I'm just about to land that change, bug 1634782 was more straightforward than anticipated.

(In reply to Jan Beich from comment #12)

Idea for separate bug: maybe check if /usr/ports/devel/breakpad/files/patch-* can be applied to the Firefox version.

huh, turns out that's very easy.

So I have dump_syms built (but not anything else under crashreporter) and the UUID code working. And the base profiler seems to work.

Trying to build the symbols file without MOZ_CRASHREPORTER..

Huh, wait, looking at how this symbolication works, it doesn't need crashreporter style symbols, it actually processes binaries themselves using WASM, and the only "breakpad" thing it needs are build IDs??

Well I'm hitting an unreachable in that wasm file now..

(In reply to greg v [:myfreeweb] from comment #15)

Huh, wait, looking at how this symbolication works, it doesn't need crashreporter style symbols, it actually processes binaries themselves using WASM, and the only "breakpad" thing it needs are build IDs??

That's right!

Well I'm hitting an unreachable in that wasm file now..

That's bug 1615066 :-(

Attachment #9144505 - Attachment description: Bug 1634205 - Support Gecko Profiler on FreeBSD → Bug 1634205 - Support Gecko Profiler and Base Profiler on FreeBSD

Okay, pushed an update with all the things. Example: https://perfht.ml/2SrHnqZ

Pushed by mstange@themasta.com:
https://hg.mozilla.org/integration/autoland/rev/74707e8ecf38
Support Gecko Profiler and Base Profiler on FreeBSD r=mstange
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla78

jan, does this mean only openbsd will now see random build breakage due to assuming everything builds with SPS ?

Flags: needinfo?(jbeich)

Solaris, DragonFly, NetBSD and FreeBSD i386/powerpc64 also lack GeckoProfiler. Assuming bug 1635350 doesn't break those and doesn't gate #include lines (to avoid bootlegging) then occasional Tier3 bustage is unlikely. Since bug 1403868 there's not much bustage, anyway.

Flags: needinfo?(jbeich)

https://svnweb.freebsd.org/ports?view=revision&revision=546279 noted with thanks,

www/firefox: document GeckoProfiler is no longer missing

extremely cool.

Thanks, everyone!

You need to log in before you can comment on or make changes to this bug.