867721 - Build b2g Nightly to support profiling with native stacks

Status: RESOLVED WONTFIX

(Core :: Gecko Profiler, defect)

Description

[Benoit Girard (:BenWa)]

+++ This bug was initially created as a clone of Bug #863264 +++

B2G nightlies don't support Breakpad unwinding.  We need to
enable that. We need to do something similar to bug 863264.

Comment 1

[Benoit Girard (:BenWa)]

Another option is to deploy jld's custom toolchain to the build machines and use it for enable-profiling nightly builds.

How feasible is that?

Comment 2

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

Do we want this for profiling or for crash reporting?  Are we using Breakpad unwinding on B2G, or should this bug be retitled?  Note that my change to NS_StackWalk to work with my altered toolchain isn't landed yet (nor are any of the pieces for perf(1) profiling, which also depends on the unlandable toolchain changes); also, the compiler flags would have to be set manually.

Comment 3

[Julian Seward [:jseward]]

AFAIK this is for profiling, at least.  Whether it's for crash reporting too,
I don't know.

Comment 4

[(not currently active) Ted Mielczarek]

This isn't something we need for crash reporting. I don't know if Breakpad will use frame pointers properly with -fno-omit-framepointer on arm given the fact that we build everything for thumb.

Jed: we do have a Breakpad-powered unwinder in the profiler now, but it's not on by default.

Comment 5

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

As mentioned on bug 863264, as far as I can tell -fno-omit-frame-pointer on Thumb isn't helpful: it sets r7 to a value that isn't at a constant offset from the previous frame pointer and the saved LR, so it still requires interpreting PC-indexed unwinding information.  The same thing could be accomplished with -fomit-frame-pointer using SP.

(That's Thumb on arm-linux-androideabi.  iOS does things differently: http://developer.apple.com/library/ios/#documentation/Xcode/Conceptual/iPhoneOSABIReference/Articles/ARMv6FunctionCallingConventions.html but that doesn't help us.)

Comment 6

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

I think I want some performance numbers on FramePointerStackWalk vs. breakpad with -mthumb2-fake-apcs-frame vs. breakpad with just -fasynchronous-unwind-tables.

(I'm also slightly curious about a hypothetical allocation-free EHABI unwinder that could be async-signal-safe without stack copying — which I think is how breakpad works currently? — but I'd have to write it first.)

For reference, I already have numbers for the kernel-based profiler with -mthumb2-fake-apcs-frame: 3% of an unagi at 2.5 kHz, rising to 7% with 100 added stack frames.

Comment 7

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

However: 4182c5c0b, the change that adds breakpad, hasn't been uplifted to b2g18.  This is a problem.

Comment 8

[Benoit Girard (:BenWa)]

Sorry for the confusion. For this bug we have two options (1) Compile with the info breakpad requires to unwind with b2g (CFI/EXIDX) or (2) Use your unwind method by compiling with a custom toolchain.

I think right now our default is to proceed with (1) but I wanted to know how much work (2) would be if it's feasible at all say we install your custom toolchain and specify your custom toolchain in nightly mozconfigs when producing the nightly builds

Comment 9

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

The breakpad on m-c understands only CFI.  I managed to get the .debug_frame sections for gonk and gecko onto my hamachi, and the result is that breakpad uses too much memory for Homescreen to start without crashing.

I'm told there's ARM EHABI support for the breakpad unwinder in an uncommitted state; I don't know whether or not we'd expect it to have substantially less memory usage.

Probably the fastest way forward is to finish bug 810526, get the altered toolchain onto the official build hosts (how?), and adjust mozconfigs accordingly.  That gives native stacks to the tools everyone is used to from other Geckos.

Bug 831611 needs some more thought and cleanup before parts of it start being landed.

Comment 10

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

Attachment: The patches from my experiment in comment #9

Some of this is outright uncommittable as-is (adding very b2g-specific code to what should be generic), but it worked enough for my test.

Also, "export MOZ_BREAKPAD=1" in .userconfig.  The profile.sh line is set to native instead of combined because I wanted to see what the native tracer was doing (before I got it working) without the pseudostacks in the way.

Comment 11

[Benoit Girard (:BenWa)]

(In reply to Jed Davis [:jld] from comment #9)
> Probably the fastest way forward is to finish bug 810526, get the altered
> toolchain onto the official build hosts (how?)

We can put a request with releng with a package toolchain if we decide to do that. Shall we wait for the results on android before attempting to cut b2g nightly off the custom toolchain?

Comment 12

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

I assume we want this bug to apply to b2g18 builds as well as m-c?

Comment 13

[Benoit Girard (:BenWa)]

(In reply to Jed Davis [:jld] from comment #12)
> I assume we want this bug to apply to b2g18 builds as well as m-c?

I was under the impression that the b2g18 landing were fairly selective and this bug may be invasive (changing build option plus either use a custom toolchain or breakpad). But I'd be happy to be wrong on this one.

At the very least we could make a profiling nightly fork.

Comment 14

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

(In reply to Benoit Girard (:BenWa) from comment #13)
> (In reply to Jed Davis [:jld] from comment #12)
> > I assume we want this bug to apply to b2g18 builds as well as m-c?
> 
> I was under the impression that the b2g18 landing were fairly selective and
> this bug may be invasive (changing build option plus either use a custom
> toolchain or breakpad). But I'd be happy to be wrong on this one.

There shouldn't be anything here that would affect a non-profiling build of b2g, so I hope that uplift approval wouldn't be difficult.  I'm under the impression that b2g18 is going to be where a lot of the performance attention will be focused for at least the immediate future, so (if I'm right about that) it would be good to include it.

Comment 15

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

Current status: When bug 810526 lands, it should suffice to uncomment the "ac_add_options --enable-profiling" from gecko/b2g/config/mozconfigs/ics_armv7a_gecko/nightly, if I understand the mozconfig system correctly.