Closed Bug 1504101 Opened 4 years ago Closed 3 years ago

Add UI to the performance pane that lets the user pick an objdir, and then look up symbols from .sym files in it

Categories

(DevTools :: Performance Tools (Profiler/Timeline), enhancement, P1)

enhancement

Tracking

(firefox67 fixed)

RESOLVED FIXED
Firefox 67
Tracking Status
firefox67 --- fixed

People

(Reporter: mstange, Assigned: mstange)

References

(Blocks 1 open bug)

Details

Attachments

(5 files, 1 obsolete file)

The profiler currently cannot symbolicate profiles captured on local Android builds.

This has two reasons:
 - We have code in the Gecko Android binary which can dump an entire symbol table, but it can only do that for binaries that exist in an uncompressed form on the file system on the Android device. So it works for system libraries and for libmozglue.so, but not for libxul.so, because libxul.so is only stored in the apk in compressed form.
 - The uncompressed binaries exist in the objdir on the developer's host machine where this build was compiled. But the profiler does not know where that objdir is.

If we add some UI to the performance panel to let the developer select an objdir, we can use that objdir to look up symbol information.

The objdir contains two potential sources of symbol information:
 - If the developer has run "mach buildsymbols", then the objdir contains text files with the extension .sym. We have existing code that knows how to parse those .sym files and would just need to hook it up.
 - The objdir also contains the raw binaries. We currently do not have any code in the profiler that knows how to dump symbols from those. We can add that in a follow-up bug.

Let's use this bug to implement the UI to pick the objdir.

Proposed UI:

> Local build
> -----------
>
> When profiling a build that you compiled locally, symbol information needs
> to be looked up from the build's object directory.
> 
> Known object directories:
> +---------------------------+
> | [F] obj-m-android-opt     |
> | [F] obj-fxr-opt           |
> |                           |
> +---------------------------+
>   [-] Remove     [+] Add

(where [F] is the OS icon for a folder)
I had WIP patch to analyze local issue.

Android's sym file may be too big, ParseBreakpadSymbols-worker.js causes out of memory error when parsing Android's libxul.sym.
Can you attach your patch? I'm planning to work on this bug soon.
Flags: needinfo?(m_kato)
(In reply to Makoto Kato [:m_kato] from comment #1)
> Android's sym file may be too big, ParseBreakpadSymbols-worker.js causes out
> of memory error when parsing Android's libxul.sym.

I filed https://github.com/devtools-html/perf.html/issues/1516 about adding an API on the perf.html/WebIDE boundary that allows us to dump only parts of the symbol table.
Attached patch WIP (obsolete) — Splinter Review
This was old patch when I used it. So you cannot apply this in current m-c simply.

Since sym file of libxul.so is too large, when I use it on Android, I remove a lot of lines that isn't function symbol by python.
Flags: needinfo?(m_kato)
(In reply to Markus Stange [:mstange] from comment #0)
> The profiler currently cannot symbolicate profiles captured on local Android
> builds.
> 
> This has two reasons:
>  - We have code in the Gecko Android binary which can dump an entire symbol
> table, but it can only do that for binaries that exist in an uncompressed
> form on the file system on the Android device. So it works for system
> libraries and for libmozglue.so, but not for libxul.so, because libxul.so is

We might remove compress packages by bug 1486524.
This adds a pref called devtools.performance.recording.objdirs that is set to a
JSON-ified array of strings of objdir paths on the host machine that should be
consulted during symbolication.
I haven't added the actual UI yet, but if you apply the patches from bug 1509549 and this patch, and then manually set the pref devtools.performance.recording.objdirs to '["/path/to/your/objdir"]', then symbolication should work for local builds.
Attachment #9027798 - Attachment description: Bug 1504101 - WIP: Symbolicate from the specified objdirs, in order to facilitate profiling of local Firefox for Android builds. → Bug 1504101 - Add UI to the performance pane that lets the user pick an objdir for local builds.
Assignee: nobody → mstange
Status: NEW → ASSIGNED
Attached image screenshot
Attachment #9026674 - Attachment is obsolete: true
Attachment #9027798 - Attachment description: Bug 1504101 - Add UI to the performance pane that lets the user pick an objdir for local builds. → Bug 1504101 - Add UI to the performance pane that lets the user pick an objdir for local builds. r?julienw
Depends on: 1509549
(In reply to Makoto Kato [:m_kato] from comment #5)
> (In reply to Markus Stange [:mstange] from comment #0)
> > The profiler currently cannot symbolicate profiles captured on local Android
> > builds.
> > 
> > This has two reasons:
> >  - We have code in the Gecko Android binary which can dump an entire symbol
> > table, but it can only do that for binaries that exist in an uncompressed
> > form on the file system on the Android device. So it works for system
> > libraries and for libmozglue.so, but not for libxul.so, because libxul.so is
> 
> We might remove compress packages by bug 1486524.

This has landed now, and points out a flaw in the current patch: The current patch requests symbols from the device first, and from the objdir second. And now that we get symbol tables for libxul.so from the device, the profile looks like this: http://bit.ly/2DOwDMI
So it looks like the libxul.so that's on the device is stripped of most symbol information; it probably only has the dynamic symbols that are needed for dynamic linking and nothing else.
So I'll need to change the patch to consult the objdirs first, and only then call out to the debuggee.
Here's a profile from the same build with symbols from the objdir: http://bit.ly/2DQ4wwy
Component: WebIDE → Performance Tools (Profiler/Timeline)
This gives the new devtools performance panel the same symbolication
capabilities as the Gecko Profiler add-on.
Priority: -- → P1
Pushed by mstange@themasta.com:
https://hg.mozilla.org/integration/autoland/rev/f40b30f80cb1
Use the ProfilerGetSymbols module for symbolicating profiles captured using the perf panel. r=julienw
https://hg.mozilla.org/integration/autoland/rev/eb8643ae792e
Add UI to the performance pane that lets the user pick an objdir for local builds. r=julienw
https://hg.mozilla.org/integration/autoland/rev/69e008db7e1e
When symbolicating profiles from remote targets, consult objdirs for symbol information. r=julienw
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 67
You need to log in before you can comment on or make changes to this bug.