Closed Bug 812070 Opened 8 years ago Closed 8 years ago

Greatly speed up fix-linux-stack.pl

Categories

(Core :: General, defect)

x86_64
Linux
defect
Not set
normal

Tracking

()

RESOLVED FIXED
mozilla19

People

(Reporter: njn, Assigned: njn)

References

Details

Attachments

(2 files)

fix-linux-stack.pl is amazingly slow.  This makes using NewDMD (bug 717853) harder, because it leans heavily on that script.  I have a big improvement coming shortly.

(BTW, I have no idea what the correct product/component is for this bug.)
This patch does some preliminary rearrangement.

We currently have %pipes and %address_adjustment, both of which hold per-file
info.  This patch just combines them into a single hash called %file_infos.
Attachment #681852 - Flags: review?(dbaron)
This patch adds a table for each file, which saves the results of previous
queries.  This avoids many calls to addr2line, which are slow.

On a 94,240 line DMD output file, this reduced the processing time on my
machine from 18.5 minutes to 2 minutes.  On a truncated 4,000 line file, it
reduced the processing time from 65 seconds to 27 seconds.
Attachment #681853 - Flags: review?(dbaron)
I wonder if a similar improvement could be made for the OSX script.
> This avoids many calls to addr2line, which are slow.

We have just one addr2line process, which we pipe all requests to, right?

I ask because that's much, much faster than launching a new process for each request.  But I don't want to read perl to figure out what we do.  :)
> We have just one addr2line process, which we pipe all requests to, right?

We have one per library/executable file (libxul.so, firefox, libstdc++.so, etc).  On my examples there are typically about 20 distinct files.

Upon further investigation it's clear that the piped requests are accounting for almost all the execution time.  Even in piped mode, addr2line is slow.  The very first lookup in libxul.so takes about 8 seconds.  For my 4,000 line example the cache takes care of 2,131 cases, and I end up doing 1,112 piped requests, each of which take 24ms on average.  Even if I ignore the 8 second start-up its about 17ms per request.  We're talking well under 100 requests per second -- slooow.

Then again, libxul.so is over 650 MiB on this machine when built with debugging info.  If I strip the debugging info (but keep the symbols) the processing time for a 4,000 line file drops from 27 seconds to 5 seconds, but then I don't get filenames and line numbers, which is much less useful.

Turning off demangling doesn't speed things up at all, which surprised me.
> We have one per library/executable file (libxul.so, firefox, libstdc++.so, etc).  On my examples 
> there are typically about 20 distinct files.

Okay.  Since addr2line is a pure function, you could run multiple processes in parallel on the same file.
Comment on attachment 681852 [details] [diff] [review]
(part 1) - Merge %pipes and %address_adjustment in fix-linux-stack.pl.

r=dbaron
Attachment #681852 - Flags: review?(dbaron) → review+
Comment on attachment 681853 [details] [diff] [review]
(part 2) - Speed up fix-linux-stack.pl by adding a cache for previously queried addresses.

Maybe change address2 to adj_address, or even skip the variable entirely and pass the whole thing to the printf call?

r=dbaronx
Attachment #681853 - Flags: review?(dbaron) → review+
> I wonder if a similar improvement could be made for the OSX script.

fix_macosx_stack.py is much faster.  It processed a 100,000 line file in just over 30 seconds, which is about 4x faster than the improved fix-linux-stack.pl.  It uses otool, atos and c++filt, and I guess they're just much faster than addr2line.
https://hg.mozilla.org/mozilla-central/rev/a702c73d82a6
https://hg.mozilla.org/mozilla-central/rev/e72aeaefb691
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla19
You need to log in before you can comment on or make changes to this bug.