1654815 - split up symbolstore.py into separate tasks for better parallelization

Status: NEW

(Toolkit :: Crash Reporting, enhancement)

Description

[Nathan Froyd [:froydnj]]

This is sort of akin to bug 1393436, but I think it's going in a different direction.

symbolstore.py does a lot of work for each file:

1. Dumping breakpad symbols, the thing we all know and love.
2. Adding source server indexing to the PDB file on Windows.
3. Copying debug information to separate files for inclusion into the full crashreporter symbols.
4. Printing information about static constructors for perfherder's purposes (for libxul only).

These are done serially inside symbolstore.py for reasonable things: 2, 3, and 4 need information from the breakpad symbols (MODULE and INFO lines) and information about the functions contained therein.

But now that we have a dump_syms implementation and supporting infrastructure behind it that's not terrible, there's no particular reason that we have to do these things serially. They can all be done in parallel (well, 3 can't on Linux because it modifies the original file, but we could at least do part of it); we can teach dump_syms how to do bits, or we can teach dump_syms how to extract just the information we need and then write everything else in Python.

Doing all of this would help make symbolstore.py simpler, and would also make it easier to use the source mapping functionality that's ready in dump_syms but currently unused.

Comment 1

[Nathan Froyd [:froydnj]]

I filed this bug on the basis of more parallelization is more better, but timings suggest we can either do better by splitting things up, or by using faster tools for at the very least, copying debug information:

Linux x64 debug (Android is roughly similar, about 10s faster for both):

Copied debug for /builds/worker/workspace/obj-build/toolkit/library/build/libxul.so in 62.21s
Finished processing /builds/worker/workspace/obj-build/toolkit/library/build/libxul.so in 97.91s

So dump_syms is only ~1/3 of the time here.

Mac opt:

Finished processing /builds/worker/workspace/obj-build/toolkit/library/build/XUL in 134.76s (this is running llvm-dsymutil)
Copied debug for /builds/worker/workspace/obj-build/toolkit/library/build/XUL in 167.46s (!!!)
Finished processing /builds/worker/workspace/obj-build/toolkit/library/build/XUL in 193.47s (this is the actual symbol dumping)

So dump_syms itself is barely noticeable amidst all the noise here.

Win64 opt:

Source indexing for /builds/worker/workspace/obj-build/toolkit/library/build/xul.dll in 0.04s
Copied debug for /builds/worker/workspace/obj-build/toolkit/library/build/xul.dll in 81.13s
Finished processing /builds/worker/workspace/obj-build/toolkit/library/build/xul.dll in 92.65s

Again, the dump_syms time is a small component of the total time (~1/9!)

I guess this explains why landing dump_syms didn't work as well as we thought it might -- there was still a lot of other work happening, particularly on Mac.

Comment 2

[Gabriele Svelto [:gsvelto]]

I never paid attention to the copy step. It's set from here, we use it on all platforms... and the name is highly misleading because it's not copying the debug info. It's doing all sort of processing and also compressing it.

It's using makecab on Windows; objcopy to strip the files and gzip to compress them on Linux; and finally bzip2 on macOS which probably explains why it's so painfully slow.

One step to make this faster would be bug 1396084, but that requires work on the symbol server. I'm not sure if increasing parallelism would help here though because I'm sure those steps will be dominated by libxul which dwarfs all the other executables.

Comment 3

[Nathan Froyd [:froydnj]]

glandium apparently plans to do copy + compression in upload-symbols, which would be nice.