compress pdb files in symbol store

RESOLVED FIXED

Status

()

Toolkit
Crash Reporting
RESOLVED FIXED
11 years ago
9 years ago

People

(Reporter: ted, Assigned: ted)

Tracking

({fixed1.9.1})

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Assignee)

Description

11 years ago
The MS Symbol Server client supports compressed pdb files in a symbol store.  The compression must be done using compress.exe (or its equivalent).  The compressed pdb files must use a .pd_ extension.

This will help with symbol download speed since compress gives ~50% reduction in pdb size, and since xul.pdb is >70Mb, that's a big win.  Should also lower server load for the same reason.

This will be a little tricky to implement, only that compress.exe doesn't come standard with Windows, and installing the reskit puts it in Program Files somewhere, so MozillaBuild won't pick it up in PATH.  There is a GPL implementation of the tool that we might consider rolling into MozillaBuild here:
http://gnuwin32.sourceforge.net/packages/mscompress.htm

The only downside to this tool is that its commandline options are pretty weak compared to compress.exe.  I might do a quick hack to make it support -R (rename the file after compressing it, so file.exe -> file.ex_) like compress.exe.
(Assignee)

Updated

10 years ago
Assignee: ted.mielczarek → nobody
(Assignee)

Comment 1

10 years ago
Just as a point of information, I tried manually compressing firefox.pdb with compress.exe (to create firefox.pd_) and put it on a test symbol server. symchk.exe was unable to successfully download it. Sniffing the HTTP traffic shows that the HTTP request goes through fine, it just fails to use it for some reason.

Comment 2

10 years ago
You can try makecab.exe, which ships with Windows. Just typing makecab xul.pdb will produce xul.pd_, which is much smaller
(Assignee)

Comment 4

10 years ago
Yuhong: I might give that a try, but per comment 1 I tried using compress.exe to do this and couldn't get it to work with a debugger.

Comment 5

10 years ago
"but per comment 1 I tried using compress.exe
to do this and couldn't get it to work with a debugger."
That is why I am asking you to try a different utility, in fact, I was able to makecab xul.pdb on my symbol cache and WinDbg successfully decompressed it.
(Assignee)

Comment 6

10 years ago
Thanks, I'll give this a shot!

Comment 7

10 years ago
"Thanks, I'll give this a shot!"
Please do, because pdbs this large are annoying, especially because the majority of apps that uses the symbol server provide no download progress, only WinDbg does, and I had several occasions where I ended up killing the process, leaving a corrupt pdb on the hard drive.

Comment 8

10 years ago
symstore also has a /compress option.
(Assignee)

Comment 9

10 years ago
We're not using the actual symstore script. We create the symbol store as a side effect of generating the Breakpad symbols in this script:
http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/symbolstore.py
(Assignee)

Updated

10 years ago
Assignee: nobody → ted.mielczarek
Status: NEW → ASSIGNED
(Assignee)

Comment 11

10 years ago
Created attachment 357520 [details] [diff] [review]
use makecab to compress symbols

Pretty simple patch.
Attachment #357520 - Flags: review?(benjamin)

Comment 12

10 years ago
Works, and file is much smaller:
SYMSRV:  xul.pdb from http://mavra.perilith.com/~luser/symbols/: 19397600 bytes
- copied
DBGHELP: xul - private symbols & lines
         c:\downloads\symbols\xul.pdb\4F20D7FCA8484AD7A8E5B68BF6D0C3091\xul.pdb

Updated

10 years ago
Attachment #357520 - Flags: review?(benjamin) → review+
(Assignee)

Comment 13

10 years ago
Pushed to m-c:
http://hg.mozilla.org/mozilla-central/rev/db2cabebef39
Status: ASSIGNED → RESOLVED
Last Resolved: 10 years ago
Resolution: --- → FIXED
When I add the following command line arguments to makecab, I get xul.pd_ down to 12.3MB.
/D CompressionType=LZX /D CompressionMemory=21
(Assignee)

Comment 15

10 years ago
Are those standard options included with all versions of makecab? (Just a sanity check!) Also, care to attach a patch?
And more importantly can windbg and vstudio still download and decompress them?
They've been standard options in the Microsoft CAB SDK since version 1.0. I've got a paper dated March of 1997 documenting those settings. So yes, it's been around for awhile.

There's no reason windbg and vstudio shouldn't be able to work with them. They're still standard CAB files. All those settings do is use LZX compression (a standard CAB feature as noted above) and a larger dictionary size setting. I can attach a patch, but I won't be easily able to test the resulting files.

Comment 18

10 years ago
Yep, I posted a link to MSDN that has the makecab info.
Blocks: 475165

Comment 19

10 years ago
"So yes, it's been around for awhile."
Yep, LZX was added to the CAB format when Forbes went to work for MS back in around 1997.
(Assignee)

Updated

9 years ago
Attachment #357520 - Flags: approval1.9.1?
(Assignee)

Comment 20

9 years ago
Comment on attachment 357520 [details] [diff] [review]
use makecab to compress symbols

We should take this on branch, it makes using the symbol server a lot less painful. It's a very small patch which only impacts the Windows symbol server portion of symbol generation.

Comment 21

9 years ago
Yep, I agree, because I just recently felt the pain you are talking about with a release in the 1.9.0 branch.
Attachment #357520 - Flags: approval1.9.1? → approval1.9.1+
Comment on attachment 357520 [details] [diff] [review]
use makecab to compress symbols

Agreed, makes the process of getting stacks from helpful reporters a ton more pleasant. a191=shaver

Comment 24

9 years ago
In fact, I am now in the habit of running SYMCHK on the Firefox directory each time I install a new release of Firefox exactly because of this. That in fact was how I found bug 489719.
(Assignee)

Updated

9 years ago
Depends on: 520141
You need to log in before you can comment on or make changes to this bug.