Closed Bug 385792 Opened 17 years ago Closed 16 years ago

compress pdb files in symbol store


(Toolkit :: Crash Reporting, defect)

Windows XP
Not set





(Reporter: ted, Assigned: ted)



(Keywords: fixed1.9.1)


(1 file)

The MS Symbol Server client supports compressed pdb files in a symbol store.  The compression must be done using compress.exe (or its equivalent).  The compressed pdb files must use a .pd_ extension.

This will help with symbol download speed since compress gives ~50% reduction in pdb size, and since xul.pdb is >70Mb, that's a big win.  Should also lower server load for the same reason.

This will be a little tricky to implement, only that compress.exe doesn't come standard with Windows, and installing the reskit puts it in Program Files somewhere, so MozillaBuild won't pick it up in PATH.  There is a GPL implementation of the tool that we might consider rolling into MozillaBuild here:

The only downside to this tool is that its commandline options are pretty weak compared to compress.exe.  I might do a quick hack to make it support -R (rename the file after compressing it, so file.exe -> file.ex_) like compress.exe.
Assignee: ted.mielczarek → nobody
Just as a point of information, I tried manually compressing firefox.pdb with compress.exe (to create firefox.pd_) and put it on a test symbol server. symchk.exe was unable to successfully download it. Sniffing the HTTP traffic shows that the HTTP request goes through fine, it just fails to use it for some reason.
You can try makecab.exe, which ships with Windows. Just typing makecab xul.pdb will produce xul.pd_, which is much smaller
Yuhong: I might give that a try, but per comment 1 I tried using compress.exe to do this and couldn't get it to work with a debugger.
"but per comment 1 I tried using compress.exe
to do this and couldn't get it to work with a debugger."
That is why I am asking you to try a different utility, in fact, I was able to makecab xul.pdb on my symbol cache and WinDbg successfully decompressed it.
Thanks, I'll give this a shot!
"Thanks, I'll give this a shot!"
Please do, because pdbs this large are annoying, especially because the majority of apps that uses the symbol server provide no download progress, only WinDbg does, and I had several occasions where I ended up killing the process, leaving a corrupt pdb on the hard drive.
symstore also has a /compress option.
We're not using the actual symstore script. We create the symbol store as a side effect of generating the Breakpad symbols in this script:
Assignee: nobody → ted.mielczarek
Pretty simple patch.
Attachment #357520 - Flags: review?(benjamin)
Works, and file is much smaller:
SYMSRV:  xul.pdb from 19397600 bytes
- copied
DBGHELP: xul - private symbols & lines
Attachment #357520 - Flags: review?(benjamin) → review+
Pushed to m-c:
Closed: 16 years ago
Resolution: --- → FIXED
When I add the following command line arguments to makecab, I get xul.pd_ down to 12.3MB.
/D CompressionType=LZX /D CompressionMemory=21
Are those standard options included with all versions of makecab? (Just a sanity check!) Also, care to attach a patch?
And more importantly can windbg and vstudio still download and decompress them?
They've been standard options in the Microsoft CAB SDK since version 1.0. I've got a paper dated March of 1997 documenting those settings. So yes, it's been around for awhile.

There's no reason windbg and vstudio shouldn't be able to work with them. They're still standard CAB files. All those settings do is use LZX compression (a standard CAB feature as noted above) and a larger dictionary size setting. I can attach a patch, but I won't be easily able to test the resulting files.
Yep, I posted a link to MSDN that has the makecab info.
"So yes, it's been around for awhile."
Yep, LZX was added to the CAB format when Forbes went to work for MS back in around 1997.
Attachment #357520 - Flags: approval1.9.1?
Comment on attachment 357520 [details] [diff] [review]
use makecab to compress symbols

We should take this on branch, it makes using the symbol server a lot less painful. It's a very small patch which only impacts the Windows symbol server portion of symbol generation.
Yep, I agree, because I just recently felt the pain you are talking about with a release in the 1.9.0 branch.
Attachment #357520 - Flags: approval1.9.1? → approval1.9.1+
Comment on attachment 357520 [details] [diff] [review]
use makecab to compress symbols

Agreed, makes the process of getting stacks from helpful reporters a ton more pleasant. a191=shaver
In fact, I am now in the habit of running SYMCHK on the Firefox directory each time I install a new release of Firefox exactly because of this. That in fact was how I found bug 489719.
Depends on: 520141
You need to log in before you can comment on or make changes to this bug.