Closed Bug 385792 Opened 14 years ago Closed 12 years ago
compress pdb files in symbol store
The MS Symbol Server client supports compressed pdb files in a symbol store. The compression must be done using compress.exe (or its equivalent). The compressed pdb files must use a .pd_ extension. This will help with symbol download speed since compress gives ~50% reduction in pdb size, and since xul.pdb is >70Mb, that's a big win. Should also lower server load for the same reason. This will be a little tricky to implement, only that compress.exe doesn't come standard with Windows, and installing the reskit puts it in Program Files somewhere, so MozillaBuild won't pick it up in PATH. There is a GPL implementation of the tool that we might consider rolling into MozillaBuild here: http://gnuwin32.sourceforge.net/packages/mscompress.htm The only downside to this tool is that its commandline options are pretty weak compared to compress.exe. I might do a quick hack to make it support -R (rename the file after compressing it, so file.exe -> file.ex_) like compress.exe.
Assignee: ted.mielczarek → nobody
Just as a point of information, I tried manually compressing firefox.pdb with compress.exe (to create firefox.pd_) and put it on a test symbol server. symchk.exe was unable to successfully download it. Sniffing the HTTP traffic shows that the HTTP request goes through fine, it just fails to use it for some reason.
You can try makecab.exe, which ships with Windows. Just typing makecab xul.pdb will produce xul.pd_, which is much smaller
More on makecab.exe: http://msdn.microsoft.com/en-us/library/bb417343.aspx
Yuhong: I might give that a try, but per comment 1 I tried using compress.exe to do this and couldn't get it to work with a debugger.
"but per comment 1 I tried using compress.exe to do this and couldn't get it to work with a debugger." That is why I am asking you to try a different utility, in fact, I was able to makecab xul.pdb on my symbol cache and WinDbg successfully decompressed it.
Thanks, I'll give this a shot!
"Thanks, I'll give this a shot!" Please do, because pdbs this large are annoying, especially because the majority of apps that uses the symbol server provide no download progress, only WinDbg does, and I had several occasions where I ended up killing the process, leaving a corrupt pdb on the hard drive.
symstore also has a /compress option.
We're not using the actual symstore script. We create the symbol store as a side effect of generating the Breakpad symbols in this script: http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/symbolstore.py
Assignee: nobody → ted.mielczarek
Status: NEW → ASSIGNED
This appears to work. Test build at: http://mavra.perilith.com/~luser/firefox-3.2a1pre.en-US.win32.zip With symbols available at: http://mavra.perilith.com/~luser/symbols/
Pretty simple patch.
Attachment #357520 - Flags: review?(benjamin)
Works, and file is much smaller: SYMSRV: xul.pdb from http://mavra.perilith.com/~luser/symbols/: 19397600 bytes - copied DBGHELP: xul - private symbols & lines c:\downloads\symbols\xul.pdb\4F20D7FCA8484AD7A8E5B68BF6D0C3091\xul.pdb
Pushed to m-c: http://hg.mozilla.org/mozilla-central/rev/db2cabebef39
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
When I add the following command line arguments to makecab, I get xul.pd_ down to 12.3MB. /D CompressionType=LZX /D CompressionMemory=21
Are those standard options included with all versions of makecab? (Just a sanity check!) Also, care to attach a patch?
And more importantly can windbg and vstudio still download and decompress them?
They've been standard options in the Microsoft CAB SDK since version 1.0. I've got a paper dated March of 1997 documenting those settings. So yes, it's been around for awhile. There's no reason windbg and vstudio shouldn't be able to work with them. They're still standard CAB files. All those settings do is use LZX compression (a standard CAB feature as noted above) and a larger dictionary size setting. I can attach a patch, but I won't be easily able to test the resulting files.
Yep, I posted a link to MSDN that has the makecab info.
"So yes, it's been around for awhile." Yep, LZX was added to the CAB format when Forbes went to work for MS back in around 1997.
Attachment #357520 - Flags: approval1.9.1?
Comment on attachment 357520 [details] [diff] [review] use makecab to compress symbols We should take this on branch, it makes using the symbol server a lot less painful. It's a very small patch which only impacts the Windows symbol server portion of symbol generation.
Yep, I agree, because I just recently felt the pain you are talking about with a release in the 1.9.0 branch.
Attachment #357520 - Flags: approval1.9.1? → approval1.9.1+
Comment on attachment 357520 [details] [diff] [review] use makecab to compress symbols Agreed, makes the process of getting stacks from helpful reporters a ton more pleasant. a191=shaver
Pushed to 1.9.1: http://hg.mozilla.org/releases/mozilla-1.9.1/rev/712b74ad6d37
In fact, I am now in the habit of running SYMCHK on the Firefox directory each time I install a new release of Firefox exactly because of this. That in fact was how I found bug 489719.
You need to log in before you can comment on or make changes to this bug.