419882 - fetch missing Win32 system symbols from MS symbol server

Status: VERIFIED FIXED

(Socorro :: General, task)

Description

[(not currently active) Ted Mielczarek]

We have some Win32 system symbols available to the processor, but every time Microsoft releases a security update, they get revved and we no longer have symbols for them. They're easily available from the MS Symbol Server, and we have all the info we need to get them, but we'd need to run a process on a Windows machine (could be a VM) to convert them from PDB to .sym. We should have a little monitor that (nightly?) looks at the modules list from Win32 reports, and downloads any symbols that we're missing from the symbol server.

The URL is a link to a utility library in the Breakpad source for downloading symbols from the symbol server and converting them to .sym.

Comment 1

[(not currently active) Ted Mielczarek]

Attachment: mostly-working python script

This pretty much works on my Windows box. I should test it on a VM with a clean install of python and psycopg2, I'll do that before I'm done. The only part that doesn't work is the query using last_run, I can't seem to figure out what psycopg2 wants to do with timestamp values. We'll need to run this on a Windows machine (or probably VM) that has access to the database, and write access to symbols_os. I don't know if we can mount that directly as a drive in a Windows VM, but that would be super useful. This also depends on some binary bits. I have a little program built on that Breakpad lib that does the downloading, I'll get that source checked in somewhere. The program depends on having symsrv.dll and dbghelp.dll in the same dir, and I don't know that I can redist those both, so it would probably be easiest for me to just give you a zip of the binary components.

I'm not sure where this script would want to live. Somewhere in the Socorro repo?

Comment 2

[(not currently active) Ted Mielczarek]

morgamic: do you know anything about psycopg2? I am failing at it with datetimes.

Comment 3

[(not currently active) Ted Mielczarek]

Attachment: working python script [checked in]

Ok, this works. I figured out my psycopg2 stupidity. Filed a bug blocking this to actually get a place to run this. morgamic: any preference on where I check this code in? I was thinking just a socorro/tools or utils dir. It's more related to socorro than anything else. I should try to get the C++ code that wraps that Breakpad lib checked into Breakpad.

Comment 4

[Michael Morgan [:morgamic]]

I think a good place for it would be in /scripts?

Comment 5

[(not currently active) Ted Mielczarek]

Checked the python script into Socorro at scripts/symbol-server. I'll sort out the rest soon.

Comment 6

[Nikolay Shopik]

I wonder if this bug affects crash-stats.mozilla.com so it can display function names for MS libraries?

Comment 7

[(not currently active) Ted Mielczarek]

Yes, that's the whole point. I have this partially implemented, I need to find some time to finish it up.

Comment 8

[(not currently active) Ted Mielczarek]

I filed bug 492017 now that Socorro provides JSON output for crash reports. If my script could occasionally poll a list of recent crashes, it could read the list of modules from each one and use that as its input.

Comment 9

[(not currently active) Ted Mielczarek]

Revived the old code from Socorro SVN and pushed it here:
http://hg.mozilla.org/users/tmielczarek_mozilla.com/fetch-win32-symbols/

Going to rewrite it to use the ATOM feed+JSON output provided in bug 492017.

Comment 10

[(not currently active) Ted Mielczarek]

Status update: I've finished implementing a version that uses the ATOM feed+JSON output, the code is available at the HG repo mentioned above. It works in my local testing, I'm now testing on the bm-symbolfetch01 VM. If all goes well, I'll set it up as a daily job and close this bug out.

Comment 11

[(not currently active) Ted Mielczarek]

After a bit of testing, I have this up and running as a daily job on the VM. This should mean that popular missing Win32 symbols should be found pretty quickly. The script grabs the last 500 Windows crashes when it runs and attempts to download symbols for any modules listed in their modules list for which we don't already have symbols. Hopefully we'll see unknown Win32 system symbols go down in the coming days.

Comment 12

[Nikolay Shopik]

Thanks, now I see symbols