575817 - Setup new fetch-win32-symbols cron job

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

A crash report such as the one in 575718 doesn't get symbols, even though the symbols for those libs are available from the MS symbol server, e.g.:

http://crash-stats.mozilla.com/report/index/bp-e5a4f19d-0323-4519-8817-c40d82100629

This is in a d2d build, which isn't enabled by default but we'd like it to be.. so collecting this data is pretty important.

ted tells me that his script samples the 500 most recent crashes for dlls to fetch once a day; can we manually fetch the relevant d2d/d3d dlls (the ones referenced in that report would be a good start) and get them in the system?

As a future thing, would be nice if we just kept a list of all dlls we didn't have symbols for, and tried each one against the MS server.

Comment 1

[(not currently active) Ted Mielczarek]

This is the script that's currently running:
http://hg.mozilla.org/users/tmielczarek_mozilla.com/fetch-win32-symbols/

It's running on a VM on the build network (dm-symbolfetch01).

Ideally it'd be able to look at the complete list of crashes every day and all the modules within, but that's a lot of data. We could probably do it with a map/reduce job or something to collect the complete list of unique modules per-day, then run it against that.

Comment 2

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Is there some format etc. that we can provide the list of dlls that we need symbols for?  We'll really want d2d/d3d symbols up on crash-stats before b5 ships..

Comment 3

[(not currently active) Ted Mielczarek]

I pointed Bas at this yesterday, but the tool that the symbol fetching script uses is checked in as a binary to that repo:
http://hg.mozilla.org/users/tmielczarek_mozilla.com/fetch-win32-symbols/raw-file/0c8822ecfd80/symsrv_convert.exe

You need to follow the requirements here:
http://hg.mozilla.org/users/tmielczarek_mozilla.com/fetch-win32-symbols/file/0c8822ecfd80/symsrv-fetch.py#l8

Then you just run it like:
symsrv_convert http://msdl.microsoft.com/download/symbols c:\wherever <pdb filename> <debug id>
Where something.pdb and the debug ID are the info you can find on the "Modules" tab of a crash report. It will spit out a .sym file in c:\wherever.

If you send me the .sym files I'll put them on the symbol store for you. Ideally you could get access to dm-symbolpush01 directly, you might need to file an IT bug for that.

Comment 4

[(not currently active) Ted Mielczarek]

Once the script from bug 594777 gets deployed into production, I'll fix my script to use it as input, and it will fetch symbols for every Win32 module over the entire day's worth of crash reports. I think that should completely solve this problem.

Comment 5

[(not currently active) Ted Mielczarek]

I've fixed the script:
http://hg.mozilla.org/users/tmielczarek_mozilla.com/fetch-win32-symbols/rev/a3701f248944

I have a sample file from bug 594777 (a full day's worth of modules), which I intend to do a test run on, that should give us a bunch more symbols. Unfortunately first I need to deal with bug 598757.

Comment 6

[(not currently active) Ted Mielczarek]

I ran the script on the sample file last week and let it upload to the symbol server. This should have backfilled a lot of missing symbols (it uploaded a few gigs worth). I'll leave this open until we get it up and running on a regular basis.

Comment 7

[(not currently active) Ted Mielczarek]

I ran another one-off run to backfill some symbols since we're missing a bunch of Windows system symbols now. It uploaded ~4400 new symbols.

Comment 8

[(not currently active) Ted Mielczarek]

The cron job in bug 598908 had been running, so I did another run of this yesterday, but it looks like it hasn't run in about a month, so I reopened that bug to get it sorted out before I set this up as a daily job.

Comment 9

[(not currently active) Ted Mielczarek]

The cron job got busted again. I keep wanting to close this bug out, but things keep breaking...

Comment 10

[(not currently active) Ted Mielczarek]

This finally seems to be running okay.