1117741 - Set up Microsoft symbol fetching proxy in AWS

Status: RESOLVED DUPLICATE of bug 1226689

(Socorro :: General, task)

Description

[(not currently active) Ted Mielczarek]

Right now I have this script:
http://hg.mozilla.org/users/tmielczarek_mozilla.com/fetch-win32-symbols/

which takes as input the missing symbols cron output:
https://crash-analysis.mozilla.com/crash_analysis/20141229/20141229-missing-symbols.txt

and attempts to fetch any symbols the Microsoft symbol server has and upload them to our symbol store. This gets us symbols for anything from Microsoft that winds up in our stacks, like Windows system libraries. It works pretty well, but it needs to run on Windows because it uses Breakpad's code to fetch and dump the PDB files, which uses Microsoft libraries to do the hard work.

I have this script installed on a Windows VM (symbolfetch1.dmz.phx1) and I had it running as a scheduled task, but that doesn't seem to work reliably. I've never figured out exactly what's wrong there--either dbghelp.dll doesn't want to work in that scenario, or the mounted filesystems from the NetApp don't want to work, not sure, but it just doesn't run properly. As a result of this I wind up RDPing into that machine and doing manual runs to flesh out missing Windows symbols.

I need to figure out how to get this running as a real cron job to take out this manual step.

If I can diagnose what's broken with the current setup and just turn the scheduled task back on, great.

My alternate plan is to take the alternate dump_syms implementation:
https://github.com/luser/dump_syms

and wire that up so that it can replace the existing Breakpad code. We could then run that on a Linux system which would be much nicer.

Comment 1

[(not currently active) Ted Mielczarek]

Orphis (at Spotify, Socorro user) has a Python proxy that fetches and converts symbols from Microsoft's symbol server on-the-fly and stores them in S3. He put the code up somewhere but I've lost the link, I'll put it here when I find it again. I think we should work on standing this up in our AWS production environment instead of working on the cron job. We could have it listed as a http symbol path after the S3 buckets, so that processors looking for symbols would fall through to it only for symbols that weren't found. The main benefit here would be that we'd fetch missing symbols as soon as they were available from Microsoft instead of having to wait for a cron job to backfill reports that were processed without those symbols.

This still depends on running the Windows dump_syms under Wine (bug 1140392), but we've tested and that works fine, so it's just a matter of defining the machine configuration in Puppet or whatever to have everything necessary installed.

Comment 2

[(not currently active) Ted Mielczarek]

Symbol server proxy code (since I keep forgetting where it is):
https://gist.github.com/luser/92d5bc88478665554898

Comment 3

[Peter Bengtsson [:peterbe]]

I know https://bugzilla.mozilla.org/show_bug.cgi?id=1226689 was created earlier but this bug has more recent and fresh comments.