1849988 - Automatically scrape symbols for Windows graphics drivers

Status: RESOLVED FIXED

(Toolkit :: Crash Reporting, enhancement)

Description

[Gabriele Svelto [:gsvelto]]

In bug 1655476 we've added the official symbol servers for Intel, AMD and Nvidia in the hope that we'd be able to find some debugging information for their graphics drivers. Unfortunately only a handful of executable files are available on those servers and the coverage they provide is very limited and we lack unwinding information for all but a subset of the drivers available in the wild.

I have attempted in the past to scrape the drivers directly from the vendors' websites but they make it especially hard to list them in a meaningful way. The pages also change over time or are impossible to enumerate.

Fortunately the TechPowerUp website offers downloads of the graphics drivers from all vendors, they promptly update them and have a simple-to-use HTTP interface in front of the downloads that can be easily used for scraping. See this page as an example.

So this bug involves the following:

• Add a script to the symbol scraper tasks that uses the TechPowerUp pages to get a list of available graphics drivers, download the ones we haven't seen yet, extract them and dump the symbols for use by our crash reporting infrastructure
• Make sure the script treats this precious resource gently: let's cap driver downloads to 10 per day and make requests sequential. We don't want to be 429'ed or worse blocked
• Write a polite e-mail to TechPowerUp to inform them that we'll be doing this beforehand, don't do this in case they're not happy with it

FYI we'll only retain the Breakpad symbol files we'll have extracted from the drivers, not the EXE/DLLs or PDBs. Distribution of those might be subject to restrictions so we don't want them to our servers. Users that need them (for example for use in Visual Studio) will have to use the vendor's symbol servers directly.

Comment 1

[Gabriele Svelto [:gsvelto]]

I've talked with TechPowerUp editor-in-chief and they're cool with this. We're hashing out some details but I should have a script up and running in no time.

Comment 2

[Gabriele Svelto [:gsvelto]]

Attachment: Bug 1849988 - Scrape symbols Windows graphics drivers r=afranchuk

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/94c394a85b3a
Scrape symbols Windows graphics drivers r=afranchuk

Comment 4

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/94c394a85b3a