Set _NT_SYMBOL_PATH on Windows test machines

NEW
Unassigned

Status

Release Engineering
Platform Support
2 years ago
2 years ago

People

(Reporter: aklotz, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(URL)

I'd like to start a discussion about improved symbolication of call stacks in our Windows builds. In particular, the lack of symbolication of system DLLs when we log a call stack (due to assertions, crashes, etc).

In particular, I'd like us to consider setting _NT_SYMBOL_PATH such that the Microsoft dbghelp library can properly reference Microsoft symbols.

These symbols may either be accessed via HTTP on demand, or cached locally for later use. Obviously in the latter case, they do take up a non-trivial amount of disk space. Every time there is an update of an OS component, the symbol client may need to download one or more new symbol files.

Of course, these could be cached on a centralized server, or the local cache could be purged monthly, or something else... anyway, I'm curious what the RelEng opinion is on doing something like this.

Comment 1

2 years ago
(In reply to Aaron Klotz [:aklotz] (please use needinfo) from comment #0) 
> These symbols may either be accessed via HTTP on demand, or cached locally
> for later use. Obviously in the latter case, they do take up a non-trivial
> amount of disk space. Every time there is an update of an OS component, the
> symbol client may need to download one or more new symbol files.
> 
> Of course, these could be cached on a centralized server, or the local cache
> could be purged monthly, or something else... anyway, I'm curious what the
> RelEng opinion is on doing something like this.

We definitely want to facilitate easier debugging. If we can do so while keeping the machine images as simple as possible, all the better. If we can access the symbols on-demand, that would be the best. 

We don't tend to rev the underlying OS or components very often because it can mess with performance data and cause other test failures. We need a period to validate a new baseline when this happens.

Do all the Windows test machine types support this variable, i.e. w10/w8/w7/xp?
(In reply to Chris Cooper [:coop] from comment #1)
> Do all the Windows test machine types support this variable, i.e.
> w10/w8/w7/xp?
Yes.
The most trivial setting for this variable is:
_NT_SYMBOL_PATH=srv*http://msdl.microsoft.com/download/symbols

if we were to create a downstream store to cache the files locally, we'd set it to:
_NT_SYMBOL_PATH=srv*C:\symbols*http://msdl.microsoft.com/download/symbols
where C:\Symbols is an absolute path to a directory created for that purpose.

Comment 4

2 years ago
Looping in Rob to get his take here.

Rob: what's the correct way to do this so that it covers all our test systems? Is this something puppet can manage now, or do we still need to make changes in https://hg.mozilla.org/build/buildbotcustom/file/d56b91bb3b1f/env.py ?
Flags: needinfo?(rthijssen)
Puppet can manage this but the timing isn't good to submit a puppet patch this week (transitioning from puppetless ec2 2008 instances to puppeted ones this week). After this week, when the dust starts to settle, its a simple patch/commit/merge.

markco or myself can assist.
Flags: needinfo?(rthijssen)

Comment 6

2 years ago
(In reply to Rob Thijssen (:grenade - GMT) from comment #5)
> Puppet can manage this but the timing isn't good to submit a puppet patch
> this week (transitioning from puppetless ec2 2008 instances to puppeted ones
> this week). After this week, when the dust starts to settle, its a simple
> patch/commit/merge.
> 
> markco or myself can assist.

Totally fair. Thanks for the info.
You need to log in before you can comment on or make changes to this bug.