1270190 - Missing symbols for Windows ntdll.dll@0x4bb7a and kernelbase.dll@0x10ab

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Andrew McCreight [:mccr8]]

The signature [@ shutdownhang | ntdll.dll@0x4bb7a ] is currently a top crash on Nightly. It would be nice if ntdll.dll@0x4bb7a was symbolized. Also, the next frame kernelbase.dll@0x10ab should be symbolized. They look related to the Windows implementation of CondVar. This would let us add them to the skip list or the irrelevant list.

https://crash-stats.mozilla.com/report/index/2ac95b88-47d4-4c72-a7a0-015712160428

Comment 1

[Benjamin Smedberg]

Peter, who handles Windows symbol import nowadays?

mccr8 FWIW since this is a server-side change it needs to be in the Socorro product instead of the Toolkit/client world.

Comment 2

[Andrew McCreight [:mccr8]]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #1)
> mccr8 FWIW since this is a server-side change it needs to be in the Socorro
> product instead of the Toolkit/client world.

Thanks for the information. I guess symbolizing in the client would not be practical...

Comment 3

[Peter Bengtsson [:peterbe]]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #1)
> Peter, who handles Windows symbol import nowadays?
> 
Ted does. He still has a manual process whereby he uses CSV files spat out by our processor. I don't remember if he uses TaskCluster or just a script on his laptop. 

We initiated a project to entirely fix that with a real-time symbol fetcher thing. But we couldn't get it started because deep down it relies on a Windows only executable and our only way to run that would be by using Wine and that's not an option. 

I believe the latest, potential idea is if that executable has been or can be ported to *nix so we can proceed to build that real-time symbol-fetcher thing. Ted's keeping an eye on it.

Comment 4

[(not currently active) Ted Mielczarek]

This crash is on Win64. Unfortunately we are not fetching symbols for Win64 system libraries(bug 1140358), because we currently can't get stack unwind info for them, because dump_syms needs to have the EXE or DLL file present to dump stack unwind info, and the script I use for scraping missing symbols from Microsoft's symbol server uses the missing-symbols CSV as input, which does not have enough info to fetch the associated DLLs.

We were previously dumping symbols for system libraries on Win64, but without stack unwind info we'd get symbols but really awful stacks, so I disabled that so as to not produce misleading results.

We either need to fix bug 1140937, fix bug 1117741, or check if this unlanded Breakpad dump_syms patch allows us to get the unwind info out of PDB files from Microsoft's symbol server:
https://codereview.chromium.org/1773183003/

(In reply to Peter Bengtsson [:peterbe] from comment #3)
> I believe the latest, potential idea is if that executable has been or can
> be ported to *nix so we can proceed to build that real-time symbol-fetcher
> thing. Ted's keeping an eye on it.

That's something I've poked at but not something I'm actively working on.

Comment 5

[Kan-Ru Chen [:kanru] (UTC+9)]

[@ shutdownhang | ntdll.dll@0x4bb7a] is the top #6 crash in Nightly 20160509040557 but it's actually an aggregation of different shutdown hangs. Fixing this would break them apart.

Comment 6

[Andrew McCreight [:mccr8]]

[@ shutdownhang | ntdll.dll@0xa5164] is similarly the top #2 Windows crasher on 5/11.

Comment 8

[Ting-Yu Chou [:ting] (away)]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #4)
> to dump stack unwind info, and the script I use for scraping missing symbols
> from Microsoft's symbol server uses the missing-symbols CSV as input, which
> does not have enough info to fetch the associated DLLs.

What else information are needed for fetching the DLLs? Since VS can show the symbols properly by openning the minidump.

Comment 9

[Ting-Yu Chou [:ting] (away)]

(In reply to Ting-Yu Chou [:ting] from comment #8)
> What else information are needed for fetching the DLLs? Since VS can show
> the symbols properly by openning the minidump.

I found it in bug 1140937, filename and the code_id.

Comment 10

[Ting-Yu Chou [:ting] (away)]

Forgot to clear the NI because I've found the answer.

Comment 11

[(not currently active) Ted Mielczarek]

(In reply to Andrew McCreight [:mccr8] from comment #0)
> The signature [@ shutdownhang | ntdll.dll@0x4bb7a ] is currently a top crash
> on Nightly. It would be nice if ntdll.dll@0x4bb7a was symbolized. Also, the
> next frame kernelbase.dll@0x10ab should be symbolized. They look related to
> the Windows implementation of CondVar. This would let us add them to the
> skip list or the irrelevant list.
> 
> https://crash-stats.mozilla.com/report/index/2ac95b88-47d4-4c72-a7a0-
> 015712160428

Since fixing bug 1140358 we have acquired symbols for the modules in this report, such as:
http://symbols.mozilla.org/ntdll.pdb/0A2571A3E36A4F909B719773CC176B032/ntdll.sym

This should be fixed in the general case, as we are now backfilling Win64 symbols just like we've historically done for Win32 symbols (thanks to peterbe fixing bug 1140937). If you look at the graph for that crash signature you can see that it has dropped to virtually zero since we started getting symbols:
https://crash-stats.mozilla.com/signature/?product=Firefox&signature=shutdownhang%20%7C%20ntdll.dll%400x4bb7a#graphs

We should be able to reprocess these crashes to get good signatures out of the existing reports, but that's not a self-service action right now. (We recently gained the ability to self-service reprocess individual crash IDs, but not yet by signature etc.) I filed bug 1280213 for that.

Comment 12

[Andrew McCreight [:mccr8]]

Thanks, Ted! I posted about this on the Uptime list, for people who are looking at Nightly crashes to be aware of it.

Comment 13

[(not currently active) Ted Mielczarek]

The crash from comment 0 winds up with the signature [@ shutdownhang | WaitForSingleObjectEx | PR_WaitCondVar | mozilla::CondVar::Wait | nsEventQueue::GetEvent ] after reprocessing:
https://crash-stats.mozilla.com/report/index/2ac95b88-47d4-4c72-a7a0-015712160428