1346860 - [tracker] improve shutdownhang signatures

Status: NEW

(Socorro :: General, task)

Description

[[:philipp]]

a couple of browser shutdownhang signatures for the same issues seem to differ on release from beta. it would be great if they could be unified. a few examples:

bug 1158189
*release: https://crash-stats.mozilla.com/signature/?product=Firefox&signature=shutdownhang%20|%20mozilla%3A%3ACondVar%3A%3AWait%20|%20nsEventQueue%3A%3AGetEvent%20|%20nsThread%3A%3AnsChainedEventQueue%3A%3AGetEvent%20|%20nsThread%3A%3AGetEvent%20|%20nsThread%3A%3AProcessNextEvent%20|%20NS_ProcessNextEvent%20|%20mozilla%3A%3Anet%3A%3AnsHttpConnectionMgr%3A%3AShutdown
*beta: https://crash-stats.mozilla.com/signature/?product=Firefox&signature=shutdownhang%20|%20_PR_MD_WAIT_CV%20|%20_PR_WaitCondVar%20|%20mozilla%3A%3ACondVar%3A%3AWait%20|%20nsEventQueue%3A%3AGetEvent%20|%20nsThread%3A%3AnsChainedEventQueue%3A%3AGetEvent%20|%20nsThread%3A%3AGetEvent%20|%20nsThread%3A%3AProcessNextEvent%20|%20NS_ProcessNextEvent%20|%20mozilla%3A%3Anet%3A%3AnsHttpConnectionMgr%3A%3AShu...

bug 1248818
*release: https://crash-stats.mozilla.com/signature/?product=Firefox&signature=shutdownhang%20|%20PR_JoinThread%20|%20SmartCardMonitoringThread%3A%3A~SmartCardMonitoringThread
*beta:https://crash-stats.mozilla.com/signature/?product=Firefox&signature=shutdownhang%20|%20_PR_MD_WAIT_CV%20|%20_PR_WaitCondVar%20|%20PR_JoinThread%20|%20SmartCardMonitoringThread%3A%3A~SmartCardMonitoringThread

bug 1248819
*release: https://crash-stats.mozilla.com/signature/?product=Firefox&signature=shutdownhang%20|%20mozilla%3A%3ACondVar%3A%3AWait%20|%20nsEventQueue%3A%3AGetEvent%20|%20nsThread%3A%3AnsChainedEventQueue%3A%3AGetEvent%20|%20nsThread%3A%3AGetEvent%20|%20nsThread%3A%3AProcessNextEvent%20|%20NS_InvokeByIndex
*beta: https://crash-stats.mozilla.com/signature/?product=Firefox&signature=shutdownhang%20|%20_PR_MD_WAIT_CV%20|%20_PR_WaitCondVar%20|%20mozilla%3A%3ACondVar%3A%3AWait%20|%20nsEventQueue%3A%3AGetEvent%20|%20nsThread%3A%3AnsChainedEventQueue%3A%3AGetEvent%20|%20nsThread%3A%3AGetEvent%20|%20nsThread%3A%3AProcessNextEvent%20|%20NS_InvokeByIndex

Comment 1

[(Away)]

We could probably just MOZ_ALWAYS_INLINE or MOZ_NEVER_INLINE the _PR functions.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Is this still an issue? Is this something that's fixable in Socorro? Seems like the stacks for those crashes is different.

Comment 3

[[:philipp]]

yeah, the situation for shutdownhang signatures is still a bit of a mess. the top browser crashes are now regularly dominated by shutdownhangs (i want to believe that's because we've got rid of so many other fundamental stability problems). but it's very hard to track those as hangs with the same underlying issue are usually not grouped together, but spread over multiple different signatures. that may be due to the stacks slightly changing with code changes in different versions or issues with missing windows symbols resulting in signatures like [@ shutdownhang | ntdll.dll@0x...] (bug 1493999, bug 1486825).

here's a sad example of the current state of top shutdownhang signatures in 63.0: https://screenshots.firefox.com/x7zD4NvcWl1DWvGz/crash-stats.mozilla.com
i'm out of ideas though on how to systematically approach this issue without constant manual work (adapting skiplists, etc).

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

That shows a screenshot of shutdown hangs in a single release--not across channels. Is the issue here that shutdown hangs are split across too many buckets or something specific to release vs. beta channels?

Comment 5

[[:philipp]]

both :))

i guess the meta issue here is that shutdownhangs with one underlying cause don't usually get grouped together, and as a result it's not possible to reliably judge the impact of a particular shutdown hang issue by just tracking one signature. 
that's the case when looking just at one release where there are usually many signatures for the same crash, but signatures for an issue also often change slightly when moving from one release version to another.

here'd be a comparison of top shutdownhangs between 62.0.3 (#1) and 63.0 (#2) as example - (beware that clicking on this link might get your ip banned from accessing bugzilla for a bit due to aggressive rate-limiting, bug 1482644!):
https://mozilla.github.io/stab-crashes/scomp.html?limit=50&common=product%3DFirefox%26signature%3D%5Eshutdownhang%26date%3D%3E%253D2018-07-01&p1=version%3D62.0.3&p2=version%3D63.0

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

I think this bug is too general to be actionable. I'm going to change it into a general tracker bug. When there are actionable items on it, I can do stuff, but until then, there isn't much I can do.