1375511 - Make shutdownhang signatures for Firefox 55 more relevant

Status: RESOLVED DUPLICATE of bug 1402037

(Socorro :: General, task)

Description

[[:philipp]]

something seems to have changed in the signature generation for browser shutdownhangs in firefox 55. the list of top shutdownhangs in 55.0b currently looks like and is lumping together various different crash reasons:
https://crash-stats.mozilla.com/search/?signature=^shutdownhang&product=Firefox&version=55.0b3&process_type=browser&date=%3E%3D2017-06-15T14%3A08%3A00.000Z&date=%3C2017-06-22T14%3A08%3A00.000Z

it would be great if a number of steps could be removed from the signatures through the skiplist or ignorelist in order to end up with more meaningful signatures:
mozilla::TimeStamp::Now
NtCreateUserProcess
NtWaitForAlertByThreadId
NtWaitForKeyedEvent
RtlSleepConditionVariableSRW
SleepConditionVariableSRW
ZwCreateUserProcess
ZwWaitForKeyedEvent

Comment 1

[Marco Castelluccio [:marco]]

"NtWaitForAlertByThreadId", "NtWaitForKeyedEvent", "RtlSleepConditionVariableSRW" and "ZwWaitForKeyedEvent" are already skipped.

I'm not sure if it's useful to skip "NtCreateUserProcess" and "ZwCreateUserProcess".

I will add "SleepConditionVariableSRW" and "mozilla::TimeStamp::Now".

Comment 2

[Marco Castelluccio [:marco]]

Attachment: Link to Github pull-request: https://github.com/mozilla-services/socorro/pull/3827

Comment 3

[Nathan Froyd [:froydnj]]

(In reply to Marco Castelluccio [:marco] from comment #1)
> I will add "SleepConditionVariableSRW" and "mozilla::TimeStamp::Now".

If we are crashing in TimeStamp::Now, I think we want to know about that and don't want to skip it.

Comment 4

[Marco Castelluccio [:marco]]

There are a lot of shutdownhangs where TimeStamp::Now is being called by waiting on a condition variable.

Comment 5

[[github robot]]

Commit pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/7266062784ed5e7ab82a1787dd47c4774e45fe04
Fixes bug 1375511 - Add more shutdownhang-related signatures to the skiplist (#3827)

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

Whoops--this shouldn't be closed by the bot. Re-opening.

Comment 7

[Will Kahn-Greene [:willkg] ET needinfo? me]

Got -stage deployed. I checked a couple of crashes:

https://crash-stats.allizom.org/report/index/e0ae05e2-4ad6-40f1-b927-e249f0170622#tab-details

Old signature: shutdownhang | NtWaitForAlertByThreadId | RtlSleepConditionVariableSRW | SleepConditionVariableSRW

New signature: shutdownhang | NtWaitForAlertByThreadId | RtlSleepConditionVariableSRW | SleepConditionVariableSRW | mozilla::detail::ConditionVariableImpl::wait | mozilla::CondVar::Wait | nsEventQueue::GetEvent | nsThread::nsChainedEventQueue::GetEvent | nsThread::Ge...


https://crash-stats.allizom.org/report/index/54a0182d-975e-4d63-a7eb-a1d1a0170622

Old signature: shutdownhang | ZwWaitForKeyedEvent | RtlSleepConditionVariableSRW | mozilla::TimeStamp::Now

New signature: shutdownhang | ZwWaitForKeyedEvent | RtlSleepConditionVariableSRW | mozilla::TimeStamp::Now | mozilla::CondVar::Wait | nsEventQueue::GetEvent | nsThread::nsChainedEventQueue::GetEvent | nsThread::ProcessNextEvent


We'll push those changes to -prod today or tomorrow.

Comment 8

[Will Kahn-Greene [:willkg] ET needinfo? me]

We pushed PR 3827 to prod. Those should be available now.

Comment 9

[Marco Castelluccio [:marco]]

Thanks Will!

I see the signatures got really long and we might need to add some functions to the ignore list. I've asked on [e-mail] and Benjamin suggested trying that out on a sample of crashes to make sure it doesn't cause bad things.

How could we do that? Can we land a patch on staging and test it there for a while before deciding to merge it to production?

Comment 10

[Will Kahn-Greene [:willkg] ET needinfo? me]

I think we can definitely do rounds of land patches, wait for a stage deploy, test signatures.

Schedule-wise, we started the Elasticsearch migration today and we're going to need to wait until that's done before we can land anything. Further, Tuesday is a US holiday. Between those, it might be next week before we can do this.