630233 - Hang signatures are incorrect

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Benjamin Smedberg]

The parent-side hang signatures are supposed to use a sentinel value to skip up to the caller of RPCChannel::Call, but it is apparently not working. See this report for a recent example:

https://crash-stats.mozilla.com/report/index/5f1a402f-8246-4fbf-88ec-10ea52110131

This appears like it might be caused by a whitespace change in the signature algorithm? According to that URL the CreatePairedMinidumps frame is

CrashReporter::CreatePairedMinidumps(void*,unsigned long,nsAString_internal*,nsILocalFile**,nsILocalFile**)

And the regex at http://code.google.com/p/socorro/source/browse/trunk/scripts/config/processorconfig.py.dist#98 is searching for

CrashReporter::CreatePairedMinidumps(void*, unsigned long, nsAString_internal*, nsILocalFile**, nsILocalFile**)

It's critical to get this working again so that hang report signatures can be classified effectively.

Comment 1

[K Lars Lohn [:lars] [:klohn]]

On digging into this, I find that your conjecture is not correct.  The problem stems from the specification in the SignatureSentinel rule:

('mozilla::ipc::RPCChannel::Call', lambda x: ... )

the signature generation object successfully finds the 'CrashReporter...' signature, but then fails to find 'mozilla::ipc::RPCChannel::Call' as a sentinel.  That's because it's looking for an exact string match and it appears in the list of normalized frames as "mozilla::ipc::RPCChannel::Call(IPC::Message *,IPC::Message *)"

The solution is to change the rule in the processor config to include the entire signature with the parameters as it appears in the frames list.

If you concur with this change, I will get the change pushed to production.

Comment 2

[Benjamin Smedberg]

Hrm, I guess that's ok, although it's going to be a bit fragile. Is there a way to substring-search for "mozilla::ipc::RPCChannel::Call" instead? Either way is fine for now.

Comment 3

[K Lars Lohn [:lars] [:klohn]]

yes, it would be possible to do a substring match, however it would require coding changes.

moving on with the good enough for now solution, I've changed the config and tested it.  For your crash above, it now generates the signature "hang | mozilla::plugins::PPluginInstanceParent::CallNPP_SetWindow(mozilla::plugins::NPRemoteWindow const&)".  

Is this correct and what you expect?  If not, what would you rather it have emitted as a signature?

Comment 4

[Benjamin Smedberg]

Yes, that is the expected signature.

Comment 5

[K Lars Lohn [:lars] [:klohn]]

the change has been made in production to include the parameters in the signatureSentinel for 'mozilla::ipc::RPCChannel::Call'.  Please be on the watch for how the resultant signatures appear in production data.  If there are further problems, please reopen this bug.