1292608 - Searching doesn't support all threads stack trace

Status: RESOLVED WONTFIX

(Socorro :: Webapp, task)

Description

[Dragana Damjanovic [:dragana]]

For example:
https://crash-stats.mozilla.com/report/index/1ea97466-a39c-4730-9959-b80952160805#allthreads
has mozilla::net::nsHttpConnectionMgr::Shutdown() on the stack on the main thread, but if I search for nsHttpConnectionMgr this crash does not show up.

it is hard to search crash stacks.
The crash with signature [@ shutdownhang | WaitForSingleObjectEx | WaitForSingleObject | PR_WaitCondVar | mozilla::CondVar::Wait | nsEventQueue::GetEvent ] are at least 10 different crash signatures.

We should extend the title as well so that in the example above the title also shows mozilla::net::nsHttpConnectionMgr::Shutdown()

Comment 1

[[DEACTIVATED] Adrian Gaudebert]

I'm not sure I understand what you want here. If you want to search through the entire stack trace, it's not possible. There are however ways to search the top frames of the crashing thread: use the `proto_signature` field instead of `signature`. For example: https://crash-stats.mozilla.com/search/?proto_signature=~moz&_sort=-date&_facets=proto_signature&_columns=signature&_columns=proto_signature

If, on the other hand, you need `mozilla::net::nsHttpConnectionMgr::Shutdown()` to be included in signatures, then this is a siglist bug, and you'll want to edit our signatures lists according to these instructions: https://github.com/mozilla/socorro/tree/master/socorro/siglists

Comment 2

[Jason Duell]

> If you want to search through the entire stack trace, it's not possible.

This is what we want.  How hard would that be to implement?

Comment 3

[Dragana Damjanovic [:dragana]]

proto_signature helps. Thanks!

Comment 4

[Jason Duell]

Some context:  my team (necko, the networking stack) has been tracking down one of our largest crashes (socket transport hangs in poll() and connect()).  We though we'd killed the bug, but it turns out that apparently the signature of the crash morphs a lots (and the function at the top of the stack gets shared with a lot of other unrelated crashes).  So our biggest crash went "missing" for a long time.

My people are telling me that they're wading through a lot of crash reports manually to try to get the data they want.  There's talk of writing scripts that download the crash report web pages and slurp the data we need (functions on the stack) and parse out the crashes we're interested in. It's a lot of work to do, but we've got our topcrashers to fix.

It sounds like we have a plan to merge crash data into the data platform, after which this kind of search will be supported.  That's great, though of course not helpful just yet.  If you've got any other ideas that might help in the meantime, we're very open to hearing them.  This is definitely a productivity bottleneck for us right now.

Comment 5

[Jonathan Howard]

Bug 1272911 will solve the signature.

Comment 6

[[DEACTIVATED] Adrian Gaudebert]

The easiest method for searching more of the stack trace right now is to use ``proto_signature``. We can also work on improving our signatures, like in the bug linked to by Jonathan. 

Searching the full stack requires some amount of coding, and financial decisions to improve the capacities of our databases. I am not in capacity to make these decisions (302 Lonnen).

Comment 7

[Lonnen :lonnen]

Full stack analysis is available in a.t.m.o dataset. Chutton presents some example analysis of crash data in that tooling in this bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1280659

We are unlikely to ever put the full stack in elastic search, but you should be able to do arbitrarily complex analysis with a notebook in a.t.m.o