Flag hangs so that we can tell them apart from crashes
Categories
(Socorro :: Signature, task)
Tracking
(Not tracked)
People
(Reporter: gsvelto, Assigned: gsvelto)
References
Details
Attachments
(1 file)
This is a step we need to do before we start capturing Firefox hangs - not just shutdown hangs - so we can tell them apart from crashes during triage. The idea is to add an annotation to the crashes that will indicate the crash is a hang (this is tentatively IsHang
, we'll see if we need something more complex than a boolean).
The existing signature generation code prepends the shutdownhang
string to crashes following this rule.
We also have AsyncShutdownTimeout
crashes, these are also hangs of some sort and we tweak their signatures here.
Finally we have IPCError-browser | ShutDownKill
which are also hangs, but they involve child processes not shutting down fast enough, signature generation for them is done here.
So, as the bare minimum we could add a rule similar to the one we have for shutdownhang
which would just prepend hang
for crashes that have the is IsHang
annotation. Alternatively we might do an extra step and try to generalize all the types of hang we have above so that they're processed in the same way and we can search them. A possibility would be to have something like this in the signature:
- For shutdown hangs
hang: shutdown | ...
instead ofshutdownhang
- For child process shutdown hangs
hang: childshutdown| ...
instead ofIPCError-browser | ShutDownKill
- etc...
The above might require some extra work in Firefox to simplify signature generation, for example by having a Hang: <type>
annotation on all hangs so that they can be queried instead of relying on the signature alone.
Comment 1•2 years ago
|
||
I think this overlaps with bug #1667997 and we should work on that bug first to build out the categorization code that we can use for this bug. Does that sound right?
Assignee | ||
Comment 2•2 years ago
|
||
Yes, if we could do that first it would be great!
Assignee | ||
Comment 3•2 years ago
|
||
I just realized that we might be in a Catch-22 situation here. I'd like us to land bug 1716946 after we do some work to generate the appropriate signatures here, however do process the signatures we need to add the new annotation (Hang
) to the ones that are indexed by Socorro, is that correct Will? In this case I guess we'll land bug 1716946, get the new annotation indexed and then do the signature work. We'll have to live for a bit with hangs being indistinguishable from crashes but I think we can live with that.
Comment 4•2 years ago
|
||
As near as I can tell, this bug is solely about adjusting signature generation. Given that, we can identify the crash reports that have the Hang
annotation and reprocess them. Given that, I don't think there's a catch-22 between the work in this bug and the work in bug #1716946.
Assignee | ||
Comment 5•2 years ago
|
||
(In reply to Will Kahn-Greene [:willkg] ET needinfo? me from comment #4)
As near as I can tell, this bug is solely about adjusting signature generation. Given that, we can identify the crash reports that have the
Hang
annotation and reprocess them. Given that, I don't think there's a catch-22 between the work in this bug and the work in bug #1716946.
So signature generation doesn't depend on a certain annotation being indexed? I've got a patch incoming then.
Comment 6•2 years ago
|
||
Assuming by "indexed" you mean "indexed into Elasticsearch", then no--signature generation uses processed crash data which gets generated every time the crash report is processed.
Assignee | ||
Comment 7•2 years ago
|
||
I've tweaked the signature generations rule to flag hangs. I think I've done it properly but I'm not 100% sure.
Assignee | ||
Updated•2 years ago
|
Comment 8•1 years ago
|
||
Comment 9•1 years ago
|
||
I don't know when all these changes will get deployed to production. Maybe next week. I'll update the bug when that happens.
Comment 10•1 years ago
|
||
I deployed this to prod just now in bug #1851648. Marking as FIXED.
Description
•