Closed Bug 1826703 Opened 2 years ago Closed 1 years ago

Flag hangs so that we can tell them apart from crashes

Categories

(Socorro :: Signature, task)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: gsvelto, Assigned: gsvelto)

References

Details

Attachments

(1 file)

This is a step we need to do before we start capturing Firefox hangs - not just shutdown hangs - so we can tell them apart from crashes during triage. The idea is to add an annotation to the crashes that will indicate the crash is a hang (this is tentatively IsHang, we'll see if we need something more complex than a boolean).

The existing signature generation code prepends the shutdownhang string to crashes following this rule.

We also have AsyncShutdownTimeout crashes, these are also hangs of some sort and we tweak their signatures here.

Finally we have IPCError-browser | ShutDownKill which are also hangs, but they involve child processes not shutting down fast enough, signature generation for them is done here.

So, as the bare minimum we could add a rule similar to the one we have for shutdownhang which would just prepend hang for crashes that have the is IsHang annotation. Alternatively we might do an extra step and try to generalize all the types of hang we have above so that they're processed in the same way and we can search them. A possibility would be to have something like this in the signature:

  • For shutdown hangs hang: shutdown | ... instead of shutdownhang
  • For child process shutdown hangs hang: childshutdown| ... instead of IPCError-browser | ShutDownKill
  • etc...

The above might require some extra work in Firefox to simplify signature generation, for example by having a Hang: <type> annotation on all hangs so that they can be queried instead of relying on the signature alone.

I think this overlaps with bug #1667997 and we should work on that bug first to build out the categorization code that we can use for this bug. Does that sound right?

Yes, if we could do that first it would be great!

I just realized that we might be in a Catch-22 situation here. I'd like us to land bug 1716946 after we do some work to generate the appropriate signatures here, however do process the signatures we need to add the new annotation (Hang) to the ones that are indexed by Socorro, is that correct Will? In this case I guess we'll land bug 1716946, get the new annotation indexed and then do the signature work. We'll have to live for a bit with hangs being indistinguishable from crashes but I think we can live with that.

Flags: needinfo?(willkg)

As near as I can tell, this bug is solely about adjusting signature generation. Given that, we can identify the crash reports that have the Hang annotation and reprocess them. Given that, I don't think there's a catch-22 between the work in this bug and the work in bug #1716946.

Flags: needinfo?(willkg)

(In reply to Will Kahn-Greene [:willkg] ET needinfo? me from comment #4)

As near as I can tell, this bug is solely about adjusting signature generation. Given that, we can identify the crash reports that have the Hang annotation and reprocess them. Given that, I don't think there's a catch-22 between the work in this bug and the work in bug #1716946.

So signature generation doesn't depend on a certain annotation being indexed? I've got a patch incoming then.

Assuming by "indexed" you mean "indexed into Elasticsearch", then no--signature generation uses processed crash data which gets generated every time the crash report is processed.

I've tweaked the signature generations rule to flag hangs. I think I've done it properly but I'm not 100% sure.

Assignee: nobody → gsvelto
Status: NEW → ASSIGNED
No longer blocks: 1716946
Depends on: 1716946

I don't know when all these changes will get deployed to production. Maybe next week. I'll update the bug when that happens.

I deployed this to prod just now in bug #1851648. Marking as FIXED.

Status: ASSIGNED → RESOLVED
Closed: 1 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: