1538274 - Investigate ShutDownKill spike in builds from March 19th

Status: RESOLVED FIXED

(Core :: DOM: Content Processes, defect)

Description

[Marcia Knous [:marcia]]

Starting in https://bit.ly/2TnVXgM it appears we started a spike in ShutDownKill crashes on Nightly 68. In the last 7 days we have 14818 crashes in that signature.

Comment 1

[[:philipp]]

could the spike in shutdownkill reports be related to the changes in bug 1533842?:
https://crash-stats.mozilla.com/signature/?product=Firefox&release_channel=nightly&signature=IPCError-browser%20%7C%20ShutDownKill&date=%3E%3D2018-09-22T17%3A08%3A00.000Z#graphs

back in november bug 1498942 had dramatically reduced the number of these reports...

Comment 2

[Marcia Knous [:marcia]]

Since it is the end of the work day, adding a ni on Jim M to see if we think the bug in Comment 1 may be related.

Comment 3

[Marcia Knous [:marcia]]

Here is what I believe to be the regression range based on the Build ID: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=9421b477d67cfc4c9e03350cd554a9e6acc7f435&tochange=d15d511d7fed036d69d37cecbacdea779a983a34

Comment 4

[Gabriele Svelto [:gsvelto]]

Yes, bug 1533842 but the issue turned out to be more complex than I had anticipated. Bug 1498942 might have had an oversized impact because it stopped minidumps from being taken when shutting down hung content processes even when it should not have to. More details in bug 1536850 comment 2. I'll have a patch ready on Monday.

Comment 5

[Marcia Knous [:marcia]]

Work ongoing in Bug 1536850.

Comment 6

[Mike Conley (:mconley) (:⚙️)]

With bug 1536850 fixed, has the investigation concluded and can we close this bug?

Comment 7

[Marcia Knous [:marcia]]

Would prefer to keep this open - it has only been about 3 days since that bug was fixed. We can certainly close it out when we see a little more data.

Comment 8

[Gabriele Svelto [:gsvelto]]

As you can see the ShutDownKill crashes have measurably gone down but they've not been eliminated. This is to be expected; the issue I introduced when fixing bug 1498942 effectively eliminated taking minidumps from hung content processes so we didn't see them anymore. The remaining problem is that hung content processes have all the same signature now - as they had before - and we should probably fix that to be able to tell them apart. I'll file a bug for that.

Comment 9

[Marcia Knous [:marcia]]

I think we can close this one out. It would great to reference the bug Gabriele mentions in Comment 8 so we have it for future reference.

Comment 10

[Gabriele Svelto [:gsvelto]]

Yes, here we go.

Comment 11

[Marcia Knous [:marcia]]

Closing this one out per Comment 8.