1406442 - Improve crash annotations for IPC ShutDownKill after ContentChild::SendFinishShutdown called

Status: RESOLVED FIXED

(Core :: DOM: Content Processes, enhancement)

Description

[Andrew Osmond [:aosmond] (he/him)]

We have seen a significant increase (2-3x) since the days of 55b in the proportion of IPC ShutDownKill crash reports after ContentChild::SendFinishShutdown has been called. This could be explained by either the SendFinishShutdown call failing in the content process, or the RecvFinishShutdown call not happening in time in the parent process. We should improve the crash annotation to figure where it is going wrong.

Comment 1

[Andrew Osmond [:aosmond] (he/him)]

Specifically 54b and 55b had < 20% of the IPC ShutDownKill crash reports in the IPC shutdown state of "SendFinishShutdown". 56b, 57b and 58a are all >40%. This is the single largest "cause" to explain the increase in crash reports that I have been able to glean from the data thus far.

Comment 2

[Andrew Osmond [:aosmond] (he/him)]

What I hope to get from this:

1) If the content process is failing to send the message, we should consider quick exiting immediately. We cleanly shutdown but were otherwise unable to notify the parent.

2) If the parent process is failing to receive the message before we time out, we should consider increasing the timeout period slightly and seeing if that gives us enough time to receive the message we know should be in flight.

Comment 3

[Andrew Osmond [:aosmond] (he/him)]

Attachment: 0001-Bug-1406442-Improve-IPCShutdownState-annotation-for-.patch

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/22e994d66684
Improve IPCShutdownState annotation for SendFinishShutdown state to know message status. r=ehsan

Comment 5

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/22e994d66684

Comment 6

[Andrew Osmond [:aosmond] (he/him)]

It appears that they are successfully sending and the parent isn't receiving them before the timer expires.

Comment 7

[Andrew Osmond [:aosmond] (he/him)]

(In reply to Andrew Osmond [:aosmond] from comment #6)
> It appears that they are successfully sending and the parent isn't receiving
> them before the timer expires.

I should amend this after a few more days of reports. *None* of them fail to send, and *none* of them are still pending when we get killed. My instincts were that if we saw a lot of "sent" reports, and we were getting killed because it was a race with the parent's timer, then we would also see a lot of "sending" reports (because it is a race condition!). While increasing the timer by 1 second is probably a worthy experiment to try on nightly, I expect something else is going wrong. Maybe the parent isn't getting woken up by the child's message.