1242119 - Intermittent xpcshell-child-process.ini:dom/indexedDB/test/unit/test_transaction_abort.js,test_multientry.js,test_blocked_order.js | application crashed [@ moz_abort]

Status: RESOLVED FIXED

(Core :: Gecko Profiler, defect)

Description

[Phil Ringnalda (:philor)]

https://treeherder.mozilla.org/logviewer.html#?job_id=20341719&repo=mozilla-inbound

Comment 1

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

It looks like we're destroying a Mutex too late, and the deadlock detector is already shut down.  Should this be using OffTheBooksMutex

Comment 2

[Nathan Froyd [:froydnj]]

I assume this is sRegisteredThreadsMutex, which we're so eager to destroy, we try to do it twice:

http://mxr.mozilla.org/mozilla-central/source/tools/profiler/core/platform.cpp#139

But on Windows, we go through:

http://mxr.mozilla.org/mozilla-central/source/xpcom/build/XPCOMInit.cpp#971

which runs before profiler_shutdown() and thereby destroys the Mutex via the UniquePtr destructor registered at static construction time, rather than explicit shutdown later on.

What I don't get (having not looked at this very long) is why are we not hitting this more often?

Comment 3

[Intermittent Failures Robot]

8 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 3
* mozilla-aurora: 2
* fx-team: 2
* mozilla-inbound: 1

Platform breakdown:
* windowsxp: 8

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1242119&startday=2016-01-25&endday=2016-01-31&tree=all

Comment 4

[Benoit Girard (:BenWa)]

IMO this is really a regression caused by bug 1035454 which introduced a huge footgun to paper of it's own issues. I wouldn't be overly surprised if there was other similar bugs caused by this but we don't have a way to filter on winxp only intermittent issues.

Ideally we would backout http://hg.mozilla.org/mozilla-central/rev/9134c098f0ee. It is stated to be temporary anyways. Otherwise we might want to consider using _exit(0) instead which IIRC wont run static destructors.

Otherwise if we want to plaster over a plaster bug we can use OffTheBooksMutex =\.

Comment 5

[Andrew McCreight [:mccr8]]

(In reply to Benoit Girard (:BenWa) from comment #4)
> Ideally we would backout
> http://hg.mozilla.org/mozilla-central/rev/9134c098f0ee. It is stated to be
> temporary anyways.

I can see if the issue in bug 1083664 went away, and if it still around, see if the WebRTC people have a preference between investigating the issue and disabling the test on XP.

> Otherwise we might want to consider using _exit(0)
> instead which IIRC wont run static destructors.

Ah, good point. That's what ContentChild::QuickExit() does.

Comment 6

[Andrew McCreight [:mccr8]]

changing to _exit() seems to work:
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=4be9c404009d
But the WebRTC intermittent failure seems to have gone away, so I'll try just removing this chunk of code once I've worked out a leak suppression that works. (Removing the block causes us to run leak checking on WindowsXP, and we don't run it at all in m-c right now, so there are additional leaks revealed.)

Comment 7

[Andrew McCreight [:mccr8]]

Attachment: part 1 - Remove early exit in WinXP debug content processes.

This was disabled because it was causing intermittent failures in a test, but that failure seems to have stopped.

This will cause us to start doing leak checking in content processes on Windows XP. (We do not run them on other Windows platforms either due to bug 1219369.)

try run with a bunch of retriggers for the test suite with the WebRTC failure:

  https://treeherder.mozilla.org/#/jobs?repo=try&revision=edf6c3934483

The oranges are all leaks, because this patch makes it so that we actually run leak checking on Windows XP. I'm adding suppressions for these leaks in bug 1219919.

Here is a try run with those suppressions added:
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=bdc3933abda0

Comment 8

[Andrew McCreight [:mccr8]]

Attachment: part 2 - Use _exit(0) to exit in B2G debug content processes.

This skips destructors, so hopefully it avoids some odd behaviors. My attempt to do a try run for this failed somehow, but it seems better than leaving it alone. I split this into a separate patch to make it easier to back out one without backing out the other. I also updated the comment.

Comment 9

[Andrew McCreight [:mccr8]]

I'm marking this MemShrink because the patch is needed to run leak checking on Windows XP content processes.

Comment 10

[Nathan Froyd [:froydnj]]

Comment on attachment 8715026 [details] [diff] [review]
part 2 - Use _exit(0) to exit in B2G debug content processes.

Review of attachment 8715026 [details] [diff] [review]:
-----------------------------------------------------------------

Why are we changing the B2G behavior in a Windows intermittent bug?  For consistency with the (former) Windows implementation?

r=me with answers to the above.

Comment 11

[Andrew McCreight [:mccr8]]

(In reply to Nathan Froyd [:froydnj] from comment #10)
> Why are we changing the B2G behavior in a Windows intermittent bug?  For
> consistency with the (former) Windows implementation?

I'll file a separate bug for part 2. Benoit just pointed out in this bug that exit() instead of _exit() is probably a bad idea, so I figured I'd fix it.

Comment 12

[Andrew McCreight [:mccr8]]

Filed bug 1245513 for the B2G changes.

Comment 13

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/f6c9abced33d

Comment 14

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/f6c9abced33d

Comment 15

[Intermittent Failures Robot]

5 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-aurora: 2
* try: 1
* mozilla-central: 1
* fx-team: 1

Platform breakdown:
* windowsxp: 5

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1242119&startday=2016-02-01&endday=2016-02-07&tree=all