Last Comment Bug 1185499 - Please split Linux debug xpcshell into at least 2 chunks
: Please split Linux debug xpcshell into at least 2 chunks
Status: RESOLVED FIXED
:
Product: Release Engineering
Classification: Other
Component: General Automation (show other bugs)
: unspecified
: Unspecified Unspecified
-- critical (vote)
: ---
Assigned To: Jonathan Griffin (:jgriffin)
: Chris AtLee [:catlee]
:
Mentors:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2015-07-20 05:53 PDT by Ryan VanderMeulen [:RyanVM]
Modified: 2015-07-27 10:55 PDT (History)
4 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
Split linux32/64 debug xpcshell into two chunks, (3.14 KB, patch)
2015-07-20 09:51 PDT, Jonathan Griffin (:jgriffin)
no flags Details | Diff | Splinter Review
Split linux32/64 debug xpcshell into two chunks, (11.79 KB, patch)
2015-07-20 11:59 PDT, Jonathan Griffin (:jgriffin)
catlee: review+
jgriffin: checked‑in+
Details | Diff | Splinter Review

Description User image Ryan VanderMeulen [:RyanVM] 2015-07-20 05:53:52 PDT
Linux32 debug xpcshell is routinely hitting 7200s timeouts these days (see bug 1073442). In order to keep this suite unhidden, we need to split it into chunks. Linux64 is running consistently at 100+ min as well, so we might as well do both at the same time.

This is high-priority as we'll have to hide the suite otherwise.
Comment 1 User image Jonathan Griffin (:jgriffin) 2015-07-20 09:51:38 PDT
Created attachment 8636073 [details] [diff] [review]
Split linux32/64 debug xpcshell into two chunks,

Builders added:
+ Ubuntu VM 12.04 alder debug test xpcshell-1
buntu VM 12.04 x64 mozilla-central debug test xpcshell
VM 12.04 ash debug test xpcshell-1
nd debug test xpcshell
ebug test xpcshell-2
 oak debug test xpcshell
ound debug test xpcshell-1
ebug test xpcshell
2g-inbound debug test xpcshell-2
+ Ubuntu VM 12.04 cedar debug test xpcshell-1
+ Ubuntu VM 12.04 cedar debug test xpcshell-2
+ Ubuntu VM 12.04 elm debug test xpcshell-1
+ Ubuntu VM 12.04 elm debug test xpcshell-2
+ Ubuntu VM 12.04 fx-team debug test xpcshell-1
+ Ubuntu VM 12.04 fx-team debug test xpcshell-2
+ Ubuntu VM 12.04 gum debug test xpcshell-1
+ Ubuntu VM 12.04 gum debug test xpcshell-2
+ Ubuntu VM 12.04 holly debug test xpcshell-1
+ Ubuntu VM 12.04 holly debug test xpcshell-2
+ Ubuntu VM 12.04 jamun debug test xpcshell-1
+ Ubuntu VM 12.04 jamun debug test xpcshell-2
+ Ubuntu VM 12.04 maple debug test xpcshell-1
+ Ubuntu VM 12.04 maple debug test xpcshell-2
+ Ubuntu VM 12.04 mozilla-aurora debug test xpcshell-1
+ Ubuntu VM 12.04 mozilla-aurora debug test xpcshell-2
+ Ubuntu VM 12.04 mozilla-central debug test xpcshell-1
+ Ubuntu VM 12.04 mozilla-central debug test xpcshell-2
+ Ubuntu VM 12.04 mozilla-inbound debug test xpcshell-1
+ Ubuntu VM 12.04 mozilla-inbound debug test xpcshell-2
+ Ubuntu VM 12.04 oak debug test xpcshell-1
+ Ubuntu VM 12.04 oak debug test xpcshell-2
+ Ubuntu VM 12.04 try debug test xpcshell-1
+ Ubuntu VM 12.04 try debug test xpcshell-2
+ Ubuntu VM 12.04 x64 alder debug test xpcshell-1
+ Ubuntu VM 12.04 x64 alder debug test xpcshell-2
+ Ubuntu VM 12.04 x64 ash debug test xpcshell-1
+ Ubuntu VM 12.04 x64 ash debug test xpcshell-2
+ Ubuntu VM 12.04 x64 b2g-inbound debug test xpcshell-1
+ Ubuntu VM 12.04 x64 b2g-inbound debug test xpcshell-2
+ Ubuntu VM 12.04 x64 cedar debug test xpcshell-1
+ Ubuntu VM 12.04 x64 cedar debug test xpcshell-2
+ Ubuntu VM 12.04 x64 elm debug test xpcshell-1
+ Ubuntu VM 12.04 x64 elm debug test xpcshell-2
+ Ubuntu VM 12.04 x64 fx-team debug test xpcshell-1
+ Ubuntu VM 12.04 x64 fx-team debug test xpcshell-2
+ Ubuntu VM 12.04 x64 gum debug test xpcshell-1
+ Ubuntu VM 12.04 x64 gum debug test xpcshell-2
+ Ubuntu VM 12.04 x64 holly debug test xpcshell-1
+ Ubuntu VM 12.04 x64 holly debug test xpcshell-2
+ Ubuntu VM 12.04 x64 jamun debug test xpcshell-1
+ Ubuntu VM 12.04 x64 jamun debug test xpcshell-2
+ Ubuntu VM 12.04 x64 maple debug test xpcshell-1
+ Ubuntu VM 12.04 x64 maple debug test xpcshell-2
+ Ubuntu VM 12.04 x64 mozilla-aurora debug test xpcshell-1
+ Ubuntu VM 12.04 x64 mozilla-aurora debug test xpcshell-2
+ Ubuntu VM 12.04 x64 mozilla-central debug test xpcshell-1
+ Ubuntu VM 12.04 x64 mozilla-central debug test xpcshell-2
+ Ubuntu VM 12.04 x64 mozilla-inbound debug test xpcshell-1
+ Ubuntu VM 12.04 x64 mozilla-inbound debug test xpcshell-2
+ Ubuntu VM 12.04 x64 oak debug test xpcshell-1
+ Ubuntu VM 12.04 x64 oak debug test xpcshell-2
+ Ubuntu VM 12.04 x64 try debug test xpcshell-1
+ Ubuntu VM 12.04 x64 try debug test xpcshell-2
Builders removed
- Ubuntu VM 12.04 alder debug test xpcshell
- Ubuntu VM 12.04 ash debug test xpcshell
- Ubuntu VM 12.04 b2g-inbound debug test xpcshell
- Ubuntu VM 12.04 cedar debug test xpcshell
- Ubuntu VM 12.04 elm debug test xpcshell
- Ubuntu VM 12.04 fx-team debug test xpcshell
- Ubuntu VM 12.04 gum debug test xpcshell
- Ubuntu VM 12.04 jamun debug test xpcshell
- Ubuntu VM 12.04 maple debug test xpcshell
- Ubuntu VM 12.04 mozilla-aurora debug test xpcshell
- Ubuntu VM 12.04 mozilla-central debug test xpcshell
- Ubuntu VM 12.04 mozilla-inbound debug test xpcshell
- Ubuntu VM 12.04 oak debug test xpcshell
- Ubuntu VM 12.04 try debug test xpcshell
- Ubuntu VM 12.04 x64 alder debug test xpcshell
- Ubuntu VM 12.04 x64 ash debug test xpcshell
- Ubuntu VM 12.04 x64 b2g-inbound debug test xpcshell
- Ubuntu VM 12.04 x64 cedar debug test xpcshell
- Ubuntu VM 12.04 x64 elm debug test xpcshell
- Ubuntu VM 12.04 x64 fx-team debug test xpcshell
- Ubuntu VM 12.04 x64 gum debug test xpcshell
- Ubuntu VM 12.04 x64 jamun debug test xpcshell
- Ubuntu VM 12.04 x64 maple debug test xpcshell
- Ubuntu VM 12.04 x64 mozilla-aurora debug test xpcshell
- Ubuntu VM 12.04 x64 mozilla-central debug test xpcshell
- Ubuntu VM 12.04 x64 mozilla-inbound debug test xpcshell
- Ubuntu VM 12.04 x64 oak debug test xpcshell
- Ubuntu VM 12.04 x64 try debug test xpcshell
Comment 2 User image Jonathan Griffin (:jgriffin) 2015-07-20 09:54:21 PDT
Uh, not sure what happened to that list, here it is again:

Builders added:
+ Ubuntu VM 12.04 alder debug test xpcshell-1
+ Ubuntu VM 12.04 alder debug test xpcshell-2
+ Ubuntu VM 12.04 ash debug test xpcshell-1
+ Ubuntu VM 12.04 ash debug test xpcshell-2
+ Ubuntu VM 12.04 b2g-inbound debug test xpcshell-1
+ Ubuntu VM 12.04 b2g-inbound debug test xpcshell-2
+ Ubuntu VM 12.04 cedar debug test xpcshell-1
+ Ubuntu VM 12.04 cedar debug test xpcshell-2
+ Ubuntu VM 12.04 elm debug test xpcshell-1
+ Ubuntu VM 12.04 elm debug test xpcshell-2
+ Ubuntu VM 12.04 fx-team debug test xpcshell-1
+ Ubuntu VM 12.04 fx-team debug test xpcshell-2
+ Ubuntu VM 12.04 gum debug test xpcshell-1
+ Ubuntu VM 12.04 gum debug test xpcshell-2
+ Ubuntu VM 12.04 holly debug test xpcshell-1
+ Ubuntu VM 12.04 holly debug test xpcshell-2
+ Ubuntu VM 12.04 jamun debug test xpcshell-1
+ Ubuntu VM 12.04 jamun debug test xpcshell-2
+ Ubuntu VM 12.04 maple debug test xpcshell-1
+ Ubuntu VM 12.04 maple debug test xpcshell-2
+ Ubuntu VM 12.04 mozilla-aurora debug test xpcshell-1
+ Ubuntu VM 12.04 mozilla-aurora debug test xpcshell-2
+ Ubuntu VM 12.04 mozilla-central debug test xpcshell-1
+ Ubuntu VM 12.04 mozilla-central debug test xpcshell-2
+ Ubuntu VM 12.04 mozilla-inbound debug test xpcshell-1
+ Ubuntu VM 12.04 mozilla-inbound debug test xpcshell-2
+ Ubuntu VM 12.04 oak debug test xpcshell-1
+ Ubuntu VM 12.04 oak debug test xpcshell-2
+ Ubuntu VM 12.04 try debug test xpcshell-1
+ Ubuntu VM 12.04 try debug test xpcshell-2
+ Ubuntu VM 12.04 x64 alder debug test xpcshell-1
+ Ubuntu VM 12.04 x64 alder debug test xpcshell-2
+ Ubuntu VM 12.04 x64 ash debug test xpcshell-1
+ Ubuntu VM 12.04 x64 ash debug test xpcshell-2
+ Ubuntu VM 12.04 x64 b2g-inbound debug test xpcshell-1
+ Ubuntu VM 12.04 x64 b2g-inbound debug test xpcshell-2
+ Ubuntu VM 12.04 x64 cedar debug test xpcshell-1
+ Ubuntu VM 12.04 x64 cedar debug test xpcshell-2
+ Ubuntu VM 12.04 x64 elm debug test xpcshell-1
+ Ubuntu VM 12.04 x64 elm debug test xpcshell-2
+ Ubuntu VM 12.04 x64 fx-team debug test xpcshell-1
+ Ubuntu VM 12.04 x64 fx-team debug test xpcshell-2
+ Ubuntu VM 12.04 x64 gum debug test xpcshell-1
+ Ubuntu VM 12.04 x64 gum debug test xpcshell-2
+ Ubuntu VM 12.04 x64 holly debug test xpcshell-1
+ Ubuntu VM 12.04 x64 holly debug test xpcshell-2
+ Ubuntu VM 12.04 x64 jamun debug test xpcshell-1
+ Ubuntu VM 12.04 x64 jamun debug test xpcshell-2
+ Ubuntu VM 12.04 x64 maple debug test xpcshell-1
+ Ubuntu VM 12.04 x64 maple debug test xpcshell-2
+ Ubuntu VM 12.04 x64 mozilla-aurora debug test xpcshell-1
+ Ubuntu VM 12.04 x64 mozilla-aurora debug test xpcshell-2
+ Ubuntu VM 12.04 x64 mozilla-central debug test xpcshell-1
+ Ubuntu VM 12.04 x64 mozilla-central debug test xpcshell-2
+ Ubuntu VM 12.04 x64 mozilla-inbound debug test xpcshell-1
+ Ubuntu VM 12.04 x64 mozilla-inbound debug test xpcshell-2
+ Ubuntu VM 12.04 x64 oak debug test xpcshell-1
+ Ubuntu VM 12.04 x64 oak debug test xpcshell-2
+ Ubuntu VM 12.04 x64 try debug test xpcshell-1
+ Ubuntu VM 12.04 x64 try debug test xpcshell-2
Builders removed
- Ubuntu VM 12.04 alder debug test xpcshell
- Ubuntu VM 12.04 ash debug test xpcshell
- Ubuntu VM 12.04 b2g-inbound debug test xpcshell
- Ubuntu VM 12.04 cedar debug test xpcshell
- Ubuntu VM 12.04 elm debug test xpcshell
- Ubuntu VM 12.04 fx-team debug test xpcshell
- Ubuntu VM 12.04 gum debug test xpcshell
- Ubuntu VM 12.04 jamun debug test xpcshell
- Ubuntu VM 12.04 maple debug test xpcshell
- Ubuntu VM 12.04 mozilla-aurora debug test xpcshell
- Ubuntu VM 12.04 mozilla-central debug test xpcshell
- Ubuntu VM 12.04 mozilla-inbound debug test xpcshell
- Ubuntu VM 12.04 oak debug test xpcshell
- Ubuntu VM 12.04 try debug test xpcshell
- Ubuntu VM 12.04 x64 alder debug test xpcshell
- Ubuntu VM 12.04 x64 ash debug test xpcshell
- Ubuntu VM 12.04 x64 b2g-inbound debug test xpcshell
- Ubuntu VM 12.04 x64 cedar debug test xpcshell
- Ubuntu VM 12.04 x64 elm debug test xpcshell
- Ubuntu VM 12.04 x64 fx-team debug test xpcshell
- Ubuntu VM 12.04 x64 gum debug test xpcshell
- Ubuntu VM 12.04 x64 jamun debug test xpcshell
- Ubuntu VM 12.04 x64 maple debug test xpcshell
- Ubuntu VM 12.04 x64 mozilla-aurora debug test xpcshell
- Ubuntu VM 12.04 x64 mozilla-central debug test xpcshell
- Ubuntu VM 12.04 x64 mozilla-inbound debug test xpcshell
- Ubuntu VM 12.04 x64 oak debug test xpcshell
- Ubuntu VM 12.04 x64 try debug test xpcshell
Comment 3 User image Jonathan Griffin (:jgriffin) 2015-07-20 09:59:39 PDT
try run to make sure nothing blows up when xpcshell is chunked:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=2537d27619f4
Comment 4 User image Chris AtLee [:catlee] 2015-07-20 11:33:26 PDT
Comment on attachment 8636073 [details] [diff] [review]
Split linux32/64 debug xpcshell into two chunks,

Review of attachment 8636073 [details] [diff] [review]:
-----------------------------------------------------------------

How hard would it be to flip this around so that the multiple chunk configuration is the default, and we make exceptions for older branches to run the single chunk config?
Comment 5 User image Jonathan Griffin (:jgriffin) 2015-07-20 11:59:11 PDT
Created attachment 8636143 [details] [diff] [review]
Split linux32/64 debug xpcshell into two chunks,

Here's the patch which defaults to two chunks on linux32/64 debug, but special-cases gecko < 41.
Comment 6 User image Jonathan Griffin (:jgriffin) 2015-07-20 14:51:34 PDT
Sadly, this causes X2 to go perma-fail:  https://treeherder.mozilla.org/#/jobs?repo=try&revision=2537d27619f4

Bill, any idea why splitting xpcshell into two chunks would cause this?  Is there possibly a dependency between extensions/cookie/test/unit_ipc/test_parent.js and another xpcshell test?
Comment 7 User image Bill McCloskey (:billm) 2015-07-20 15:04:52 PDT
I think this is a recent regression from bug 1184397. It looks like we initialize the permissions manager in both the parent and the child:
http://mxr.mozilla.org/mozilla-central/source/extensions/cookie/nsPermissionManager.cpp#656
(note the IsChildProcessCheck).

However, ::Observe asserts that it only runs in the parent. However, xpcom-shutdown can fire in the child.

Ehsan, could you sort this out?
Comment 8 User image :Ehsan Akhgari 2015-07-20 15:36:42 PDT
Please push to try again, your previous push is on a broken patch that was backed out.
Comment 9 User image Jonathan Griffin (:jgriffin) 2015-07-20 15:52:19 PDT
Thanks, rebased and pushed to try again: https://treeherder.mozilla.org/#/jobs?repo=try&revision=8e8154fda0da
Comment 10 User image Jonathan Griffin (:jgriffin) 2015-07-21 09:25:42 PDT
The try runs look good now.
Comment 11 User image Jonathan Griffin (:jgriffin) 2015-07-21 09:28:51 PDT
Comment on attachment 8636143 [details] [diff] [review]
Split linux32/64 debug xpcshell into two chunks,

https://hg.mozilla.org/build/buildbot-configs/rev/3701f9b8de82
Comment 12 User image Chris Cooper [:coop] 2015-07-22 07:58:10 PDT
In production: https://hg.mozilla.org/build/buildbot-configs/rev/3701f9b8de82
Comment 13 User image Chris Cooper [:coop] 2015-07-22 07:58:12 PDT
In production: https://hg.mozilla.org/build/buildbot-configs/rev/8f9ec68b1af7

Note You need to log in before you can comment on or make changes to this bug.