The default bug view has changed. See this FAQ.

Please split Linux debug xpcshell into at least 2 chunks

RESOLVED FIXED

Status

Release Engineering
General Automation
--
critical
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: RyanVM, Assigned: jgriffin)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment, 1 obsolete attachment)

(Reporter)

Description

2 years ago
Linux32 debug xpcshell is routinely hitting 7200s timeouts these days (see bug 1073442). In order to keep this suite unhidden, we need to split it into chunks. Linux64 is running consistently at 100+ min as well, so we might as well do both at the same time.

This is high-priority as we'll have to hide the suite otherwise.
(Assignee)

Updated

2 years ago
Assignee: nobody → jgriffin
Status: NEW → ASSIGNED
(Assignee)

Comment 1

2 years ago
Created attachment 8636073 [details] [diff] [review]
Split linux32/64 debug xpcshell into two chunks,

Builders added:
+ Ubuntu VM 12.04 alder debug test xpcshell-1
buntu VM 12.04 x64 mozilla-central debug test xpcshell
VM 12.04 ash debug test xpcshell-1
nd debug test xpcshell
ebug test xpcshell-2
 oak debug test xpcshell
ound debug test xpcshell-1
ebug test xpcshell
2g-inbound debug test xpcshell-2
+ Ubuntu VM 12.04 cedar debug test xpcshell-1
+ Ubuntu VM 12.04 cedar debug test xpcshell-2
+ Ubuntu VM 12.04 elm debug test xpcshell-1
+ Ubuntu VM 12.04 elm debug test xpcshell-2
+ Ubuntu VM 12.04 fx-team debug test xpcshell-1
+ Ubuntu VM 12.04 fx-team debug test xpcshell-2
+ Ubuntu VM 12.04 gum debug test xpcshell-1
+ Ubuntu VM 12.04 gum debug test xpcshell-2
+ Ubuntu VM 12.04 holly debug test xpcshell-1
+ Ubuntu VM 12.04 holly debug test xpcshell-2
+ Ubuntu VM 12.04 jamun debug test xpcshell-1
+ Ubuntu VM 12.04 jamun debug test xpcshell-2
+ Ubuntu VM 12.04 maple debug test xpcshell-1
+ Ubuntu VM 12.04 maple debug test xpcshell-2
+ Ubuntu VM 12.04 mozilla-aurora debug test xpcshell-1
+ Ubuntu VM 12.04 mozilla-aurora debug test xpcshell-2
+ Ubuntu VM 12.04 mozilla-central debug test xpcshell-1
+ Ubuntu VM 12.04 mozilla-central debug test xpcshell-2
+ Ubuntu VM 12.04 mozilla-inbound debug test xpcshell-1
+ Ubuntu VM 12.04 mozilla-inbound debug test xpcshell-2
+ Ubuntu VM 12.04 oak debug test xpcshell-1
+ Ubuntu VM 12.04 oak debug test xpcshell-2
+ Ubuntu VM 12.04 try debug test xpcshell-1
+ Ubuntu VM 12.04 try debug test xpcshell-2
+ Ubuntu VM 12.04 x64 alder debug test xpcshell-1
+ Ubuntu VM 12.04 x64 alder debug test xpcshell-2
+ Ubuntu VM 12.04 x64 ash debug test xpcshell-1
+ Ubuntu VM 12.04 x64 ash debug test xpcshell-2
+ Ubuntu VM 12.04 x64 b2g-inbound debug test xpcshell-1
+ Ubuntu VM 12.04 x64 b2g-inbound debug test xpcshell-2
+ Ubuntu VM 12.04 x64 cedar debug test xpcshell-1
+ Ubuntu VM 12.04 x64 cedar debug test xpcshell-2
+ Ubuntu VM 12.04 x64 elm debug test xpcshell-1
+ Ubuntu VM 12.04 x64 elm debug test xpcshell-2
+ Ubuntu VM 12.04 x64 fx-team debug test xpcshell-1
+ Ubuntu VM 12.04 x64 fx-team debug test xpcshell-2
+ Ubuntu VM 12.04 x64 gum debug test xpcshell-1
+ Ubuntu VM 12.04 x64 gum debug test xpcshell-2
+ Ubuntu VM 12.04 x64 holly debug test xpcshell-1
+ Ubuntu VM 12.04 x64 holly debug test xpcshell-2
+ Ubuntu VM 12.04 x64 jamun debug test xpcshell-1
+ Ubuntu VM 12.04 x64 jamun debug test xpcshell-2
+ Ubuntu VM 12.04 x64 maple debug test xpcshell-1
+ Ubuntu VM 12.04 x64 maple debug test xpcshell-2
+ Ubuntu VM 12.04 x64 mozilla-aurora debug test xpcshell-1
+ Ubuntu VM 12.04 x64 mozilla-aurora debug test xpcshell-2
+ Ubuntu VM 12.04 x64 mozilla-central debug test xpcshell-1
+ Ubuntu VM 12.04 x64 mozilla-central debug test xpcshell-2
+ Ubuntu VM 12.04 x64 mozilla-inbound debug test xpcshell-1
+ Ubuntu VM 12.04 x64 mozilla-inbound debug test xpcshell-2
+ Ubuntu VM 12.04 x64 oak debug test xpcshell-1
+ Ubuntu VM 12.04 x64 oak debug test xpcshell-2
+ Ubuntu VM 12.04 x64 try debug test xpcshell-1
+ Ubuntu VM 12.04 x64 try debug test xpcshell-2
Builders removed
- Ubuntu VM 12.04 alder debug test xpcshell
- Ubuntu VM 12.04 ash debug test xpcshell
- Ubuntu VM 12.04 b2g-inbound debug test xpcshell
- Ubuntu VM 12.04 cedar debug test xpcshell
- Ubuntu VM 12.04 elm debug test xpcshell
- Ubuntu VM 12.04 fx-team debug test xpcshell
- Ubuntu VM 12.04 gum debug test xpcshell
- Ubuntu VM 12.04 jamun debug test xpcshell
- Ubuntu VM 12.04 maple debug test xpcshell
- Ubuntu VM 12.04 mozilla-aurora debug test xpcshell
- Ubuntu VM 12.04 mozilla-central debug test xpcshell
- Ubuntu VM 12.04 mozilla-inbound debug test xpcshell
- Ubuntu VM 12.04 oak debug test xpcshell
- Ubuntu VM 12.04 try debug test xpcshell
- Ubuntu VM 12.04 x64 alder debug test xpcshell
- Ubuntu VM 12.04 x64 ash debug test xpcshell
- Ubuntu VM 12.04 x64 b2g-inbound debug test xpcshell
- Ubuntu VM 12.04 x64 cedar debug test xpcshell
- Ubuntu VM 12.04 x64 elm debug test xpcshell
- Ubuntu VM 12.04 x64 fx-team debug test xpcshell
- Ubuntu VM 12.04 x64 gum debug test xpcshell
- Ubuntu VM 12.04 x64 jamun debug test xpcshell
- Ubuntu VM 12.04 x64 maple debug test xpcshell
- Ubuntu VM 12.04 x64 mozilla-aurora debug test xpcshell
- Ubuntu VM 12.04 x64 mozilla-central debug test xpcshell
- Ubuntu VM 12.04 x64 mozilla-inbound debug test xpcshell
- Ubuntu VM 12.04 x64 oak debug test xpcshell
- Ubuntu VM 12.04 x64 try debug test xpcshell
Attachment #8636073 - Flags: review?(catlee)
(Assignee)

Comment 2

2 years ago
Uh, not sure what happened to that list, here it is again:

Builders added:
+ Ubuntu VM 12.04 alder debug test xpcshell-1
+ Ubuntu VM 12.04 alder debug test xpcshell-2
+ Ubuntu VM 12.04 ash debug test xpcshell-1
+ Ubuntu VM 12.04 ash debug test xpcshell-2
+ Ubuntu VM 12.04 b2g-inbound debug test xpcshell-1
+ Ubuntu VM 12.04 b2g-inbound debug test xpcshell-2
+ Ubuntu VM 12.04 cedar debug test xpcshell-1
+ Ubuntu VM 12.04 cedar debug test xpcshell-2
+ Ubuntu VM 12.04 elm debug test xpcshell-1
+ Ubuntu VM 12.04 elm debug test xpcshell-2
+ Ubuntu VM 12.04 fx-team debug test xpcshell-1
+ Ubuntu VM 12.04 fx-team debug test xpcshell-2
+ Ubuntu VM 12.04 gum debug test xpcshell-1
+ Ubuntu VM 12.04 gum debug test xpcshell-2
+ Ubuntu VM 12.04 holly debug test xpcshell-1
+ Ubuntu VM 12.04 holly debug test xpcshell-2
+ Ubuntu VM 12.04 jamun debug test xpcshell-1
+ Ubuntu VM 12.04 jamun debug test xpcshell-2
+ Ubuntu VM 12.04 maple debug test xpcshell-1
+ Ubuntu VM 12.04 maple debug test xpcshell-2
+ Ubuntu VM 12.04 mozilla-aurora debug test xpcshell-1
+ Ubuntu VM 12.04 mozilla-aurora debug test xpcshell-2
+ Ubuntu VM 12.04 mozilla-central debug test xpcshell-1
+ Ubuntu VM 12.04 mozilla-central debug test xpcshell-2
+ Ubuntu VM 12.04 mozilla-inbound debug test xpcshell-1
+ Ubuntu VM 12.04 mozilla-inbound debug test xpcshell-2
+ Ubuntu VM 12.04 oak debug test xpcshell-1
+ Ubuntu VM 12.04 oak debug test xpcshell-2
+ Ubuntu VM 12.04 try debug test xpcshell-1
+ Ubuntu VM 12.04 try debug test xpcshell-2
+ Ubuntu VM 12.04 x64 alder debug test xpcshell-1
+ Ubuntu VM 12.04 x64 alder debug test xpcshell-2
+ Ubuntu VM 12.04 x64 ash debug test xpcshell-1
+ Ubuntu VM 12.04 x64 ash debug test xpcshell-2
+ Ubuntu VM 12.04 x64 b2g-inbound debug test xpcshell-1
+ Ubuntu VM 12.04 x64 b2g-inbound debug test xpcshell-2
+ Ubuntu VM 12.04 x64 cedar debug test xpcshell-1
+ Ubuntu VM 12.04 x64 cedar debug test xpcshell-2
+ Ubuntu VM 12.04 x64 elm debug test xpcshell-1
+ Ubuntu VM 12.04 x64 elm debug test xpcshell-2
+ Ubuntu VM 12.04 x64 fx-team debug test xpcshell-1
+ Ubuntu VM 12.04 x64 fx-team debug test xpcshell-2
+ Ubuntu VM 12.04 x64 gum debug test xpcshell-1
+ Ubuntu VM 12.04 x64 gum debug test xpcshell-2
+ Ubuntu VM 12.04 x64 holly debug test xpcshell-1
+ Ubuntu VM 12.04 x64 holly debug test xpcshell-2
+ Ubuntu VM 12.04 x64 jamun debug test xpcshell-1
+ Ubuntu VM 12.04 x64 jamun debug test xpcshell-2
+ Ubuntu VM 12.04 x64 maple debug test xpcshell-1
+ Ubuntu VM 12.04 x64 maple debug test xpcshell-2
+ Ubuntu VM 12.04 x64 mozilla-aurora debug test xpcshell-1
+ Ubuntu VM 12.04 x64 mozilla-aurora debug test xpcshell-2
+ Ubuntu VM 12.04 x64 mozilla-central debug test xpcshell-1
+ Ubuntu VM 12.04 x64 mozilla-central debug test xpcshell-2
+ Ubuntu VM 12.04 x64 mozilla-inbound debug test xpcshell-1
+ Ubuntu VM 12.04 x64 mozilla-inbound debug test xpcshell-2
+ Ubuntu VM 12.04 x64 oak debug test xpcshell-1
+ Ubuntu VM 12.04 x64 oak debug test xpcshell-2
+ Ubuntu VM 12.04 x64 try debug test xpcshell-1
+ Ubuntu VM 12.04 x64 try debug test xpcshell-2
Builders removed
- Ubuntu VM 12.04 alder debug test xpcshell
- Ubuntu VM 12.04 ash debug test xpcshell
- Ubuntu VM 12.04 b2g-inbound debug test xpcshell
- Ubuntu VM 12.04 cedar debug test xpcshell
- Ubuntu VM 12.04 elm debug test xpcshell
- Ubuntu VM 12.04 fx-team debug test xpcshell
- Ubuntu VM 12.04 gum debug test xpcshell
- Ubuntu VM 12.04 jamun debug test xpcshell
- Ubuntu VM 12.04 maple debug test xpcshell
- Ubuntu VM 12.04 mozilla-aurora debug test xpcshell
- Ubuntu VM 12.04 mozilla-central debug test xpcshell
- Ubuntu VM 12.04 mozilla-inbound debug test xpcshell
- Ubuntu VM 12.04 oak debug test xpcshell
- Ubuntu VM 12.04 try debug test xpcshell
- Ubuntu VM 12.04 x64 alder debug test xpcshell
- Ubuntu VM 12.04 x64 ash debug test xpcshell
- Ubuntu VM 12.04 x64 b2g-inbound debug test xpcshell
- Ubuntu VM 12.04 x64 cedar debug test xpcshell
- Ubuntu VM 12.04 x64 elm debug test xpcshell
- Ubuntu VM 12.04 x64 fx-team debug test xpcshell
- Ubuntu VM 12.04 x64 gum debug test xpcshell
- Ubuntu VM 12.04 x64 jamun debug test xpcshell
- Ubuntu VM 12.04 x64 maple debug test xpcshell
- Ubuntu VM 12.04 x64 mozilla-aurora debug test xpcshell
- Ubuntu VM 12.04 x64 mozilla-central debug test xpcshell
- Ubuntu VM 12.04 x64 mozilla-inbound debug test xpcshell
- Ubuntu VM 12.04 x64 oak debug test xpcshell
- Ubuntu VM 12.04 x64 try debug test xpcshell
(Assignee)

Comment 3

2 years ago
try run to make sure nothing blows up when xpcshell is chunked:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=2537d27619f4
Comment on attachment 8636073 [details] [diff] [review]
Split linux32/64 debug xpcshell into two chunks,

Review of attachment 8636073 [details] [diff] [review]:
-----------------------------------------------------------------

How hard would it be to flip this around so that the multiple chunk configuration is the default, and we make exceptions for older branches to run the single chunk config?
(Assignee)

Comment 5

2 years ago
Created attachment 8636143 [details] [diff] [review]
Split linux32/64 debug xpcshell into two chunks,

Here's the patch which defaults to two chunks on linux32/64 debug, but special-cases gecko < 41.
Attachment #8636143 - Flags: review?(catlee)
(Assignee)

Updated

2 years ago
Attachment #8636073 - Attachment is obsolete: true
Attachment #8636073 - Flags: review?(catlee)
(Assignee)

Comment 6

2 years ago
Sadly, this causes X2 to go perma-fail:  https://treeherder.mozilla.org/#/jobs?repo=try&revision=2537d27619f4

Bill, any idea why splitting xpcshell into two chunks would cause this?  Is there possibly a dependency between extensions/cookie/test/unit_ipc/test_parent.js and another xpcshell test?
Flags: needinfo?(wmccloskey)
I think this is a recent regression from bug 1184397. It looks like we initialize the permissions manager in both the parent and the child:
http://mxr.mozilla.org/mozilla-central/source/extensions/cookie/nsPermissionManager.cpp#656
(note the IsChildProcessCheck).

However, ::Observe asserts that it only runs in the parent. However, xpcom-shutdown can fire in the child.

Ehsan, could you sort this out?
Flags: needinfo?(wmccloskey) → needinfo?(ehsan)
Please push to try again, your previous push is on a broken patch that was backed out.
Flags: needinfo?(ehsan)
(Assignee)

Comment 9

2 years ago
Thanks, rebased and pushed to try again: https://treeherder.mozilla.org/#/jobs?repo=try&revision=8e8154fda0da

Updated

2 years ago
Attachment #8636143 - Flags: review?(catlee) → review+
(Assignee)

Comment 10

2 years ago
The try runs look good now.
(Assignee)

Comment 11

2 years ago
Comment on attachment 8636143 [details] [diff] [review]
Split linux32/64 debug xpcshell into two chunks,

https://hg.mozilla.org/build/buildbot-configs/rev/3701f9b8de82
Attachment #8636143 - Flags: checked-in+
In production: https://hg.mozilla.org/build/buildbot-configs/rev/3701f9b8de82
In production: https://hg.mozilla.org/build/buildbot-configs/rev/8f9ec68b1af7
(Assignee)

Updated

2 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.