Let e10s tests ride the trains

RESOLVED FIXED

Status

Release Engineering
General
RESOLVED FIXED
3 years ago
2 months ago

People

(Reporter: billm, Assigned: billm)

Tracking

unspecified

Firefox Tracking Flags

(e10sm8+)

Details

Attachments

(1 attachment)

Basically, this is bug 1177854 but locked to version 41. We've decided that's the version we should go with.
tracking-e10s: --- → m8+
Now we're looking at 42.
Summary: Let e10s tests ride the trains starting with 41 → Let e10s tests ride the trains starting with 42
Created attachment 8701594 [details] [diff] [review]
tests-ride-train

We're doing our beta experiment on Firefox 44, so I think it makes sense for this to be the first release where tests ride the trains. I'll do a try push to verify that e10s tests pass on beta right now.
Attachment #8701594 - Flags: review?(rail)
It looks like a lot of stuff has changed since I last ran tests like this. Rail, do you know how to simulate running e10s tests on beta? I just need to pass the --e10s option to mochitests and reftests.
(In reply to Bill McCloskey (:billm) from comment #3)
> It looks like a lot of stuff has changed since I last ran tests like this.
> Rail, do you know how to simulate running e10s tests on beta? I just need to
> pass the --e10s option to mochitests and reftests.

I'm not quite sure what you mean by "simulating". Do you want to test them on try or locally? Try is optimized for m-c, so m-b based pushes may be not reliable.
Comment on attachment 8701594 [details] [diff] [review]
tests-ride-train

lgtm
Attachment #8701594 - Flags: review?(rail) → review+

Comment 7

3 years ago
Probably dupe bug 1219342 (or maybe following step if this not covering enabling.)

Bug 1230196 likely needs resolving on 44 before the buildbot change gets enabled.

Comment 8

3 years ago
Hi Bill, Rail: 

1. Given that e10s is not enabled by default in FF44 and I am halfway through the Beta44 cycle, I want to suggest disabling e10s automated tests on moz-beta branch until 44 goes live. 

2. How stable are the e10s automated tests? And what is the pass rate like? I really do not want to spend a lot of time the next few weeks, waiting for treeherder to go green and investigate e10s failures, when those issues will not necessarily block Beta44 release.
Flags: needinfo?(wmccloskey)
Flags: needinfo?(rail)

Comment 9

3 years ago
in production
Brad, what do you think about comment 8? We can keep those enabled *if* we assume that e10s test results (pass or fail) do not block me from going to build the next beta on FF44. Agreed?
Flags: needinfo?(blassey.bugs)
I just added an exclusion profile on Treeherder to hide a large swath of mochitest and wpt e10s tests on mozilla-beta since they don't appear to be passing. Those tests will need to be fixed and that exclusion profile will need to be removed when those tests become important to people.
(In reply to Ritu Kothari (:ritu) from comment #8)
> Hi Bill, Rail: 
> 
> 1. Given that e10s is not enabled by default in FF44 and I am halfway
> through the Beta44 cycle, I want to suggest disabling e10s automated tests
> on moz-beta branch until 44 goes live. 
> 
> 2. How stable are the e10s automated tests? And what is the pass rate like?
> I really do not want to spend a lot of time the next few weeks, waiting for
> treeherder to go green and investigate e10s failures, when those issues will
> not necessarily block Beta44 release.

Whatever best works for you and devs. I have no strong opinion here.
Flags: needinfo?(rail)
(In reply to Ritu Kothari (:ritu) from comment #8)
> Hi Bill, Rail: 
> 
> 1. Given that e10s is not enabled by default in FF44 and I am halfway
> through the Beta44 cycle, I want to suggest disabling e10s automated tests
> on moz-beta branch until 44 goes live. 
why?
Flags: needinfo?(blassey.bugs)
Hi Brad, as I mentioned in comment 8, I really do not want to spend a lot of time the next few weeks, waiting for treeherder to go green and investigate e10s failures, when those issues will not necessarily block Beta44 release. At the moment, all the tests are failing (see comment 11)

Normally test failures block us from gtb beta builds. What would e10s test failure on Beta44 imply? It means a scenario is failing in e10s mode but should it block 44.0b# from going to build?
Flags: needinfo?(blassey.bugs)
I think I already answered all these questions on IRC yesterday. We need e10s tests on beta because some beta users are getting e10s. Turning on the tests on beta exposed bug 1236754. So I think the tests are very important.

Again, I'm sorry they were enabled in such a haphazard way.

We're trying to get some fixes into beta so that we can gather more data for the e10s experiment. My understanding is that we still have one 44 beta left in which the e10s experiment will be enabled. I'd like to get some fixes into that build, particularly bug 1236754. Probably none of this should block the next beta build, but it would be really nice to get it in by then, and hence to have green e10s tests by then.
Flags: needinfo?(wmccloskey)
Flags: needinfo?(blassey.bugs)

Updated

3 years ago
Summary: Let e10s tests ride the trains starting with 42 → Let e10s tests ride the trains
Ryan, we want to allow e10s to be enablable on beta but not on release. Right now we use this code:
https://dxr.mozilla.org/mozilla-central/source/toolkit/xre/nsAppRunner.cpp#4688
However, that doesn't seem to work in automation. Do you know of an alternate way to do this? Please redirect if there's a better person to answer.

I guess I could add a testing pref as well (similar to the OMTC testing pref), but that seems awkward.
Flags: needinfo?(ryanvm)
jgriffin might be able to help
Flags: needinfo?(ryanvm) → needinfo?(jgriffin)
this requires buildbot changes; I'll enable these on aurora first to see how they do, and once they're stable there, enable them on beta. Leaving the needinfo for now so I can track.
Sorry, not sure what you mean. Buildbot changes to do what?
Heh.

The update channel on beta is 'default', just like on release - beta actually intentionally tries to be just like release because it wants to be a perfect test ground for what we will release with virtually no changes hours or days after it merges to release, so while there may be an answer for your question (which I don't know), mostly the answer is "if you are trying to do things differently on beta and release you're trying to do things wrong."
(In reply to Bill McCloskey (:billm) from comment #19)
> Sorry, not sure what you mean. Buildbot changes to do what?

"Let e10s tests ride the trains" is a different problem than "enable e10s on beta". I can help with the former, which requires buildbot changes; I don't know how to solve the latter.
Flags: needinfo?(jgriffin)
(In reply to Phil Ringnalda (:philor) from comment #20)
> Heh.
> 
> The update channel on beta is 'default', just like on release - beta
> actually intentionally tries to be just like release because it wants to be
> a perfect test ground for what we will release with virtually no changes
> hours or days after it merges to release, so while there may be an answer
> for your question (which I don't know), mostly the answer is "if you are
> trying to do things differently on beta and release you're trying to do
> things wrong."

OK, thanks, it sounds like using a test pref is the right way to go.

(In reply to Jonathan Griffin (:jgriffin) from comment #21)
> (In reply to Bill McCloskey (:billm) from comment #19)
> > Sorry, not sure what you mean. Buildbot changes to do what?
> 
> "Let e10s tests ride the trains" is a different problem than "enable e10s on
> beta". I can help with the former, which requires buildbot changes; I don't
> know how to solve the latter.

I think we've got the buildbot changes handled. Thanks.
Flags: needinfo?(wmccloskey)
This is now fixed.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Flags: needinfo?(wmccloskey)
Resolution: --- → FIXED
Component: General Automation → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.