1186966 - Let e10s tests ride the trains

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Basically, this is bug 1177854 but locked to version 41. We've decided that's the version we should go with.

Comment 1

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Now we're looking at 42.

Comment 2

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Attachment: tests-ride-train

We're doing our beta experiment on Firefox 44, so I think it makes sense for this to be the first release where tests ride the trains. I'll do a try push to verify that e10s tests pass on beta right now.

Comment 3

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

It looks like a lot of stuff has changed since I last ran tests like this. Rail, do you know how to simulate running e10s tests on beta? I just need to pass the --e10s option to mochitests and reftests.

Comment 4

[Rail Aliiev [:rail]]

(In reply to Bill McCloskey (:billm) from comment #3)
> It looks like a lot of stuff has changed since I last ran tests like this.
> Rail, do you know how to simulate running e10s tests on beta? I just need to
> pass the --e10s option to mochitests and reftests.

I'm not quite sure what you mean by "simulating". Do you want to test them on try or locally? Try is optimized for m-c, so m-b based pushes may be not reliable.

Comment 5

[Rail Aliiev [:rail]]

Comment on attachment 8701594 [details] [diff] [review]
tests-ride-train

lgtm

Comment 6

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

https://hg.mozilla.org/build/buildbot-configs/rev/ffed8ef8ab24

Comment 7

[Jonathan Howard]

Probably dupe bug 1219342 (or maybe following step if this not covering enabling.)

Bug 1230196 likely needs resolving on 44 before the buildbot change gets enabled.

Comment 8

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Hi Bill, Rail: 

1. Given that e10s is not enabled by default in FF44 and I am halfway through the Beta44 cycle, I want to suggest disabling e10s automated tests on moz-beta branch until 44 goes live. 

2. How stable are the e10s automated tests? And what is the pass rate like? I really do not want to spend a lot of time the next few weeks, waiting for treeherder to go green and investigate e10s failures, when those issues will not necessarily block Beta44 release.

Comment 9

[Kim Moir [:kmoir] ET]

in production

Comment 10

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Brad, what do you think about comment 8? We can keep those enabled *if* we assume that e10s test results (pass or fail) do not block me from going to build the next beta on FF44. Agreed?

Comment 11

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

I just added an exclusion profile on Treeherder to hide a large swath of mochitest and wpt e10s tests on mozilla-beta since they don't appear to be passing. Those tests will need to be fixed and that exclusion profile will need to be removed when those tests become important to people.

Comment 12

[Rail Aliiev [:rail]]

(In reply to Ritu Kothari (:ritu) from comment #8)
> Hi Bill, Rail: 
> 
> 1. Given that e10s is not enabled by default in FF44 and I am halfway
> through the Beta44 cycle, I want to suggest disabling e10s automated tests
> on moz-beta branch until 44 goes live. 
> 
> 2. How stable are the e10s automated tests? And what is the pass rate like?
> I really do not want to spend a lot of time the next few weeks, waiting for
> treeherder to go green and investigate e10s failures, when those issues will
> not necessarily block Beta44 release.

Whatever best works for you and devs. I have no strong opinion here.

Comment 13

[Brad Lassey [:blassey] (use needinfo?)]

(In reply to Ritu Kothari (:ritu) from comment #8)
> Hi Bill, Rail: 
> 
> 1. Given that e10s is not enabled by default in FF44 and I am halfway
> through the Beta44 cycle, I want to suggest disabling e10s automated tests
> on moz-beta branch until 44 goes live. 
why?

Comment 14

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Hi Brad, as I mentioned in comment 8, I really do not want to spend a lot of time the next few weeks, waiting for treeherder to go green and investigate e10s failures, when those issues will not necessarily block Beta44 release. At the moment, all the tests are failing (see comment 11)

Normally test failures block us from gtb beta builds. What would e10s test failure on Beta44 imply? It means a scenario is failing in e10s mode but should it block 44.0b# from going to build?

Comment 15

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

I think I already answered all these questions on IRC yesterday. We need e10s tests on beta because some beta users are getting e10s. Turning on the tests on beta exposed bug 1236754. So I think the tests are very important.

Again, I'm sorry they were enabled in such a haphazard way.

We're trying to get some fixes into beta so that we can gather more data for the e10s experiment. My understanding is that we still have one 44 beta left in which the e10s experiment will be enabled. I'd like to get some fixes into that build, particularly bug 1236754. Probably none of this should block the next beta build, but it would be really nice to get it in by then, and hence to have green e10s tests by then.

Comment 16

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Ryan, we want to allow e10s to be enablable on beta but not on release. Right now we use this code:
https://dxr.mozilla.org/mozilla-central/source/toolkit/xre/nsAppRunner.cpp#4688
However, that doesn't seem to work in automation. Do you know of an alternate way to do this? Please redirect if there's a better person to answer.

I guess I could add a testing pref as well (similar to the OMTC testing pref), but that seems awkward.

Comment 17

[Ryan VanderMeulen [:RyanVM]]

jgriffin might be able to help

Comment 18

[Jonathan Griffin (:jgriffin)]

this requires buildbot changes; I'll enable these on aurora first to see how they do, and once they're stable there, enable them on beta. Leaving the needinfo for now so I can track.

Comment 19

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Sorry, not sure what you mean. Buildbot changes to do what?

Comment 20

[Phil Ringnalda (:philor)]

Heh.

The update channel on beta is 'default', just like on release - beta actually intentionally tries to be just like release because it wants to be a perfect test ground for what we will release with virtually no changes hours or days after it merges to release, so while there may be an answer for your question (which I don't know), mostly the answer is "if you are trying to do things differently on beta and release you're trying to do things wrong."

Comment 21

[Jonathan Griffin (:jgriffin)]

(In reply to Bill McCloskey (:billm) from comment #19)
> Sorry, not sure what you mean. Buildbot changes to do what?

"Let e10s tests ride the trains" is a different problem than "enable e10s on beta". I can help with the former, which requires buildbot changes; I don't know how to solve the latter.

Comment 22

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

(In reply to Phil Ringnalda (:philor) from comment #20)
> Heh.
> 
> The update channel on beta is 'default', just like on release - beta
> actually intentionally tries to be just like release because it wants to be
> a perfect test ground for what we will release with virtually no changes
> hours or days after it merges to release, so while there may be an answer
> for your question (which I don't know), mostly the answer is "if you are
> trying to do things differently on beta and release you're trying to do
> things wrong."

OK, thanks, it sounds like using a test pref is the right way to go.

(In reply to Jonathan Griffin (:jgriffin) from comment #21)
> (In reply to Bill McCloskey (:billm) from comment #19)
> > Sorry, not sure what you mean. Buildbot changes to do what?
> 
> "Let e10s tests ride the trains" is a different problem than "enable e10s on
> beta". I can help with the former, which requires buildbot changes; I don't
> know how to solve the latter.

I think we've got the buildbot changes handled. Thanks.

Comment 23

[Jonathan Griffin (:jgriffin)]

If you want Talos on beta, you'll also need to modify http://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla-tests/config.py#l2377

Comment 24

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

This is now fixed.