1303113 - [meta] Turn e10s-multi on in Nightly

Status: RESOLVED FIXED

(Core :: General, defect)

Description

[Blake Kaplan (:mrbkap) (inactive)]

We would like to turn e10s-multi on in Nightly (we'll start with 2 content processes and increase from there).

One question Erin asked over email is if we want to segment the Nightly population. It would probably make sense to do so. We should also probably figure out what percentage of Nightly users are already using processCount > 1.

Comment 1

[Gabor Krizsanits (INACTIVE)]

Attachment: Turning 2 content processes in Nightly. v1

This patch as we discussed turns it on for the full population. Since the goal is to get more bug reports and the purpose of nightly is to test features like this I would not split the population. We can do that later on aurora or beta when we want to get the performance numbers / crash stats before release.

Comment 2

[Gabor Krizsanits (INACTIVE)]

Ryan, could you please take a look at the current state: https://treeherder.mozilla.org/#/jobs?repo=try&revision=3b2458ebae8d&selectedJob=30546722

Some of the intermittents we had might have got a bit worse, it's hard to tell. I've re-triggered some tests. Could you help me to decide if we're good to go or if we should disable some of them and re-enable them later?

Comment 3

[Gabor Krizsanits (INACTIVE)]

Oh, and ignore test_browserElement_oop_PrivateBrowsing.html failures, it should have been disabled already...

Comment 4

[Gabor Krizsanits (INACTIVE)]

Bc4 on linux debug and bc7 on linux 64 debug seem a bit too orange, but the failing intermittents typically timeouts from known intermittents, not sure what to do about them.

Comment 5

[Ryan VanderMeulen [:RyanVM]]

We discussed on this IRC for a bit. The linux32 browser_hsts-priming failures are extremely frequent on m-c tip as well (and probably heading for disabling anyway). There are some leaks that are still concerning, and it's not clear what the situation on linux64 debug will be once browser_tab_dragdrop2.js and browser_tabkeynavigation.js get sorted out. Anyway, I think we're getting close, but it would be good to take another look after some of those bigger issues get cleaned up.

Comment 6

[Gabor Krizsanits (INACTIVE)]

BC7 failures were all over the place with the last push: https://treeherder.mozilla.org/#/jobs?repo=try&revision=38f16c7a63ffba51bfdbcf4ddba5b4153792b4b6&selectedJob=30678695

but I think it was unrelated, because today after a rebased patch the issue is gone:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=6999b4394c7fe32f78b0e78695034023f7651a7e

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/085035586d2b
Turn e10s-multi on in Nightly. r=mrbkap

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/adfcc194af1c
Turning off a test for e10s-multi temporarily. r=me

Comment 9

[Phil Ringnalda (:philor)]

Backed out in https://hg.mozilla.org/integration/mozilla-inbound/rev/e1f230913306b5ab63a64438c12f76e8fbde8c62 for landing just 5 days too early.

Originally, 52 was going to merge to aurora on the 7th and now would have been fine, but it got pushed back a week so now this landed and massively destabilized our tests and shut off quite a few of them for both single and multi-e10s right before next Monday's merge, while people are trying to shove in things at the last minute that they actually want to ship in that branch which is going to be an ESR, so we really *really* don't want to disable tests that we will never remember to reenable on aurora where we'll be back to single-e10s.

Happy to see this land again next Monday morning (Pacific time Monday morning, you can tell it's safe to land when you see a mozilla-central push talking about tagging and bumping the version number), even happier if by then it doesn't have some of the bustage that is already apparent from it, like the leak in https://treeherder.mozilla.org/logviewer.html#?job_id=38930104&repo=mozilla-inbound (one of the clearest examples of why I don't want to see it landed right now, because that failure is bustage which landed a few pushes before you, and then your leak hiding below it), the failure in https://treeherder.mozilla.org/logviewer.html#?job_id=38930048&repo=mozilla-inbound, the leaks for sure and probably the timeout itself in https://treeherder.mozilla.org/logviewer.html#?job_id=38931136&repo=mozilla-inbound, and the failures in https://treeherder.mozilla.org/logviewer.html#?job_id=38938876&repo=mozilla-inbound

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

browser_tab_dragdrop2.js  was fixed by turning browser_tab_dragdrop.js off (it was enabled and caused browser_tab_dragdrop2.js  to fail very frequently)

the hsts tests are really frustrating, we disabled all of them for linux32-debug this week, and now linux64-debug is failing very frequently as of yesterday (possibly related to backing this out?)

Comment 11

[Joel Maher ( :jmaher ) (UTC -8)]

as a note this had a large impact on talos:
== Change summary for alert #4055 (as of November 09 2016 13:46 UTC) ==

Regressions:

270%  tabpaint summary osx-10-10 opt e10s       58.11 -> 214.79
263%  tabpaint summary windowsxp pgo e10s       50.45 -> 183.01
223%  tabpaint summary windows7-32 pgo e10s     49.69 -> 160.26
217%  tabpaint summary linux64 pgo e10s         59.5 -> 188.54
214%  tabpaint summary linux64 opt e10s         68.86 -> 216.27
205%  tabpaint summary windows8-64 opt e10s     59.03 -> 180.29
196%  tabpaint summary windowsxp opt e10s       73.17 -> 216.68
177%  tabpaint summary windows7-32 opt e10s     71.44 -> 197.61
 67%  tps summary windows7-32 pgo e10s          33.25 -> 55.64
 58%  tps summary windows7-32 opt e10s          40.02 -> 63.24
 42%  tps summary osx-10-10 opt e10s            39.78 -> 56.29
 27%  tps summary windowsxp opt e10s            39.5 -> 50.02
 25%  tps summary windowsxp pgo e10s            33.74 -> 42.19
 16%  damp summary windows7-32 pgo e10s         222.94 -> 259.04
 14%  damp summary windows8-64 opt e10s         267.11 -> 303.89
 13%  damp summary osx-10-10 opt e10s           302.7 -> 342.71
 11%  damp summary linux64 opt e10s             293.99 -> 326.18
 10%  damp summary linux64 pgo e10s             245.15 -> 270.28
  9%  tps summary linux64 opt e10s              41.7 -> 45.47
  8%  damp summary windows7-32 opt e10s         300.24 -> 325.07
  8%  tps summary linux64 pgo e10s              36.23 -> 39.22

Improvements:

  3%  tps summary windows8-64 opt e10s     37.1 -> 35.86

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=4055


please address these talos issues prior to landing this again.

Comment 12

[Gabor Krizsanits (INACTIVE)]

(In reply to Joel Maher ( :jmaher) from comment #11)
> please address these talos issues prior to landing this again.

Let's branch this off: bug 1317312.

Comment 13

[Gabor Krizsanits (INACTIVE)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=55581ed910f5a53008bcc5e0bfe35a2cd1e4a51a&selectedJob=65004753

I'm going to do another try now that the talos regressions are fixed. I had to fix browser_service_workers_status.js and force single cp in one more test, otherwise this patch
is pretty much what I tried the last time. Then I will work on the new intermittent failures
a bit if it's necessary and re-enable the tests.

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/2bd53e4e662b
Turn e10s-multi on in Nightly. r=me

Comment 15

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Backed out for various test failures in tests for debug builds with e10s:

https://hg.mozilla.org/integration/mozilla-inbound/rev/0bbd4e32a321fd93c963155185fbbf015e82dd85

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=2bd53e4e662bcdd32c53cb4e09ceff088e8f6369

Comment 16

[Joel Maher ( :jmaher ) (UTC -8)]

as a note, this caused a performance regression in the damp (devtools) test:
== Change summary for alert #4699 (as of January 03 2017 17:16 UTC) ==

Regressions:

 12%  damp summary linux64 opt e10s     329.61 -> 369.16

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=4699

as this is backed out, we are not filing a new bug.

Comment 17

[Gabor Krizsanits (INACTIVE)]

(In reply to Joel Maher ( :jmaher) from comment #16)
> as a note, this caused a performance regression in the damp (devtools) test:
> == Change summary for alert #4699 (as of January 03 2017 17:16 UTC) ==
> 
> Regressions:
> 
>  12%  damp summary linux64 opt e10s     329.61 -> 369.16
> 
> For up to date results, see:
> https://treeherder.mozilla.org/perf.html#/alerts?id=4699

I think we will have to swallow this. But I will take a look at it if I can do any improvement other then
making sure that devtools do not start a new process (which might be what we will want in the end).

Comment 18

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

I backed this out for failures like https://treeherder.mozilla.org/logviewer.html#?job_id=70408216&repo=mozilla-inbound

https://hg.mozilla.org/integration/mozilla-inbound/rev/bde3fc40b9b5

Comment 19

[Gabor Krizsanits (INACTIVE)]

Hey Kris, do you know anything about these oop extension tests and why they fail with multiple content processes? Also, do you mind if I turn them off temporarily and once they fixed we can turn them back on again? This is blocking us.

Comment 20

[Gabor Krizsanits (INACTIVE)]

(In reply to Wes Kocher (:KWierso) from comment #18)
> I backed this out for failures like
> https://treeherder.mozilla.org/logviewer.html#?job_id=70408216&repo=mozilla-
> inbound
> 
> https://hg.mozilla.org/integration/mozilla-inbound/rev/bde3fc40b9b5

I have not seen this on ash (might have overlooked it), is this a frequent/perma orange or a seen once intermittent?

Comment 21

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

(In reply to Gabor Krizsanits [:krizsa :gabor] from comment #20)
> (In reply to Wes Kocher (:KWierso) from comment #18)
> > I backed this out for failures like
> > https://treeherder.mozilla.org/logviewer.html#?job_id=70408216&repo=mozilla-
> > inbound
> > 
> > https://hg.mozilla.org/integration/mozilla-inbound/rev/bde3fc40b9b5
> 
> I have not seen this on ash (might have overlooked it), is this a
> frequent/perma orange or a seen once intermittent?

It's essentially permafailing, at least on Windows debug: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&fromchange=e2f6478f748157bf82a5fd0e940a6043af076a77&bugfiler&group_state=expanded&noautoclassify&filter-searchStr=windows%20mochi%20s(5%20debug

Comment 22

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

There were a couple failures on linux as well: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&fromchange=e2f6478f748157bf82a5fd0e940a6043af076a77&bugfiler&group_state=expanded&noautoclassify&filter-searchStr=mochi%20s(10%20debug&selectedJob=70397287

Comment 23

[Kris Maglione [:kmag]]

The test_ext_cookies failure looks like a manifestation of bug 1309637. I'd be OK with disabling that test in Windows debug builds until that bug is fixed, but I'm a bit worried that e10s multi triggering that bug.

In the case of test_ext_storage_content, it looks like we're somehow running the test twice, in parallel, and the two instances are conflicting. I can't produce this locally on Linux, but it's definitely worrisome, and we can't just disable this test. I'll look into it some more.

I suspect the test_ext_i18n and test_ext_unload_frame problems may be similar, but I can't reproduce them locally either.

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/0c891a3aff93
Turn e10s-multi on in Nightly. r=me

Comment 25

[Kris Maglione [:kmag]]

OK, it actually looks like this is a cascade of failures caused by the test_ext_cookies timeout, that leads to extra windows and tabs staying open, and breaking the other tests. So let's just disable that test on Windows debug builds.

Comment 26

[Gabor Krizsanits (INACTIVE)]

(In reply to Pulsebot from comment #24)
> Pushed by [e-mail]:
> https://hg.mozilla.org/integration/mozilla-inbound/rev/0c891a3aff93
> Turn e10s-multi on in Nightly. r=me

Just to make this clear, this is the one that has got backed out, just Pulsebot was slower than the backout.

(In reply to Kris Maglione [:kmag] from comment #25)
> OK, it actually looks like this is a cascade of failures caused by the
> test_ext_cookies timeout, that leads to extra windows and tabs staying open,
> and breaking the other tests. So let's just disable that test on Windows
> debug builds.

Thanks Kris, I'll do that. Although it seems like this failure might happen on linux as well (as Wes pointed it out), so I might have to turn it off for all debug builds temporarily.

Comment 27

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/529ae909938a
Turn e10s-multi on in Nightly. r=me

Comment 28

[Gabor Krizsanits (INACTIVE)]

(In reply to Gabor Krizsanits [:krizsa :gabor] from comment #26)
> Thanks Kris, I'll do that. Although it seems like this failure might happen
> on linux as well (as Wes pointed it out), so I might have to turn it off for
> all debug builds temporarily.

Actually I see failures in release mode as well: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&selectedJob=70726841

Not too frequent yet... might have to do a followup and turn it off in release mode as well. It would be nice if there were a way to turn off the oop version only...

Comment 29

[Kris Maglione [:kmag]]

(In reply to Gabor Krizsanits [:krizsa :gabor] from comment #28)
> Not too frequent yet... might have to do a followup and turn it off in
> release mode as well. It would be nice if there were a way to turn off the
> oop version only...

There is, but the failures aren't only in OOP mode. Those just happen to show up first.

Comment 30

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

I had to back this out for frequent asan failures like https://treeherder.mozilla.org/logviewer.html#?job_id=70784059&repo=mozilla-inbound

https://hg.mozilla.org/integration/mozilla-inbound/rev/dbef9f0ee2d5

Comment 31

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Not just asan: https://treeherder.mozilla.org/logviewer.html#?job_id=70787548&repo=mozilla-inbound

Comment 32

[Kris Maglione [:kmag]]

OK, we can't disable that entire test, but we can disable the private browsing section that's failing until bug 1309637 is fixed:

http://searchfox.org/mozilla-central/rev/30fcf167af036aeddf322de44a2fadd370acfd2f/toolkit/components/extensions/test/mochitest/test_ext_cookies.html#182-215

Comment 33

[Gabor Krizsanits (INACTIVE)]

(In reply to Kris Maglione [:kmag] from comment #32)
> OK, we can't disable that entire test, but we can disable the private
> browsing section that's failing until bug 1309637 is fixed:
> 
> http://searchfox.org/mozilla-central/rev/
> 30fcf167af036aeddf322de44a2fadd370acfd2f/toolkit/components/extensions/test/
> mochitest/test_ext_cookies.html#182-215

By disabling you mean should I just remove that part of the test and it will be put back once bug 1309637 is fixed?

Comment 34

[Kris Maglione [:kmag]]

I'd rather add an `if (false)` to the beginning of that block than remove it, and add a comment about bug 1309637, but yes.

Comment 37

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/f4b8933f62ea
Turn e10s-multi on in Nightly. r=me

Comment 38

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/f6c9241b40ec
Followup for a typo in the manigest file. r=me

Comment 39

[Gabor Krizsanits (INACTIVE)]

(In reply to Kris Maglione [:kmag] from comment #34)
> I'd rather add an `if (false)` to the beginning of that block than remove
> it, and add a comment about bug 1309637, but yes.

This is not going to work, this test is failing in quite a few ways. I must disable the entire test I'm afraid. Or if you think that this is not just a bug in the test but actually a broken feature important enough to hold back the landing, please let me know so I can plan accordingly.

Comment 40

[Phil Ringnalda (:philor)]

Backed out in https://hg.mozilla.org/integration/mozilla-inbound/rev/2baeb0ea2983 for the cookies again.

Comment 41

[Kris Maglione [:kmag]]

Only the private browsing parts are failing. There are other parts of the test file that deal with private browsing that can be disabled as well, but we can't disable the entire test.

Comment 42

[Gabor Krizsanits (INACTIVE)]

(In reply to Kris Maglione [:kmag] from comment #41)
> Only the private browsing parts are failing. There are other parts of the
> test file that deal with private browsing that can be disabled as well, but
> we can't disable the entire test.

Even this one? https://treeherder.mozilla.org/logviewer.html#?job_id=70963350&repo=mozilla-inbound&lineNumber=2967

I see two blocks where we open a privateWindow, I disabled them both, this failures happens before those. Anyway, let's talk about it on Monday, and we'll sort out something for this.

Comment 44

[Kris Maglione [:kmag]]

Hm. No, I guess not. I was assuming that was related to the private browsing cookies, but from the screenshot, it looks like we're actually just winding up with an extra tab open during that test.

And it looks like that's probably a race in test_ext_contentscript_permission.html, which doesn't wait for its tab removal to succeed before ending the test.

Comment 45

[Phil Ringnalda (:philor)]

Just so it's not so boring, being only about test_ext_cookies.html, there's also https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&fromchange=c1f03c71ed58936e84f1793b5e1944d2cc55f95d&group_state=expanded&filter-searchStr=21ccd22701da8ee54aab8604116ea43d547d2c36&selectedJob=70967962

Comment 46

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/aefa445b9c77
Turn e10s-multi on in Nightly. r=me

Comment 47

[Sylvestre Ledru [:Sylvestre]]

Please nominate this for the release notes when it is ready to ride the trains.

Comment 48

[Andrew McCreight [:mccr8]]

I duped all the leak bugs over, as they haven't shown up in a while, except for bug 1328374, which has happened a few times, in around the time window when this landed.

Comment 49

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/aefa445b9c77