1229104 - [meta] Run an e10s A/B experiment on Beta 44

Status: RESOLVED FIXED

(Firefox :: General, defect)

Description

[Vladan Djeric (:vladan)]

Beta 43 experiment (bug 1225074) had to be scrapped, so now we're aiming for beta 1 of Beta 44 (merges on December 15th).

We should be able to reuse the experiment code from bug 1225074 almost 100%.

Comment 1

[Vladan Djeric (:vladan)]

Correction, the experiment code to be reused is in bug 1222894

Comment 2

[Liz Henry (:lizzard) (relman/hg->git project)]

It may be useful to track this or just check up on it from a worry list rather than tracking all the individual bugs. Up to you Ritu!

Comment 3

[Kamil Jozwiak [:kjozwiak]]

Vladan, should this be retested again when we get closer to fx44b1? Or the testing that was done in 1222894 sufficient enough?

Comment 4

[Vladan Djeric (:vladan)]

(In reply to Kamil Jozwiak [:kjozwiak] from comment #3)
> Vladan, should this be retested again when we get closer to fx44b1? Or the
> testing that was done in 1222894 sufficient enough?

There will be uplifts, so we should probably retest the a11y bits just to make sure. We will also revert to an earlier version of the experiment, but only the a11y code-paths will be changed

Comment 5

[Vladan Djeric (:vladan)]

Attachment: experiment.xpi

New experiment xpi

Jason, can you sign this?

Comment 6

[Vladan Djeric (:vladan)]

Attachment: Diff from the earlier Beta 43 experiment

Comment 7

[Vladan Djeric (:vladan)]

Attachment: Diff from the earlier Beta 43 experiment

Comment 8

[Vladan Djeric (:vladan)]

The start date is in the past (November) but the minimum version is Beta 44. Beta 44 will be released on Monday, December 14th next week.

The end date is end-of-day December 23rd (GMT). The experiment duration is 10 days.

Sampling is still 15% of Beta population.

Comment 9

[Vladan Djeric (:vladan)]

Attachment: Full patch (for reference)

Comment 10

[Jason Thomas [:jason]]

Attachment: experiment.xpi

Please see attached.

Comment 11

[Vladan Djeric (:vladan)]

Kamil: the Beta 44 build to test should be available here shortly https://treeherder.mozilla.org/#/jobs?repo=try&revision=b470943a4c8a

Comment 12

[Vladan Djeric (:vladan)]

(In reply to Vladan Djeric (:vladan) -- please needinfo | PTO Dec 14-18. Email or IRC if urgent from comment #11)
> Kamil: the Beta 44 build to test should be available here shortly
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=b470943a4c8a

There was a bug in the last config, this link should have it fixed:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=7ec80684f5a1&selectedJob=14557557

Comment 13

[Kamil Jozwiak [:kjozwiak]]

Went through verification using the following build:
- http://archive.mozilla.org/pub/firefox/try-builds/[e-mail]-7ec80684f5a1f105fe1a8a135a6b02c5e35ac40d/try-win32/
- https://treeherder.mozilla.org/#/jobs?repo=try&revision=7ec80684f5a1

** Note: I could only test on Win as that's the only build available

Test Plan/Results: https://public.etherpad-mozilla.org/p/e10sBETAExperimentTesting

Looks like everything is working as expected other than the possible issues I mentioned below. Vlad, I remember reading the mailing lists about the experiment not being installed on profiles that have a large amount of add-ons? Was that logic implemented into the experiment? Let me know if that needs testing as well..

Possible Issues:

- browser.tabs.remote.disabled-for-a11y;false and browser.tabs.remote.disabled-for-a11y;true are not being removed once the experiment is terminated/expires
- no e10s UI under about:preferences (pretty sure this is expected but double checking)

Comment 14

[Vladan Djeric (:vladan)]

> Vlad, I remember reading the mailing lists about the experiment not being installed on profiles that 
> have a large amount of add-ons? Was that logic implemented into the experiment? Let me know if that 
> needs testing as well..

No, we never added this logic to the experiment code. Can you send me a link to this discussion?

> browser.tabs.remote.disabled-for-a11y;false and browser.tabs.remote.disabled-for-a11y;true 
> are not being removed once the experiment is terminated/expires

This pref is set by Firefox code, not the experiment, so I think this is fine. Felipe, can you confirm?

> no e10s UI under about:preferences (pretty sure this is expected but double checking)

This is fine.

Comment 15

[Kamil Jozwiak [:kjozwiak]]

> No, we never added this logic to the experiment code. Can you send me a link
> to this discussion?

https://groups.google.com/forum/#!topic/mozilla.dev.platform/OTOwv2PHfV8

Comment 16

[:Felipe Gomes (needinfo for replies!)]

> > browser.tabs.remote.disabled-for-a11y;false and browser.tabs.remote.disabled-for-a11y;true 
> > are not being removed once the experiment is terminated/expires
> 
> This pref is set by Firefox code, not the experiment, so I think this is
> fine. Felipe, can you confirm?

Yep, it's the correct behavior

Comment 17

[Kamil Jozwiak [:kjozwiak]]

> Yep, it's the correct behavior

Looks like the only thing left is to double check and make sure if we need the logic to check and see if there's any add-ons installed before enabling the experiment. I believe the original concern was that some add-ons are still not working correctly with e10s and can cause performance problems. Apparently LastPass [1] doesn't play well with e10s so if we end up enabling e10s on a profile with LastPass, we might anger some users.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1008768

Comment 18

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Given that 44.0b1 (desktop) was pushed to Beta channel only a few hours ago, should we consider extending the experiment end date as the telemetry data will only start accumulating now?

Comment 19

[Jonathan Howard]

Extending would be good. Seems to be running correctly. Just a small number of beta users are currently updated onto 44, making sample low.

Overall e10s crash rate is 25% (+-10%) up.
Browser halved. Plugin two-thirds.

Low on samples
Low memory account for a few percent more.
Fixed in 45 bug 1225829 up to 5% of the increase.
Content shutdown timeout likely to be a couple.
Couple current high signatures showing user trying multiple times to restore crashing content.

https://crash-stats.mozilla.com/search/?product=Firefox&ActiveExperiment=%3De10s-enabled-beta-20151214%40experiments.mozilla.org&date=%3E%3D2015-12-18&ActiveExperimentBranch=%3Dcontrol&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports

https://crash-stats.mozilla.com/search/?product=Firefox&ActiveExperiment=%3De10s-enabled-beta-20151214%40experiments.mozilla.org&date=%3E%3D2015-12-18&ActiveExperimentBranch=%3Dexperiment&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports

Comment 20

[Vladan Djeric (:vladan)]

(In reply to Ritu Kothari (:ritu) from comment #18)
> Given that 44.0b1 (desktop) was pushed to Beta channel only a few hours ago,
> should we consider extending the experiment end date as the telemetry data
> will only start accumulating now?

Yes, I will extend its end date

Comment 21

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Tracked for FF44 and marked as "fixed" given that the experiment is now on and will likely end after 44.0b4