[meta] Run an e10s A/B experiment on Beta 44

RESOLVED FIXED

Status

()

Firefox
General
RESOLVED FIXED
2 years ago
a year ago

People

(Reporter: vladan, Assigned: vladan)

Tracking

44 Branch
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox44+ fixed)

Details

Attachments

(3 attachments, 2 obsolete attachments)

(Assignee)

Description

2 years ago
Beta 43 experiment (bug 1225074) had to be scrapped, so now we're aiming for beta 1 of Beta 44 (merges on December 15th).

We should be able to reuse the experiment code from bug 1225074 almost 100%.
(Assignee)

Updated

2 years ago
Assignee: nobody → vladan.bugzilla
(Assignee)

Updated

2 years ago
Depends on: 1222894
(Assignee)

Comment 1

2 years ago
Correction, the experiment code to be reused is in bug 1222894
It may be useful to track this or just check up on it from a worry list rather than tracking all the individual bugs. Up to you Ritu!
status-firefox44: --- → affected
tracking-firefox44: --- → ?
Vladan, should this be retested again when we get closer to fx44b1? Or the testing that was done in 1222894 sufficient enough?

Updated

2 years ago
Blocks: 1218484

Updated

2 years ago
No longer blocks: 1218484
(Assignee)

Comment 4

2 years ago
(In reply to Kamil Jozwiak [:kjozwiak] from comment #3)
> Vladan, should this be retested again when we get closer to fx44b1? Or the
> testing that was done in 1222894 sufficient enough?

There will be uplifts, so we should probably retest the a11y bits just to make sure. We will also revert to an earlier version of the experiment, but only the a11y code-paths will be changed
(Assignee)

Comment 5

2 years ago
Created attachment 8697112 [details]
experiment.xpi

New experiment xpi

Jason, can you sign this?
Flags: needinfo?(jthomas)
(Assignee)

Comment 6

2 years ago
Created attachment 8697113 [details] [diff] [review]
Diff from the earlier Beta 43 experiment
Attachment #8697113 - Flags: review?(felipc)
(Assignee)

Comment 7

2 years ago
Created attachment 8697121 [details] [diff] [review]
Diff from the earlier Beta 43 experiment
Attachment #8697113 - Attachment is obsolete: true
Attachment #8697113 - Flags: review?(felipc)
Attachment #8697121 - Flags: review?(felipc)
Attachment #8697121 - Flags: review?(felipc) → review+
(Assignee)

Comment 8

2 years ago
The start date is in the past (November) but the minimum version is Beta 44. Beta 44 will be released on Monday, December 14th next week.

The end date is end-of-day December 23rd (GMT). The experiment duration is 10 days.

Sampling is still 15% of Beta population.
(Assignee)

Comment 9

2 years ago
Created attachment 8697126 [details] [diff] [review]
Full patch (for reference)
Created attachment 8697131 [details]
experiment.xpi

Please see attached.
Attachment #8697112 - Attachment is obsolete: true
Flags: needinfo?(jthomas)
(Assignee)

Comment 11

2 years ago
Kamil: the Beta 44 build to test should be available here shortly https://treeherder.mozilla.org/#/jobs?repo=try&revision=b470943a4c8a
(Assignee)

Comment 12

2 years ago
(In reply to Vladan Djeric (:vladan) -- please needinfo | PTO Dec 14-18. Email or IRC if urgent from comment #11)
> Kamil: the Beta 44 build to test should be available here shortly
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=b470943a4c8a

There was a bug in the last config, this link should have it fixed:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=7ec80684f5a1&selectedJob=14557557
Went through verification using the following build:
- http://archive.mozilla.org/pub/firefox/try-builds/vdjeric@mozilla.com-7ec80684f5a1f105fe1a8a135a6b02c5e35ac40d/try-win32/
- https://treeherder.mozilla.org/#/jobs?repo=try&revision=7ec80684f5a1

** Note: I could only test on Win as that's the only build available

Test Plan/Results: https://public.etherpad-mozilla.org/p/e10sBETAExperimentTesting

Looks like everything is working as expected other than the possible issues I mentioned below. Vlad, I remember reading the mailing lists about the experiment not being installed on profiles that have a large amount of add-ons? Was that logic implemented into the experiment? Let me know if that needs testing as well..

Possible Issues:

- browser.tabs.remote.disabled-for-a11y;false and browser.tabs.remote.disabled-for-a11y;true are not being removed once the experiment is terminated/expires
- no e10s UI under about:preferences (pretty sure this is expected but double checking)
Flags: needinfo?(vladan.bugzilla)
(Assignee)

Comment 14

2 years ago
> Vlad, I remember reading the mailing lists about the experiment not being installed on profiles that 
> have a large amount of add-ons? Was that logic implemented into the experiment? Let me know if that 
> needs testing as well..

No, we never added this logic to the experiment code. Can you send me a link to this discussion?

> browser.tabs.remote.disabled-for-a11y;false and browser.tabs.remote.disabled-for-a11y;true 
> are not being removed once the experiment is terminated/expires

This pref is set by Firefox code, not the experiment, so I think this is fine. Felipe, can you confirm?

> no e10s UI under about:preferences (pretty sure this is expected but double checking)

This is fine.
Flags: needinfo?(vladan.bugzilla) → needinfo?(felipc)
> No, we never added this logic to the experiment code. Can you send me a link
> to this discussion?

https://groups.google.com/forum/#!topic/mozilla.dev.platform/OTOwv2PHfV8
> > browser.tabs.remote.disabled-for-a11y;false and browser.tabs.remote.disabled-for-a11y;true 
> > are not being removed once the experiment is terminated/expires
> 
> This pref is set by Firefox code, not the experiment, so I think this is
> fine. Felipe, can you confirm?

Yep, it's the correct behavior
Flags: needinfo?(felipc)
> Yep, it's the correct behavior

Looks like the only thing left is to double check and make sure if we need the logic to check and see if there's any add-ons installed before enabling the experiment. I believe the original concern was that some add-ons are still not working correctly with e10s and can cause performance problems. Apparently LastPass [1] doesn't play well with e10s so if we end up enabling e10s on a profile with LastPass, we might anger some users.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1008768
(Assignee)

Updated

2 years ago
Depends on: 1232544
Given that 44.0b1 (desktop) was pushed to Beta channel only a few hours ago, should we consider extending the experiment end date as the telemetry data will only start accumulating now?
Flags: needinfo?(vladan.bugzilla)

Comment 19

2 years ago
Extending would be good. Seems to be running correctly. Just a small number of beta users are currently updated onto 44, making sample low.

Overall e10s crash rate is 25% (+-10%) up.
Browser halved. Plugin two-thirds.

Low on samples
Low memory account for a few percent more.
Fixed in 45 bug 1225829 up to 5% of the increase.
Content shutdown timeout likely to be a couple.
Couple current high signatures showing user trying multiple times to restore crashing content.

https://crash-stats.mozilla.com/search/?product=Firefox&ActiveExperiment=%3De10s-enabled-beta-20151214%40experiments.mozilla.org&date=%3E%3D2015-12-18&ActiveExperimentBranch=%3Dcontrol&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports

https://crash-stats.mozilla.com/search/?product=Firefox&ActiveExperiment=%3De10s-enabled-beta-20151214%40experiments.mozilla.org&date=%3E%3D2015-12-18&ActiveExperimentBranch=%3Dexperiment&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports
(Assignee)

Comment 20

2 years ago
(In reply to Ritu Kothari (:ritu) from comment #18)
> Given that 44.0b1 (desktop) was pushed to Beta channel only a few hours ago,
> should we consider extending the experiment end date as the telemetry data
> will only start accumulating now?

Yes, I will extend its end date
Flags: needinfo?(vladan.bugzilla)
Tracked for FF44 and marked as "fixed" given that the experiment is now on and will likely end after 44.0b4
status-firefox44: affected → fixed
tracking-firefox44: ? → +
(Assignee)

Updated

2 years ago
Depends on: 1234389

Updated

2 years ago
Blocks: 1234647
(Assignee)

Updated

a year ago
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.