Disable e10s for russian users through the system add-on

RESOLVED FIXED in Firefox 49

Status

()

Core
Networking
--
critical
RESOLVED FIXED
11 months ago
10 months ago

People

(Reporter: Felipe, Assigned: Felipe)

Tracking

49 Branch
mozilla52
Unspecified
All
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox49+ fixed, firefox50 fixed, firefox51 fixed, firefox52 fixed)

Details

(Whiteboard: [necko-active])

MozReview Requests

()

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(5 attachments, 2 obsolete attachments)

(Assignee)

Description

11 months ago
+++ This bug was initially created as a clone of Bug #1294719 +++

Due to bug 1294719 (a crash in mail.ru). We could deploy a system add-on update probably faster than a dot release.
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
(Assignee)

Comment 3

11 months ago
I split this in two parts because even if we end up not doing this, part #1 will be the basis for bug 1301131 too.

Updated

11 months ago
Attachment #8793035 - Flags: review?(mconley)
Attachment #8793036 - Flags: review?(mconley)

Comment 4

11 months ago
mozreview-review
Comment on attachment 8793035 [details]
Bug 1304164 - Allow the e10srollout system add-on to control temporary disqualifications.

https://reviewboard.mozilla.org/r/79848/#review78586
Attachment #8793035 - Flags: review?(mconley) → review+

Comment 5

11 months ago
mozreview-review
Comment on attachment 8793036 [details]
Bug 1304164 - Temporarily disable e10s for users on the Russian locale.

https://reviewboard.mozilla.org/r/79850/#review78588

This commit is just whitespace changes, according to MozReview. Is there a part here where we're disabling for the RU locale that we're missing?
Attachment #8793036 - Flags: review-
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)

Comment 8

11 months ago
mozreview-review
Comment on attachment 8793036 [details]
Bug 1304164 - Temporarily disable e10s for users on the Russian locale.

https://reviewboard.mozilla.org/r/79850/#review78590

Gotcha, thanks!
Attachment #8793036 - Flags: review?(mconley) → review+

Comment 9

11 months ago
We probably have data on usage of Firefox and possibly hitting mail.ru, can we add that data to this bug?
(Assignee)

Comment 10

11 months ago
I don't know who have that data, can you ping/bring the right people to this bug (or bug 1294719)?
(Assignee)

Comment 11

11 months ago
Created attachment 8793074 [details]
e10srollout@mozilla.org-ff49-1.3.xpi, unsigned

Jason, could you sign this xpi for me?
Attachment #8793074 - Flags: feedback?(jthomas)

Updated

11 months ago
Attachment #8793074 - Flags: feedback?(jthomas) → feedback+

Comment 12

11 months ago
(In reply to :Felipe Gomes (needinfo me!) from comment #10)
> I don't know who have that data, can you ping/bring the right people to this
> bug (or bug 1294719)?

This is less about having the data, but being able to digest it into a human-readable form. It's been ages since I succeeded, so anything between our internal tableau and telemetry should have data about at least the correlation between Russian and using the Russian localization.

Maybe there's stuff in sites like alexa about popularity of mail.ru outside of russia?

Comment 13

11 months ago
Created attachment 8793079 [details]
e10srollout@mozilla.org-ff49-1.3.xpi, signed

Please see attached.
(In reply to Axel Hecht [:Pike] from comment #12)
> (In reply to :Felipe Gomes (needinfo me!) from comment #10)
> > I don't know who have that data, can you ping/bring the right people to this
> > bug (or bug 1294719)?
> 
> This is less about having the data, but being able to digest it into a
> human-readable form. It's been ages since I succeeded, so anything between
> our internal tableau and telemetry should have data about at least the
> correlation between Russian and using the Russian localization.
> 
> Maybe there's stuff in sites like alexa about popularity of mail.ru outside
> of russia?

See also bug 1294719 comment 8. The crash reports also have a correlation (stronger than
with mail.ru) with the ok.ru website (777 vs 25 out of 7890). 6814 crashes have an empty
URL, so it's hard to say where most users are crashing.

ok.ru is 50th in the Alexa global list, 7th in the Russian Alexa list.

Comment 15

11 months ago
The fix in bug 1294719 is going to need some bake time / testing, so lets go ahead and disable for ru for now to avoid losing users. I assume this means we'll end up pushing the fix in a dot release to 49.
(Assignee)

Comment 16

11 months ago
Created attachment 8793113 [details]
e10srollout@mozilla.org-ff48-49-1.3.xpi, unsigned

Ah I generated that xpi from a mozilla-release build, and the install.rdf had minVersion = 49. But this will need to target 48 too.
Attachment #8793074 - Attachment is obsolete: true
Attachment #8793079 - Attachment is obsolete: true
Attachment #8793113 - Flags: feedback?(jthomas)

Updated

11 months ago
Attachment #8793113 - Flags: feedback?(jthomas) → feedback+

Comment 17

11 months ago
Created attachment 8793115 [details]
e10srollout@mozilla.org-ff48-49-1.3.xpi, signed

Please see attached.
(Assignee)

Comment 18

11 months ago
OK, I tested it locally and it worked well, so it's ready to go to QA.  Andrei, could you or someone from your team verify this?

The goal of this update is to disable Multiprocess for users of Firefox in the Russian locale


1. Using a clean profile from Firefox release, 48.0.2 [1] and 49.0 [2], in Russian and in English.

2. Go to about:support and verify:
 - Multiprocess Windows: 1/1 (Enabled by default)
 - Note the list of Installed Extensions and versions (these lists are different in 48.0.2 and 49.0)

3. Go to about:config and verify:
 - e10s.rollout.cohort is "test"

4. Go to about:config and change the following:
  - devtools.chrome.enabled, set to true
  - extensions.systemAddon.update.url set to [3] or [4]  (see below)

5. Force an add-on update check, by running the following in Tools > Web Developer > Browser Console:

    Components.utils.import("resource://gre/modules/AddonManager.jsm");
    AddonManagerPrivate.backgroundUpdateCheck();

6. Wait a moment in order for the update to download, then restart

6. After a restart, check:

  ***  If using the English language Firefox:
    
   - No change in Multiprocess Windows in about:support
   - No change in e10s.rollout.cohort pref in about:config

   - Installed extensions:
     - Multi-process staged rollout should have updated to version 1.3
     - No changes to others on the list

  ***  If using the Russian language Firefox:
    
   - Multiprocess Windows should be 0/1
   - e10s.rollout.cohort pref should have changed to "temp-disqualified-ru"

   - Installed extensions:
     - Multi-process staged rollout should have updated to version 1.3
     - No changes to others on the list



[1] https://ftp.mozilla.org/pub/firefox/releases/48.0.2/ then find "en-US" and "ru" versions
[2] https://ftp.mozilla.org/pub/firefox/releases/49.0/

[3] If using 48.0.2, use https://people.mozilla.org/~fgomes/e10s-v1.3/manifest-for-48.xml
[4] If using 49.0,   use https://people.mozilla.org/~fgomes/e10s-v1.3/manifest-for-49.xml
Flags: needinfo?(andrei.vaida)
We can confirm that the update of Multi-process staged rollout to v1.3 was successful and that the expected changes were applied.

More detailed test results are available here:
    
    https://public.etherpad-mozilla.org/p/1304164
Flags: needinfo?(andrei.vaida)
(Assignee)

Comment 20

11 months ago
Thanks Jason for the signings. This latest version that you signed is good to go. Could you upload it to ftp as:

https://ftp.mozilla.org/pub/system-addons/e10srollout/e10srollout@mozilla.org-ff48-49-1.3-signed.xpi
Flags: needinfo?(jthomas)
Andrei, for FF 49, did you check to see if the other system add-ons (pocket, webcompat) stayed the same version, enabled, and that pocket was still working fine?

There's new functionality for FF 49 with a differently formatted update snippet, so I'd like to confirm that was tested on all platforms.
Flags: needinfo?(andrei.vaida)
(In reply to :Felipe Gomes (needinfo me!) from comment #20)
> Thanks Jason for the signings. This latest version that you signed is good
> to go. Could you upload it to ftp as:

I can upload it this time.

Will the patches be landing for 49/in-tree sometime (in case of a dot release)?
Flags: needinfo?(jthomas)
(Assignee)

Comment 23

11 months ago
(In reply to Mark Banner (:standard8) from comment #22)
> Will the patches be landing for 49/in-tree sometime (in case of a dot
> release)?

That remains an open question, depending on whether this dot release would contain a fix for the crash (bug 1294719) or not.
The updates for 48/49 are now on the release-sysaddon channel
Right now we don't have concrete plans for a dot release as there is no strong driver for it.

Updated

11 months ago
Whiteboard: [necko-active]
(In reply to Mark Banner (:standard8) from comment #21)
> Andrei, for FF 49, did you check to see if the other system add-ons (pocket,
> webcompat) stayed the same version, enabled, and that pocket was still
> working fine?
> 
> There's new functionality for FF 49 with a differently formatted update
> snippet, so I'd like to confirm that was tested on all platforms.

This has just been confirmed on Fx49:

    on en-US

    	- Multi-process staged rollout is successfully updating to v1.3 and remains
    	enabled
    	- Pocket version stays the same (v1.0.4) and its associated core
    	functionality is intact
    	- Web Compat stays the same (v1.0)

     on ru

    	- Multi-process staged rollout is successfully updating to v1.3 and gets
    	disabled
    	- Pocket version stays the same (v1.0.4) and its associated core
    	functionality is intact
    	- Web Compat stays the same (v1.0)
Flags: needinfo?(andrei.vaida)
(Assignee)

Comment 27

11 months ago
This has been just deployed today. I'll leave the bug open until we decide if we should take this in the tree or not
(Assignee)

Comment 28

11 months ago
Let's get this on all branches, to avoid accidentally reactivating ru users until we have a proper fix for the crash.

Whenever we have it fixed, we can back out just the part two of this patch to re-enable ru users.
(Assignee)

Comment 29

11 months ago
Created attachment 8793901 [details] [diff] [review]
version bump to 1.3, r=mconley

Posting the version bump here too to get approval. It got r+ on irc
Attachment #8793901 - Flags: review+
(Assignee)

Updated

11 months ago
Attachment #8793901 - Attachment is patch: true

Comment 30

11 months ago
Pushed by felipc@gmail.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/5e8c843b5469
Allow the e10srollout system add-on to control temporary disqualifications. r=mconley
https://hg.mozilla.org/integration/mozilla-inbound/rev/b43beebbb08c
Temporarily disable e10s for users on the Russian locale. r=mconley
https://hg.mozilla.org/integration/mozilla-inbound/rev/3fdb67f309bd
Bump e10srollout version to 1.3. r=mconley
(Assignee)

Updated

11 months ago
Blocks: 1304874
(Assignee)

Comment 31

11 months ago
Comment on attachment 8793035 [details]
Bug 1304164 - Allow the e10srollout system add-on to control temporary disqualifications.

Approval Request Comment
[Feature/regressing bug #]: We need to take this up until mozilla-release so that ru users remain blocked in case of a dot release (when the system add-on resets to the in tree version). Removing this block is tracked by bug 1304874
[User impact if declined]: Users on the russian locale would be re-enabled in case of a dot release
[Describe test coverage new/current, TreeHerder]: Landed on inbound
[Risks and why]: Small risk to the e10s system add-on, but this code has already shipped to users through GoFaster. This is just adding it built-in in newer versions
[String/UUID change made/needed]: none


Note: all 3 patches need to go together so I won't repeat the approval request individually.
Attachment #8793035 - Flags: approval-mozilla-release?
Attachment #8793035 - Flags: approval-mozilla-beta?
Attachment #8793035 - Flags: approval-mozilla-aurora?
(Assignee)

Comment 32

11 months ago
Comment on attachment 8793036 [details]
Bug 1304164 - Temporarily disable e10s for users on the Russian locale.

Approval Request Comment:

See comment 31
Attachment #8793036 - Flags: approval-mozilla-release?
Attachment #8793036 - Flags: approval-mozilla-beta?
Attachment #8793036 - Flags: approval-mozilla-aurora?
(Assignee)

Comment 33

11 months ago
Comment on attachment 8793901 [details] [diff] [review]
version bump to 1.3, r=mconley

Approval Request Comment:

See comment 31
Attachment #8793901 - Flags: approval-mozilla-release?
Attachment #8793901 - Flags: approval-mozilla-beta?
Attachment #8793901 - Flags: approval-mozilla-aurora?
(Assignee)

Comment 34

11 months ago
[Tracking Requested - why for this release]: see comment 31
status-firefox49: --- → affected
status-firefox50: --- → affected
status-firefox51: --- → affected
tracking-firefox49: --- → ?

Comment 35

11 months ago
Comment on attachment 8793035 [details]
Bug 1304164 - Allow the e10srollout system add-on to control temporary disqualifications.

Fix to avoid crashing ru locale users in specific scenarios, Aurora51+, Beta50+
Attachment #8793035 - Flags: approval-mozilla-beta?
Attachment #8793035 - Flags: approval-mozilla-beta+
Attachment #8793035 - Flags: approval-mozilla-aurora?
Attachment #8793035 - Flags: approval-mozilla-aurora+

Updated

11 months ago
Attachment #8793036 - Flags: approval-mozilla-beta?
Attachment #8793036 - Flags: approval-mozilla-beta+
Attachment #8793036 - Flags: approval-mozilla-aurora?
Attachment #8793036 - Flags: approval-mozilla-aurora+

Updated

11 months ago
Attachment #8793901 - Flags: approval-mozilla-beta?
Attachment #8793901 - Flags: approval-mozilla-beta+
Attachment #8793901 - Flags: approval-mozilla-aurora?
Attachment #8793901 - Flags: approval-mozilla-aurora+
(Assignee)

Comment 36

11 months ago
https://hg.mozilla.org/releases/mozilla-aurora/rev/c4115bc9a4e4
https://hg.mozilla.org/releases/mozilla-aurora/rev/44cea7e90ea6
https://hg.mozilla.org/releases/mozilla-aurora/rev/bc17447fefc9
status-firefox51: affected → fixed
(Assignee)

Comment 37

11 months ago
https://hg.mozilla.org/releases/mozilla-beta/rev/2075230b4a42
https://hg.mozilla.org/releases/mozilla-beta/rev/90ca0cd662a4
https://hg.mozilla.org/releases/mozilla-beta/rev/529536d33779
status-firefox50: affected → fixed

Comment 38

11 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/5e8c843b5469
https://hg.mozilla.org/mozilla-central/rev/b43beebbb08c
https://hg.mozilla.org/mozilla-central/rev/3fdb67f309bd
Status: ASSIGNED → RESOLVED
Last Resolved: 11 months ago
status-firefox52: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla52

Comment 39

11 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/5e8c843b5469
https://hg.mozilla.org/mozilla-central/rev/b43beebbb08c
https://hg.mozilla.org/mozilla-central/rev/3fdb67f309bd
status-firefox49: affected → fixed
tracking-firefox49: ? → +
Actually I'm going to leave this as "affected" for 49 for now. As I understand it we already rolled out the system add-on fix on release. Is that right?  But it is still not in mozilla-release itself. We may not need a dot release, so for now I'm leaving the potential ridealongs in the approval queue.
status-firefox49: fixed → affected
Comment on attachment 8793035 [details]
Bug 1304164 - Allow the e10srollout system add-on to control temporary disqualifications.

Let's uplift this now to release. In case we do end up with a dot release, this will already be in place.
Attachment #8793035 - Flags: approval-mozilla-release? → approval-mozilla-release+
Attachment #8793036 - Flags: approval-mozilla-release? → approval-mozilla-release+
Attachment #8793901 - Flags: approval-mozilla-release? → approval-mozilla-release+
(Assignee)

Comment 42

11 months ago
https://hg.mozilla.org/releases/mozilla-release/rev/4defd718466e59ad119815cf4ec6985f0aa8f7a4
https://hg.mozilla.org/releases/mozilla-release/rev/6b9d0bdf9d525fd632a8c41fe25e207aac2dfb45
https://hg.mozilla.org/releases/mozilla-release/rev/7ceaa3683e88b38d3b38927a667d591e9d03f5b7
(Assignee)

Updated

10 months ago
status-firefox49: affected → fixed
You need to log in before you can comment on or make changes to this bug.