Last Comment Bug 1304164 - Disable e10s for russian users through the system add-on
: Disable e10s for russian users through the system add-on
Status: RESOLVED FIXED
[necko-active]
:
Product: Core
Classification: Components
Component: Networking (show other bugs)
: 49 Branch
: Unspecified All
-- critical (vote)
: mozilla52
Assigned To: :Felipe Gomes (needinfo me!)
:
: Patrick McManus [:mcmanus]
Mentors:
Depends on: 1294719
Blocks: 1304874
  Show dependency treegraph
 
Reported: 2016-09-20 13:02 PDT by :Felipe Gomes (needinfo me!)
Modified: 2016-10-14 07:50 PDT (History)
14 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---
+
fixed
fixed
fixed
fixed

MozReview Requests
Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:
Show discarded requests

Attachments
Bug 1304164 - Allow the e10srollout system add-on to control temporary disqualifications. (58 bytes, text/x-review-board-request)
2016-09-20 13:06 PDT, :Felipe Gomes (needinfo me!)
mconley: review+
rkothari: approval‑mozilla‑aurora+
rkothari: approval‑mozilla‑beta+
lhenry: approval‑mozilla‑release+
Details | Review
Bug 1304164 - Temporarily disable e10s for users on the Russian locale. (58 bytes, text/x-review-board-request)
2016-09-20 13:06 PDT, :Felipe Gomes (needinfo me!)
mconley: review+
rkothari: approval‑mozilla‑aurora+
rkothari: approval‑mozilla‑beta+
lhenry: approval‑mozilla‑release+
Details | Review
e10srollout@mozilla.org-ff49-1.3.xpi, unsigned (3.05 KB, application/x-xpinstall)
2016-09-20 14:20 PDT, :Felipe Gomes (needinfo me!)
jthomas: feedback+
Details
e10srollout@mozilla.org-ff49-1.3.xpi, signed (6.91 KB, application/x-xpinstall)
2016-09-20 14:30 PDT, Jason Thomas [:jason]
no flags Details
e10srollout@mozilla.org-ff48-49-1.3.xpi, unsigned (3.05 KB, application/x-xpinstall)
2016-09-20 16:19 PDT, :Felipe Gomes (needinfo me!)
jthomas: feedback+
Details
e10srollout@mozilla.org-ff48-49-1.3.xpi, signed (6.91 KB, application/x-xpinstall)
2016-09-20 16:23 PDT, Jason Thomas [:jason]
no flags Details
version bump to 1.3, r=mconley (1001 bytes, patch)
2016-09-22 11:54 PDT, :Felipe Gomes (needinfo me!)
felipc: review+
rkothari: approval‑mozilla‑aurora+
rkothari: approval‑mozilla‑beta+
lhenry: approval‑mozilla‑release+
Details | Diff | Splinter Review

Description User image :Felipe Gomes (needinfo me!) 2016-09-20 13:02:51 PDT
+++ This bug was initially created as a clone of Bug #1294719 +++

Due to bug 1294719 (a crash in mail.ru). We could deploy a system add-on update probably faster than a dot release.
Comment 1 User image :Felipe Gomes (needinfo me!) 2016-09-20 13:06:43 PDT Comment hidden (mozreview-request)
Comment 2 User image :Felipe Gomes (needinfo me!) 2016-09-20 13:06:43 PDT Comment hidden (mozreview-request)
Comment 3 User image :Felipe Gomes (needinfo me!) 2016-09-20 13:07:51 PDT
I split this in two parts because even if we end up not doing this, part #1 will be the basis for bug 1301131 too.
Comment 4 User image Mike Conley (:mconley) 2016-09-20 13:09:24 PDT
Comment on attachment 8793035 [details]
Bug 1304164 - Allow the e10srollout system add-on to control temporary disqualifications.

https://reviewboard.mozilla.org/r/79848/#review78586
Comment 5 User image Mike Conley (:mconley) 2016-09-20 13:10:25 PDT
Comment on attachment 8793036 [details]
Bug 1304164 - Temporarily disable e10s for users on the Russian locale.

https://reviewboard.mozilla.org/r/79850/#review78588

This commit is just whitespace changes, according to MozReview. Is there a part here where we're disabling for the RU locale that we're missing?
Comment 6 User image :Felipe Gomes (needinfo me!) 2016-09-20 13:12:15 PDT Comment hidden (mozreview-request)
Comment 7 User image :Felipe Gomes (needinfo me!) 2016-09-20 13:12:15 PDT Comment hidden (mozreview-request)
Comment 8 User image Mike Conley (:mconley) 2016-09-20 13:15:41 PDT
Comment on attachment 8793036 [details]
Bug 1304164 - Temporarily disable e10s for users on the Russian locale.

https://reviewboard.mozilla.org/r/79850/#review78590

Gotcha, thanks!
Comment 9 User image Axel Hecht [:Pike] 2016-09-20 14:10:14 PDT
We probably have data on usage of Firefox and possibly hitting mail.ru, can we add that data to this bug?
Comment 10 User image :Felipe Gomes (needinfo me!) 2016-09-20 14:15:56 PDT
I don't know who have that data, can you ping/bring the right people to this bug (or bug 1294719)?
Comment 11 User image :Felipe Gomes (needinfo me!) 2016-09-20 14:20:16 PDT
Created attachment 8793074 [details]
e10srollout@mozilla.org-ff49-1.3.xpi, unsigned

Jason, could you sign this xpi for me?
Comment 12 User image Axel Hecht [:Pike] 2016-09-20 14:27:43 PDT
(In reply to :Felipe Gomes (needinfo me!) from comment #10)
> I don't know who have that data, can you ping/bring the right people to this
> bug (or bug 1294719)?

This is less about having the data, but being able to digest it into a human-readable form. It's been ages since I succeeded, so anything between our internal tableau and telemetry should have data about at least the correlation between Russian and using the Russian localization.

Maybe there's stuff in sites like alexa about popularity of mail.ru outside of russia?
Comment 13 User image Jason Thomas [:jason] 2016-09-20 14:30:19 PDT
Created attachment 8793079 [details]
e10srollout@mozilla.org-ff49-1.3.xpi, signed

Please see attached.
Comment 14 User image Marco Castelluccio [:marco] 2016-09-20 14:34:59 PDT
(In reply to Axel Hecht [:Pike] from comment #12)
> (In reply to :Felipe Gomes (needinfo me!) from comment #10)
> > I don't know who have that data, can you ping/bring the right people to this
> > bug (or bug 1294719)?
> 
> This is less about having the data, but being able to digest it into a
> human-readable form. It's been ages since I succeeded, so anything between
> our internal tableau and telemetry should have data about at least the
> correlation between Russian and using the Russian localization.
> 
> Maybe there's stuff in sites like alexa about popularity of mail.ru outside
> of russia?

See also bug 1294719 comment 8. The crash reports also have a correlation (stronger than
with mail.ru) with the ok.ru website (777 vs 25 out of 7890). 6814 crashes have an empty
URL, so it's hard to say where most users are crashing.

ok.ru is 50th in the Alexa global list, 7th in the Russian Alexa list.
Comment 15 User image Jim Mathies [:jimm] 2016-09-20 15:33:15 PDT
The fix in bug 1294719 is going to need some bake time / testing, so lets go ahead and disable for ru for now to avoid losing users. I assume this means we'll end up pushing the fix in a dot release to 49.
Comment 16 User image :Felipe Gomes (needinfo me!) 2016-09-20 16:19:37 PDT
Created attachment 8793113 [details]
e10srollout@mozilla.org-ff48-49-1.3.xpi, unsigned

Ah I generated that xpi from a mozilla-release build, and the install.rdf had minVersion = 49. But this will need to target 48 too.
Comment 17 User image Jason Thomas [:jason] 2016-09-20 16:23:22 PDT
Created attachment 8793115 [details]
e10srollout@mozilla.org-ff48-49-1.3.xpi, signed

Please see attached.
Comment 18 User image :Felipe Gomes (needinfo me!) 2016-09-20 18:08:54 PDT
OK, I tested it locally and it worked well, so it's ready to go to QA.  Andrei, could you or someone from your team verify this?

The goal of this update is to disable Multiprocess for users of Firefox in the Russian locale


1. Using a clean profile from Firefox release, 48.0.2 [1] and 49.0 [2], in Russian and in English.

2. Go to about:support and verify:
 - Multiprocess Windows: 1/1 (Enabled by default)
 - Note the list of Installed Extensions and versions (these lists are different in 48.0.2 and 49.0)

3. Go to about:config and verify:
 - e10s.rollout.cohort is "test"

4. Go to about:config and change the following:
  - devtools.chrome.enabled, set to true
  - extensions.systemAddon.update.url set to [3] or [4]  (see below)

5. Force an add-on update check, by running the following in Tools > Web Developer > Browser Console:

    Components.utils.import("resource://gre/modules/AddonManager.jsm");
    AddonManagerPrivate.backgroundUpdateCheck();

6. Wait a moment in order for the update to download, then restart

6. After a restart, check:

  ***  If using the English language Firefox:
    
   - No change in Multiprocess Windows in about:support
   - No change in e10s.rollout.cohort pref in about:config

   - Installed extensions:
     - Multi-process staged rollout should have updated to version 1.3
     - No changes to others on the list

  ***  If using the Russian language Firefox:
    
   - Multiprocess Windows should be 0/1
   - e10s.rollout.cohort pref should have changed to "temp-disqualified-ru"

   - Installed extensions:
     - Multi-process staged rollout should have updated to version 1.3
     - No changes to others on the list



[1] https://ftp.mozilla.org/pub/firefox/releases/48.0.2/ then find "en-US" and "ru" versions
[2] https://ftp.mozilla.org/pub/firefox/releases/49.0/

[3] If using 48.0.2, use https://people.mozilla.org/~fgomes/e10s-v1.3/manifest-for-48.xml
[4] If using 49.0,   use https://people.mozilla.org/~fgomes/e10s-v1.3/manifest-for-49.xml
Comment 19 User image Andrei Vaida, QA [:avaida] – please ni? me 2016-09-21 04:11:05 PDT
We can confirm that the update of Multi-process staged rollout to v1.3 was successful and that the expected changes were applied.

More detailed test results are available here:
    
    https://public.etherpad-mozilla.org/p/1304164
Comment 20 User image :Felipe Gomes (needinfo me!) 2016-09-21 12:29:11 PDT
Thanks Jason for the signings. This latest version that you signed is good to go. Could you upload it to ftp as:

https://ftp.mozilla.org/pub/system-addons/e10srollout/e10srollout@mozilla.org-ff48-49-1.3-signed.xpi
Comment 21 User image Mark Banner (:standard8) 2016-09-21 12:34:34 PDT
Andrei, for FF 49, did you check to see if the other system add-ons (pocket, webcompat) stayed the same version, enabled, and that pocket was still working fine?

There's new functionality for FF 49 with a differently formatted update snippet, so I'd like to confirm that was tested on all platforms.
Comment 22 User image Mark Banner (:standard8) 2016-09-21 12:35:31 PDT
(In reply to :Felipe Gomes (needinfo me!) from comment #20)
> Thanks Jason for the signings. This latest version that you signed is good
> to go. Could you upload it to ftp as:

I can upload it this time.

Will the patches be landing for 49/in-tree sometime (in case of a dot release)?
Comment 23 User image :Felipe Gomes (needinfo me!) 2016-09-21 12:45:13 PDT
(In reply to Mark Banner (:standard8) from comment #22)
> Will the patches be landing for 49/in-tree sometime (in case of a dot
> release)?

That remains an open question, depending on whether this dot release would contain a fix for the crash (bug 1294719) or not.
Comment 24 User image Mark Banner (:standard8) 2016-09-21 13:09:45 PDT
The updates for 48/49 are now on the release-sysaddon channel
Comment 25 User image Liz Henry (:lizzard) (needinfo? me) 2016-09-21 13:10:38 PDT
Right now we don't have concrete plans for a dot release as there is no strong driver for it.
Comment 26 User image Andrei Vaida, QA [:avaida] – please ni? me 2016-09-22 04:20:37 PDT
(In reply to Mark Banner (:standard8) from comment #21)
> Andrei, for FF 49, did you check to see if the other system add-ons (pocket,
> webcompat) stayed the same version, enabled, and that pocket was still
> working fine?
> 
> There's new functionality for FF 49 with a differently formatted update
> snippet, so I'd like to confirm that was tested on all platforms.

This has just been confirmed on Fx49:

    on en-US

    	- Multi-process staged rollout is successfully updating to v1.3 and remains
    	enabled
    	- Pocket version stays the same (v1.0.4) and its associated core
    	functionality is intact
    	- Web Compat stays the same (v1.0)

     on ru

    	- Multi-process staged rollout is successfully updating to v1.3 and gets
    	disabled
    	- Pocket version stays the same (v1.0.4) and its associated core
    	functionality is intact
    	- Web Compat stays the same (v1.0)
Comment 27 User image :Felipe Gomes (needinfo me!) 2016-09-22 08:51:39 PDT
This has been just deployed today. I'll leave the bug open until we decide if we should take this in the tree or not
Comment 28 User image :Felipe Gomes (needinfo me!) 2016-09-22 11:45:01 PDT
Let's get this on all branches, to avoid accidentally reactivating ru users until we have a proper fix for the crash.

Whenever we have it fixed, we can back out just the part two of this patch to re-enable ru users.
Comment 29 User image :Felipe Gomes (needinfo me!) 2016-09-22 11:54:38 PDT
Created attachment 8793901 [details] [diff] [review]
version bump to 1.3, r=mconley

Posting the version bump here too to get approval. It got r+ on irc
Comment 30 User image Pulsebot 2016-09-22 11:57:08 PDT
Pushed by felipc@gmail.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/5e8c843b5469
Allow the e10srollout system add-on to control temporary disqualifications. r=mconley
https://hg.mozilla.org/integration/mozilla-inbound/rev/b43beebbb08c
Temporarily disable e10s for users on the Russian locale. r=mconley
https://hg.mozilla.org/integration/mozilla-inbound/rev/3fdb67f309bd
Bump e10srollout version to 1.3. r=mconley
Comment 31 User image :Felipe Gomes (needinfo me!) 2016-09-22 14:02:19 PDT
Comment on attachment 8793035 [details]
Bug 1304164 - Allow the e10srollout system add-on to control temporary disqualifications.

Approval Request Comment
[Feature/regressing bug #]: We need to take this up until mozilla-release so that ru users remain blocked in case of a dot release (when the system add-on resets to the in tree version). Removing this block is tracked by bug 1304874
[User impact if declined]: Users on the russian locale would be re-enabled in case of a dot release
[Describe test coverage new/current, TreeHerder]: Landed on inbound
[Risks and why]: Small risk to the e10s system add-on, but this code has already shipped to users through GoFaster. This is just adding it built-in in newer versions
[String/UUID change made/needed]: none


Note: all 3 patches need to go together so I won't repeat the approval request individually.
Comment 32 User image :Felipe Gomes (needinfo me!) 2016-09-22 14:02:55 PDT
Comment on attachment 8793036 [details]
Bug 1304164 - Temporarily disable e10s for users on the Russian locale.

Approval Request Comment:

See comment 31
Comment 33 User image :Felipe Gomes (needinfo me!) 2016-09-22 14:03:23 PDT
Comment on attachment 8793901 [details] [diff] [review]
version bump to 1.3, r=mconley

Approval Request Comment:

See comment 31
Comment 34 User image :Felipe Gomes (needinfo me!) 2016-09-22 14:06:27 PDT
[Tracking Requested - why for this release]: see comment 31
Comment 35 User image Ritu Kothari (:ritu) 2016-09-22 15:55:39 PDT
Comment on attachment 8793035 [details]
Bug 1304164 - Allow the e10srollout system add-on to control temporary disqualifications.

Fix to avoid crashing ru locale users in specific scenarios, Aurora51+, Beta50+
Comment 40 User image Liz Henry (:lizzard) (needinfo? me) 2016-09-30 15:00:04 PDT
Actually I'm going to leave this as "affected" for 49 for now. As I understand it we already rolled out the system add-on fix on release. Is that right?  But it is still not in mozilla-release itself. We may not need a dot release, so for now I'm leaving the potential ridealongs in the approval queue.
Comment 41 User image Liz Henry (:lizzard) (needinfo? me) 2016-10-06 08:34:23 PDT
Comment on attachment 8793035 [details]
Bug 1304164 - Allow the e10srollout system add-on to control temporary disqualifications.

Let's uplift this now to release. In case we do end up with a dot release, this will already be in place.

Note You need to log in before you can comment on or make changes to this bug.