1304164 - Disable e10s for russian users through the system add-on

Status: RESOLVED FIXED

(Core :: Networking, defect)

Description

[:Felipe Gomes (needinfo for replies!)]

+++ This bug was initially created as a clone of Bug #1294719 +++

Due to bug 1294719 (a crash in mail.ru). We could deploy a system add-on update probably faster than a dot release.

Comment 1

[:Felipe Gomes (needinfo for replies!)]

Attachment: Bug 1304164 - Allow the e10srollout system add-on to control temporary disqualifications.

Review commit: https://reviewboard.mozilla.org/r/79848/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/79848/

Comment 2

[:Felipe Gomes (needinfo for replies!)]

Attachment: Bug 1304164 - Temporarily disable e10s for users on the Russian locale.

Review commit: https://reviewboard.mozilla.org/r/79850/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/79850/

Comment 3

[:Felipe Gomes (needinfo for replies!)]

I split this in two parts because even if we end up not doing this, part #1 will be the basis for bug 1301131 too.

Comment 4

[Mike Conley (:mconley) (:⚙️)]

Comment on attachment 8793035 [details]
Bug 1304164 - Allow the e10srollout system add-on to control temporary disqualifications.

https://reviewboard.mozilla.org/r/79848/#review78586

Comment 5

[Mike Conley (:mconley) (:⚙️)]

Comment on attachment 8793036 [details]
Bug 1304164 - Temporarily disable e10s for users on the Russian locale.

https://reviewboard.mozilla.org/r/79850/#review78588

This commit is just whitespace changes, according to MozReview. Is there a part here where we're disabling for the RU locale that we're missing?

Comment 6

[:Felipe Gomes (needinfo for replies!)]

Comment on attachment 8793035 [details]
Bug 1304164 - Allow the e10srollout system add-on to control temporary disqualifications.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/79848/diff/1-2/

Comment 7

[:Felipe Gomes (needinfo for replies!)]

Comment on attachment 8793036 [details]
Bug 1304164 - Temporarily disable e10s for users on the Russian locale.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/79850/diff/1-2/

Comment 8

[Mike Conley (:mconley) (:⚙️)]

Comment on attachment 8793036 [details]
Bug 1304164 - Temporarily disable e10s for users on the Russian locale.

https://reviewboard.mozilla.org/r/79850/#review78590

Gotcha, thanks!

Comment 9

[Axel Hecht [:Pike]]

We probably have data on usage of Firefox and possibly hitting mail.ru, can we add that data to this bug?

Comment 10

[:Felipe Gomes (needinfo for replies!)]

I don't know who have that data, can you ping/bring the right people to this bug (or bug 1294719)?

Comment 11

[:Felipe Gomes (needinfo for replies!)]

Attachment: [e-mail], unsigned

Jason, could you sign this xpi for me?

Comment 12

[Axel Hecht [:Pike]]

(In reply to :Felipe Gomes (needinfo me!) from comment #10)
> I don't know who have that data, can you ping/bring the right people to this
> bug (or bug 1294719)?

This is less about having the data, but being able to digest it into a human-readable form. It's been ages since I succeeded, so anything between our internal tableau and telemetry should have data about at least the correlation between Russian and using the Russian localization.

Maybe there's stuff in sites like alexa about popularity of mail.ru outside of russia?

Comment 13

[Jason Thomas [:jason]]

Attachment: [e-mail], signed

Please see attached.

Comment 14

[Marco Castelluccio [:marco]]

(In reply to Axel Hecht [:Pike] from comment #12)
> (In reply to :Felipe Gomes (needinfo me!) from comment #10)
> > I don't know who have that data, can you ping/bring the right people to this
> > bug (or bug 1294719)?
> 
> This is less about having the data, but being able to digest it into a
> human-readable form. It's been ages since I succeeded, so anything between
> our internal tableau and telemetry should have data about at least the
> correlation between Russian and using the Russian localization.
> 
> Maybe there's stuff in sites like alexa about popularity of mail.ru outside
> of russia?

See also bug 1294719 comment 8. The crash reports also have a correlation (stronger than
with mail.ru) with the ok.ru website (777 vs 25 out of 7890). 6814 crashes have an empty
URL, so it's hard to say where most users are crashing.

ok.ru is 50th in the Alexa global list, 7th in the Russian Alexa list.

Comment 15

[Jim Mathies [:jimm]]

The fix in bug 1294719 is going to need some bake time / testing, so lets go ahead and disable for ru for now to avoid losing users. I assume this means we'll end up pushing the fix in a dot release to 49.

Comment 16

[:Felipe Gomes (needinfo for replies!)]

Attachment: [e-mail], unsigned

Ah I generated that xpi from a mozilla-release build, and the install.rdf had minVersion = 49. But this will need to target 48 too.

Comment 17

[Jason Thomas [:jason]]

Attachment: [e-mail], signed

Please see attached.

Comment 18

[:Felipe Gomes (needinfo for replies!)]

OK, I tested it locally and it worked well, so it's ready to go to QA.  Andrei, could you or someone from your team verify this?

The goal of this update is to disable Multiprocess for users of Firefox in the Russian locale


1. Using a clean profile from Firefox release, 48.0.2 [1] and 49.0 [2], in Russian and in English.

2. Go to about:support and verify:
 - Multiprocess Windows: 1/1 (Enabled by default)
 - Note the list of Installed Extensions and versions (these lists are different in 48.0.2 and 49.0)

3. Go to about:config and verify:
 - e10s.rollout.cohort is "test"

4. Go to about:config and change the following:
  - devtools.chrome.enabled, set to true
  - extensions.systemAddon.update.url set to [3] or [4]  (see below)

5. Force an add-on update check, by running the following in Tools > Web Developer > Browser Console:

    Components.utils.import("resource://gre/modules/AddonManager.jsm");
    AddonManagerPrivate.backgroundUpdateCheck();

6. Wait a moment in order for the update to download, then restart

6. After a restart, check:

  ***  If using the English language Firefox:
    
   - No change in Multiprocess Windows in about:support
   - No change in e10s.rollout.cohort pref in about:config

   - Installed extensions:
     - Multi-process staged rollout should have updated to version 1.3
     - No changes to others on the list

  ***  If using the Russian language Firefox:
    
   - Multiprocess Windows should be 0/1
   - e10s.rollout.cohort pref should have changed to "temp-disqualified-ru"

   - Installed extensions:
     - Multi-process staged rollout should have updated to version 1.3
     - No changes to others on the list



[1] https://ftp.mozilla.org/pub/firefox/releases/48.0.2/ then find "en-US" and "ru" versions
[2] https://ftp.mozilla.org/pub/firefox/releases/49.0/

[3] If using 48.0.2, use https://people.mozilla.org/~fgomes/e10s-v1.3/manifest-for-48.xml
[4] If using 49.0,   use https://people.mozilla.org/~fgomes/e10s-v1.3/manifest-for-49.xml

Comment 19

[Andrei Vaida [:avaida]]

We can confirm that the update of Multi-process staged rollout to v1.3 was successful and that the expected changes were applied.

More detailed test results are available here:
    
    https://public.etherpad-mozilla.org/p/1304164

Comment 20

[:Felipe Gomes (needinfo for replies!)]

Thanks Jason for the signings. This latest version that you signed is good to go. Could you upload it to ftp as:

https://ftp.mozilla.org/pub/system-addons/e10srollout/[e-mail]

Comment 21

[Mark Banner (:standard8)]

Andrei, for FF 49, did you check to see if the other system add-ons (pocket, webcompat) stayed the same version, enabled, and that pocket was still working fine?

There's new functionality for FF 49 with a differently formatted update snippet, so I'd like to confirm that was tested on all platforms.

Comment 22

[Mark Banner (:standard8)]

(In reply to :Felipe Gomes (needinfo me!) from comment #20)
> Thanks Jason for the signings. This latest version that you signed is good
> to go. Could you upload it to ftp as:

I can upload it this time.

Will the patches be landing for 49/in-tree sometime (in case of a dot release)?

Comment 23

[:Felipe Gomes (needinfo for replies!)]

(In reply to Mark Banner (:standard8) from comment #22)
> Will the patches be landing for 49/in-tree sometime (in case of a dot
> release)?

That remains an open question, depending on whether this dot release would contain a fix for the crash (bug 1294719) or not.

Comment 24

[Mark Banner (:standard8)]

The updates for 48/49 are now on the release-sysaddon channel

Comment 25

[Liz Henry (:lizzard) (relman/hg->git project)]

Right now we don't have concrete plans for a dot release as there is no strong driver for it.

Comment 26

[Andrei Vaida [:avaida]]

(In reply to Mark Banner (:standard8) from comment #21)
> Andrei, for FF 49, did you check to see if the other system add-ons (pocket,
> webcompat) stayed the same version, enabled, and that pocket was still
> working fine?
> 
> There's new functionality for FF 49 with a differently formatted update
> snippet, so I'd like to confirm that was tested on all platforms.

This has just been confirmed on Fx49:

    on en-US

    	- Multi-process staged rollout is successfully updating to v1.3 and remains
    	enabled
    	- Pocket version stays the same (v1.0.4) and its associated core
    	functionality is intact
    	- Web Compat stays the same (v1.0)

     on ru

    	- Multi-process staged rollout is successfully updating to v1.3 and gets
    	disabled
    	- Pocket version stays the same (v1.0.4) and its associated core
    	functionality is intact
    	- Web Compat stays the same (v1.0)

Comment 27

[:Felipe Gomes (needinfo for replies!)]

This has been just deployed today. I'll leave the bug open until we decide if we should take this in the tree or not

Comment 28

[:Felipe Gomes (needinfo for replies!)]

Let's get this on all branches, to avoid accidentally reactivating ru users until we have a proper fix for the crash.

Whenever we have it fixed, we can back out just the part two of this patch to re-enable ru users.

Comment 29

[:Felipe Gomes (needinfo for replies!)]

Attachment: version bump to 1.3, r=mconley

Posting the version bump here too to get approval. It got r+ on irc

Comment 30

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/5e8c843b5469
Allow the e10srollout system add-on to control temporary disqualifications. r=mconley
https://hg.mozilla.org/integration/mozilla-inbound/rev/b43beebbb08c
Temporarily disable e10s for users on the Russian locale. r=mconley
https://hg.mozilla.org/integration/mozilla-inbound/rev/3fdb67f309bd
Bump e10srollout version to 1.3. r=mconley

Comment 31

[:Felipe Gomes (needinfo for replies!)]

Comment on attachment 8793035 [details]
Bug 1304164 - Allow the e10srollout system add-on to control temporary disqualifications.

Approval Request Comment
[Feature/regressing bug #]: We need to take this up until mozilla-release so that ru users remain blocked in case of a dot release (when the system add-on resets to the in tree version). Removing this block is tracked by bug 1304874
[User impact if declined]: Users on the russian locale would be re-enabled in case of a dot release
[Describe test coverage new/current, TreeHerder]: Landed on inbound
[Risks and why]: Small risk to the e10s system add-on, but this code has already shipped to users through GoFaster. This is just adding it built-in in newer versions
[String/UUID change made/needed]: none


Note: all 3 patches need to go together so I won't repeat the approval request individually.

Comment 32

[:Felipe Gomes (needinfo for replies!)]

Comment on attachment 8793036 [details]
Bug 1304164 - Temporarily disable e10s for users on the Russian locale.

Approval Request Comment:

See comment 31

Comment 33

[:Felipe Gomes (needinfo for replies!)]

Comment on attachment 8793901 [details] [diff] [review]
version bump to 1.3, r=mconley

Approval Request Comment:

See comment 31

Comment 34

[:Felipe Gomes (needinfo for replies!)]

[Tracking Requested - why for this release]: see comment 31

Comment 35

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Comment on attachment 8793035 [details]
Bug 1304164 - Allow the e10srollout system add-on to control temporary disqualifications.

Fix to avoid crashing ru locale users in specific scenarios, Aurora51+, Beta50+

Comment 36

[:Felipe Gomes (needinfo for replies!)]

https://hg.mozilla.org/releases/mozilla-aurora/rev/c4115bc9a4e4
https://hg.mozilla.org/releases/mozilla-aurora/rev/44cea7e90ea6
https://hg.mozilla.org/releases/mozilla-aurora/rev/bc17447fefc9

Comment 37

[:Felipe Gomes (needinfo for replies!)]

https://hg.mozilla.org/releases/mozilla-beta/rev/2075230b4a42
https://hg.mozilla.org/releases/mozilla-beta/rev/90ca0cd662a4
https://hg.mozilla.org/releases/mozilla-beta/rev/529536d33779

Comment 38

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/5e8c843b5469
https://hg.mozilla.org/mozilla-central/rev/b43beebbb08c
https://hg.mozilla.org/mozilla-central/rev/3fdb67f309bd

Comment 39

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/5e8c843b5469
https://hg.mozilla.org/mozilla-central/rev/b43beebbb08c
https://hg.mozilla.org/mozilla-central/rev/3fdb67f309bd

Comment 40

[Liz Henry (:lizzard) (relman/hg->git project)]

Actually I'm going to leave this as "affected" for 49 for now. As I understand it we already rolled out the system add-on fix on release. Is that right?  But it is still not in mozilla-release itself. We may not need a dot release, so for now I'm leaving the potential ridealongs in the approval queue.

Comment 41

[Liz Henry (:lizzard) (relman/hg->git project)]

Comment on attachment 8793035 [details]
Bug 1304164 - Allow the e10srollout system add-on to control temporary disqualifications.

Let's uplift this now to release. In case we do end up with a dot release, this will already be in place.

Comment 42

[:Felipe Gomes (needinfo for replies!)]

https://hg.mozilla.org/releases/mozilla-release/rev/4defd718466e59ad119815cf4ec6985f0aa8f7a4
https://hg.mozilla.org/releases/mozilla-release/rev/6b9d0bdf9d525fd632a8c41fe25e207aac2dfb45
https://hg.mozilla.org/releases/mozilla-release/rev/7ceaa3683e88b38d3b38927a667d591e9d03f5b7