1292721 - [e10s] Increase e10s activation on the release channel to 10% of eligible users

Status: RESOLVED FIXED

(Firefox :: General, defect)

Description

[:Felipe Gomes (needinfo for replies!)]

+++ This bug was initially created as a clone of Bug #1284958 +++

If everything is looking good, the plan is to do this via a GoFaster update of the e10srollout add-on in or around Aug 12th. Until there, let's get the add-on prepared, signed, tested and ready to go.

Comment 1

[Mark Banner (:standard8)]

Felipe, Hello already has a planned Go Faster update on 15th August. Since we don't normally push out release on a Friday, do you want to sync up & release on the 15th at the same time as Hello?

You'll also need to notify release-drivers of the intention as well:

https://wiki.mozilla.org/Firefox/Go_Faster/Process#Process_to_Push_updates_to_release_channel

Comment 2

[:Felipe Gomes (needinfo for replies!)]

Hi Mark, thanks for letting me know. I'll bring this up with the e10s team to see if they are ok with it

Comment 3

[:Felipe Gomes (needinfo for replies!)]

Plan of record is to actually do it sooner (hopefully by tomorrow, Aug 9th)

Comment 4

[:Felipe Gomes (needinfo for replies!)]

Attachment: [mq]: Bug 1292721 - Increase e10s activation on the release channel to 10% of eligible users.

Review commit: https://reviewboard.mozilla.org/r/70058/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/70058/

Comment 5

[:Felipe Gomes (needinfo for replies!)]

Comment on attachment 8778990 [details]
[mq]: Bug 1292721 - Increase e10s activation on the release channel to 10% of eligible users.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/70058/diff/1-2/

Comment 6

[Mike Conley (:mconley) (:⚙️)]

Comment on attachment 8778990 [details]
[mq]: Bug 1292721 - Increase e10s activation on the release channel to 10% of eligible users.

https://reviewboard.mozilla.org/r/70058/#review67342

\o/

Comment 7

[Mark Banner (:standard8)]

Michael, can you handle this rollout for me, I'm meant to be on PTO this week.

Please re-use the "SystemAddons-ff48-rel1" and downgrade loop to the 1.4.3 that Felipe is uploading & downgrade e10srollout. I'll create a rel2 next week for the loop release.

Felipe, whilst I realise the change to the add-on is normally simple, generally we have QA sign-off on the update before we publish it. Have you got QA set up to do that?

Comment 8

[Mike Conley (:mconley) (:⚙️)]

(In reply to Mark Banner (:standard8) (afk 8 - 12 August) from comment #7)
> Michael, can you handle this rollout for me, I'm meant to be on PTO this
> week.
> 

Hey Mark (sorry to bug you on PTO) - was there supposed to be a needinfo on this one? Which Michael are you referring to?

Comment 9

[:Felipe Gomes (needinfo for replies!)]

Comment on attachment 8778990 [details]
[mq]: Bug 1292721 - Increase e10s activation on the release channel to 10% of eligible users.

Approval Request Comment:

Sylvestre, whenever there's an update, Firefox reverts back to the bundled system add-ons (vs. the ones updated via GoFaster). This means that even after we release the 10% update to 48.0 users, we need to ship this patch built-in with the 48.0.1 dot release in order to continue with 10% (otherwise it will revert to 1%)

Comment 10

[:Felipe Gomes (needinfo for replies!)]

(I think he meant Michael Kelly)

Comment 11

[:Felipe Gomes (needinfo for replies!)]

Alright, I tested locally a system add-on set update with the newly signed xpi, the right pocket xpi and also a newer version of loop. Once we get bug 1293398 and bug 1293418, we can test the update on a staging location.
I don't think QA is necessary for this but if people want it I can ask QA to do some sanity checks across platforms tomorrow.

Comment 12

[Sylvestre Ledru [:Sylvestre]]

Andrei, could you help with this system addon verification?

Comment 13

[Sylvestre Ledru [:Sylvestre]]

Comment on attachment 8778990 [details]
[mq]: Bug 1292721 - Increase e10s activation on the release channel to 10% of eligible users.

Let's take it to the release branch

Comment 14

[Andrei Vaida [:avaida]]

(In reply to Sylvestre Ledru [:sylvestre] from comment #12)
> Andrei, could you help with this system addon verification?

Sure. Felipe, what should we focus on here? We'll definitely need the signed *.xpi and the staging location before hand, but I'm particularly concerned about how we're going to confirm the increase to 10% (if that's actually expected of QA).

Comment 15

[:Felipe Gomes (needinfo for replies!)]

Hi Andrei, thanks for the offer. Here's an step by step on all that should be tested:

1. Using a clean profile from Firefox release 48.0

2. Go to about:support and verify:

 - Multiprocess Windows: 0/1 (Disabled) 

 - Installed extensions:
   - Firefox Hello Beta, version 1.4.3
   - Pocket, version 1.0.4
   - Multi-process staged rollout, 1.0

3. Go to about:config and change the following:
  - devtools.chrome.enabled, set to true
  - e10s.rollout.cohortSample, set to "0.09" (no quotes)
  - extensions.systemAddon.update.url set to "http://people.mozilla.org/~fgomes/e10s-update/update-ff48-1.1.xml" (no quotes)

4. Force an add-on update check, by running the following in Tools > Web Developer > Browser Console:

    Components.utils.import("resource://gre/modules/AddonManager.jsm");
    AddonManagerPrivate.backgroundUpdateCheck();

5. Wait a minute in order for the update to download, then restart

6. After a restart, check in about:support

  - Multiprocess Windows: 1/1 (Enabled)

  - Installed extensions:
   - Firefox Hello Beta, version 1.4.3   (no change)
   - Pocket, version 1.0.4   (no change)
   - Multi-process staged rollout, 1.1    (* changed *)

7. And then do some basic sanity check that Hello and Pocket works

Comment 16

[:Felipe Gomes (needinfo for replies!)]

Attachment: update.xml

Ben, all the xpis are signed and uploaded to their correct locations, so I manually created the update.xml to make this easier (with the right file sizes and checksums).

I tested an update with this file locally, and now Andrei will test it remotely that I uploaded to my people.mozilla.org. But if you can get it in any staging form from Balrog we could test it from there instead. Not sure how important that is

Comment 17

[Osmose [:osmose, :mkelly]]

I've been having trouble connecting to Balrog since the AWS transition, so until that gets resolved (bhearsum is filing a bug to fix it at this very moment) he'll be on point for performing this rollout.

Comment 18

[[e-mail] (:bhearsum)]

(In reply to :Felipe Gomes (needinfo me!) from comment #16)
> Created attachment 8779392 [details]
> update.xml
> 
> Ben, all the xpis are signed and uploaded to their correct locations, so I
> manually created the update.xml to make this easier (with the right file
> sizes and checksums).
> 
> I tested an update with this file locally, and now Andrei will test it
> remotely that I uploaded to my people.mozilla.org. But if you can get it in
> any staging form from Balrog we could test it from there instead. Not sure
> how important that is

Okay, I've done this on the "release-sysaddon" update channel. Any System Addon update requests done via that channel from Firefox 48.0 should receive:
- Loop 1.4.3 (same as built in)
- Pocket 1.0.4 (same as built in)
- E10S Rollout 1.1

Comment 19

[:Felipe Gomes (needinfo for replies!)]

Thanks Ben. Andrei, you can change the value for the extensions.systemAddon.update.url from comment 15, and use this one instead:

https://aus5.mozilla.org/update/3/SystemAddons/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/release-sysaddon/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/update.xml

(note: it's similar to the default pref, but it's indeed a bit different)

Comment 20

[:Felipe Gomes (needinfo for replies!)]

Pushed the increase to 10% to mozilla-release so that it's built-in for 48.0.1:

https://hg.mozilla.org/releases/mozilla-release/rev/086d9208b431

Comment 21

[Andrei Vaida [:avaida]]

We can confirm that the Multi-process staged rollout sys add-on was
successfully updated to v1.1 via release-sysaddon channel. Two
potential issues were found while testing this:

  1. the logs displayed after forcing the add-on update check
  (Comment 15, Step 4) are not reflecting the fact that an update
  to 1.1 is actually found, (e.g.): 

     	1470816580665 addons.update-checker WARN Update manifest for
     	{972ce4c6-7e08-4474-a285-3208198ce6fd} did not contain an
     	updates property

	1470816580884 addons.update-checker WARN Update manifest for
	[e-mail] did not contain an updates property

	1470816581324 addons.update-checker WARN Update manifest for
	[e-mail] did not contain an updates property
  
  2. after restarting the browser (Comment 15, Step 6) the language
  used throughout the UI of Hello is Portuguese
  	- just to be clear, at first startup, the language of Hello was
  	English

Please note that everything else works properly and no functional
issues were found while spot-checking Hello 1.4.3 and Pocket 1.0.4.

Comment 22

[:Felipe Gomes (needinfo for replies!)]

(In reply to Andrei Vaida, QA [:avaida] – please ni? me from comment #21)
>   2. after restarting the browser (Comment 15, Step 6) the language
>   used throughout the UI of Hello is Portuguese
>   	- just to be clear, at first startup, the language of Hello was
>   	English

Hmm that's not good.. I thought that all languages were bundled in the built-in package that I got from release 48. I guess to properly build 1.4.3 it would have to be from the github repo? But we would need an answer from Standard8 or someone else who knows what's the right thing to do

Comment 23

[Osmose [:osmose, :mkelly]]

rhelmer may have some insight about using the XPIs built during the Firefox build process, he's been advocating that for a while.

Comment 24

[:Felipe Gomes (needinfo for replies!)]

The pocket-1.0.4 signed xpi that was already uploaded (I imagine in preparation for next week's GoFaster release) also only contains en-US. So I suspect the same prob would happen with it but it would go unnoticed due to being en-US.

Comment 25

[Robert Helmer [:rhelmer]]

(In reply to Michael Kelly [:mkelly,:Osmose] from comment #23)
> rhelmer may have some insight about using the XPIs built during the Firefox
> build process, he's been advocating that for a while.

There are a few bugs open for this (bug 1277920 and bug 1281578), but I know Hello's current process is a little different so we probably need to wait for ianb/standard8 to answer comment 22.

Comment 26

[Ed Lee :Mardak]

This xpi only has one packaged locale, pt-BR:

https://ftp.mozilla.org/pub/system-addons/hello/[e-mail]

Comment 27

[Ed Lee :Mardak]

Attachment: v1.4.3 [e-mail]

Following instructions for AMO release except failing on `upload_xpi` for `JPM_API_KEY and JPM_API_SECRET should be defined`:
https://github.com/mozilla/loop/blob/master/docs/Releases.md#shipping-to-amo

$ git clone [e-mail]:mozilla/loop.git
$ cd loop
$ git checkout v1.4.3
$ make distclean
$ make dist
$ ls dist/loop\@mozilla.org.xpi

Attached is the xpi from the last line.

Comment 28

[:Felipe Gomes (needinfo for replies!)]

(In reply to Ed Lee :Mardak from comment #26)
> This xpi only has one packaged locale, pt-BR:
> 
> https://ftp.mozilla.org/pub/system-addons/hello/[e-mail]-ff48-1.4.3-
> signed.xpi

yeah, this is how the issue was fortunately found by QA. The other problem now is that http://ftp.mozilla.org/pub/system-addons/pocket/[e-mail] only has en-US

Comment 29

[Ed Lee :Mardak]

(In reply to Ed Lee :Mardak from comment #27)
> Created attachment 8779778 [details]
> v1.4.3 [e-mail]
Compared to bug 1291753's attachment 8777391 [details], the only differences are indeed just the changes between the tagged versions: https://github.com/mozilla/loop/compare/v1.4.3...v1.4.4

Should I file a bug for signing or we're waiting on the pocket not-just-en-US xpi?

Comment 30

[:Felipe Gomes (needinfo for replies!)]

(In reply to Ed Lee :Mardak from comment #29)
> (In reply to Ed Lee :Mardak from comment #27)
> > Created attachment 8779778 [details]
> > v1.4.3 [e-mail]
> Compared to bug 1291753's attachment 8777391 [details], the only differences
> are indeed just the changes between the tagged versions:
> https://github.com/mozilla/loop/compare/v1.4.3...v1.4.4
> 
> Should I file a bug for signing or we're waiting on the pocket
> not-just-en-US xpi?

Thanks, and yes please. In the same bug, ask for it to be uploaded to ftp, replacing the current https://ftp.mozilla.org/pub/system-addons/hello/[e-mail]

Since I'm not involved with the Hello code I didn't want to personally make a call, so I asked Ian Bicking and he said the xpi looks correct.

Comment 31

[Mark Banner (:standard8)]

Oh yes, I forgot that shipping FFs only include the one translation for Hello. Using the packaged one (or the 1.4.3 one from AMO is the right thing to do).

For pocket, I don't know what to do. I thought the xpi didn't include the locales, and they were shipped as part of Firefox. It appears that isn't the case. So it looks like the best that can be done is to manually create an xpi from the locales in m-r.

Comment 32

[Ed Lee :Mardak]

Ah indeed, there's an already signed version of 1.4.3 on AMO:

https://addons.mozilla.org/en-US/firefox/addon/firefox-hello-beta/versions/1.4.3
https://addons.mozilla.org/firefox/downloads/file/466313/firefox_hello_beta-1.4.3-fx.xpi

Comment 33

[:Felipe Gomes (needinfo for replies!)]

Interesting, is that signature also valid for system add-ons? Jorge do you know?

Comment 34

[Robert Helmer [:rhelmer]]

(In reply to Andrei Vaida, QA [:avaida] – please ni? me from comment #21)
> We can confirm that the Multi-process staged rollout sys add-on was
> successfully updated to v1.1 via release-sysaddon channel. Two
> potential issues were found while testing this:
> 
>   1. the logs displayed after forcing the add-on update check
>   (Comment 15, Step 4) are not reflecting the fact that an update
>   to 1.1 is actually found, (e.g.): 
> 
>      	1470816580665 addons.update-checker WARN Update manifest for
>      	{972ce4c6-7e08-4474-a285-3208198ce6fd} did not contain an
>      	updates property
> 
> 	1470816580884 addons.update-checker WARN Update manifest for
> 	[e-mail] did not contain an updates property
> 
> 	1470816581324 addons.update-checker WARN Update manifest for
> 	[e-mail] did not contain an updates property

These logs are different, there are two checks happening here:

1) AMO is checked for updates (and compatibility info), or a custom URL if the updates property is specified

2) System Add-ons are checked against AUS (aka Balrog)

Comment 35

[:Felipe Gomes (needinfo for replies!)]

Attachment: pocket-all-locales.xpi.zip

Manually packaged all fully translated pocket locales. This is untested yet. I'll file a bug for signing and uploading so that we can test.

Comment 36

[:Felipe Gomes (needinfo for replies!)]

Andrei, we're now ready to repeat the same tests from comment 15, but please use the following URL for the extensions.systemAddon.update.url pref:

http://people.mozilla.org/~fgomes/e10s-update/update.xml


The focus here is to do a sanity check on Hello and Pocket and make sure they continue to work properly, specially in a different locale.

Please test the following:
 - with en-US build, it should work and remain in English
 - with de build, it should work and remain in German
 - with vi build, it should work and be presented in English (since there are no strings translated for Hello and Pocket in Vietnamese)

Comment 37

[Andrei Vaida [:avaida]]

We finished running our tests again, using the three locales and the
new update URL (Comment 36) and _almost_ everything is working
properly:

    - the strings used for Hello and Pocket are now correctly used
    based on locale
    - the core functionality of Hello and Pocket is intact
    - Multi-process staged rollout has been successfully updated to
    v1.1

Please note that, compared to the last time we ran these tests:

    - we're still seeing those logs after forcing the add-on update
    check (see Comment 34) along with the following new log:

	1470905616075 addons.xpi ERROR Attempted to load bootstrap scope
	from missing directory

    - we also noticed that before updating the e10s rollout sys
    add-on, Hello was displayed as "Firefox Hello, version 1.4.3" and
    after the update it's displayed as "Firefox Hello Beta, version
    1.4.3" (notice the "Beta" additional word added to its name)

Comment 38

[Sylvestre Ledru [:Sylvestre]]

As Hello is going to die, I don't think that as a big issue...
For the log, I don't know.

Comment 39

[:Felipe Gomes (needinfo for replies!)]

Thanks Andrei. The name change is not an issue. For the log, I believe it's also unrelated, but could you toggle the pref "extensions.logging.enabled" to true? It will give a lot more output, and right before that message there should be another one saying: "Loading bootstrap scope from ..." which will give us more info.

(I'll look into it myself too after a meeting)

I imagine this message is just due to the built-in system add-on being replaced by the downloaded one.

Comment 40

[:Felipe Gomes (needinfo for replies!)]

Ok, I looked into it and that message is coming from the hotfix code, so unrelated to system addons. 

1470933062904	addons.xpi	DEBUG	Loading bootstrap scope from /Users/felipe/moz/release/ff-opt/tmp/scratch_user/extensions/[e-mail]
1470933062905	addons.xpi	ERROR	Attempted to load bootstrap scope from missing directory /Users/felipe/moz/release/ff-opt/tmp/scratch_user/extensions/[e-mail]

There's a bunch of pre-existing warnings related to hotfix that I'll file a bug about it. But it is unrelated to this update.

Comment 41

[:Felipe Gomes (needinfo for replies!)]

This was just pushed live. Thanks to everyone involved here.

Now that the system add-ons from 48 are all properly packaged as standalone xpis, the next e10srollout updates during this cycle should be much simpler to do.