1448176 - Gradual roll-out of TLS fallback-limit to 1.3 on beta channel (60)

Status: RESOLVED FIXED

(Core :: Security: PSM, defect, P1)

Description

[Robert Helmer [:rhelmer]]

TLS 1.3 is already enabled on Beta (currently version 60), we'd like to now do a gradual roll-out of the fallback-limit pref.

This is controlled by the "security.tls.version.fallback-limit" pref, which is currently set to 3 (TLS 1.2) on Beta. The value we wish to roll out is 4 (TLS 1.3)

The plan is to use a system add-on (SAO) update, and to initially roll out to 10% of users.

Comment 1

[Robert Helmer [:rhelmer]]

Opened PR on github:

https://github.com/mozilla/one-off-system-add-ons/pull/102

This is basically the same add-on used in bug 1442042, different pref name and metadata but otherwise identical.

Comment 2

[Robert Helmer [:rhelmer]]

Attachment: TLS 1.3 fallback-limit roll-out SAO v1 (unsigned)

Please sign as a system add-on update. Thanks!

Comment 3

[:wezhou]

Attachment: TLS 1.3 fallback-limit roll-out SAO v1 (signed)

Signed file attached. Please test.

Comment 4

[Robert Helmer [:rhelmer]]

Comment on attachment 8961621 [details]
TLS 1.3 fallback-limit roll-out SAO v1 (signed)

Note that this one is very similar to bug 1442042 but it is intended for the beta channel - could this be set up on a "beta-sysaddon" instead of the usual?

Thanks!

Comment 5

[Rehan Dalal (Inactive)]

This has been shipped to "beta-sysaddon" for 60.*

Comment 6

[Rehan Dalal (Inactive)]

I've also added the rules to "beta" and it's pending sign off from relman

Comment 7

[Robert Helmer [:rhelmer]]

(In reply to Rehan Dalal [:rehan, :rdalal] from comment #6)
> I've also added the rules to "beta" and it's pending sign off from relman

Is anyone available to sign-off on this? It's for beta channel, similar to bug 1442042 but for beta channel only and a different pref (TLS 1.3 fallback-limit), should activate for 10% of beta users.

Comment 8

[Liz Henry (:lizzard) (relman/hg->git project)]

I've been on PTO since last week. I don't see any objection to this rollout, but I'll check with julien tomorrow as he's the release owner for 60.

Comment 9

[Liz Henry (:lizzard) (relman/hg->git project)]

From email, it looks like this is waiting on QA to make sure the signed addon is on the correct channel and works as expected.

Comment 10

[Robert Helmer [:rhelmer]]

(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #9)
> From email, it looks like this is waiting on QA to make sure the signed
> addon is on the correct channel and works as expected.

We've been discussing this on release-drivers, here is my question from there:

I've tested manually, and it's also identical to what we shipped in in
bug 1442042 except for:

* the pref name
* the probability that the pref is flipped
* manifest metadata (addon ID, version, description etc.)

QA did test bug 1442042 but it was a bit tricky since the add-on
determines when to activate (we won't have a great alternative for a
few more Firefox releases).

Given the above, do you want to do manual QA or is it enough if I show
a diff of the changes vs. the XPI in bug 1442042 that's already been
tested?

Comment 11

[Liz Henry (:lizzard) (relman/hg->git project)]

It seems best to do QA here since the pref name changed.

Hani, can you test similiarly to however you tested in bug 1442042? Thanks.

Comment 12

[Liz Henry (:lizzard) (relman/hg->git project)]

Stefan, can someone from the Vegas office test today?

Comment 13

[Liz Henry (:lizzard) (relman/hg->git project)]

Stefan & team will take this on after they get back from lunch.

Comment 14

[Stefan [:StefanG_QA]]

We have tested this on the following platforms: Windows 7 x86, Windows 10 x64, Ubuntu 16.04 x64 and OS X 10.13 using Beta 60.0b8 on beta-sysaddon channel. When running the Addon Manager background Update Check, we can confirm the addon is installed and the information is correct in "about:support" under Firefox Feature section. 

Note: We were not able to hit the 10% cohort.

Comment 15

[Eric Rescorla (:ekr)]

If we're serious about testing this, you should manually set the pref and try it.

Comment 16

[Stefan [:StefanG_QA]]

If we set the pref "security.tls.version.fallback-limit" to "4" before or after the addon installation, the value is NOT changed to the default after we restart the browser. The addon is installed and the displayed information is correct. This is confirmed on all tested platforms.

We tried 10+ times per OS to hit the 10% cohort, but no success.

Comment 17

[Eric Rescorla (:ekr)]

By displayed information, do you mean that you can actually connect to Web sites and you get TLS 1.2 or TLS 1.3 (depending on the site)?

Specifically:

- www.allizom.org should show TLS 1.3
- www.google.com should show TLS 1.2

Comment 18

[Hani Yacoub, Desktop QA]

I tried to verify this on Firefox 60.0b8 and Firefox 60.0b9 with "beta-sysaddon" channel. After starting Firefox with a clean profile, the "security.tls.version.fallback-limit" pref is set by default to "3". Then we run the following code snippet in the Browser Console: Components.utils.import("resource://gre/modules/AddonManager.jsm"); AddonManagerPrivate.backgroundUpdateCheck(); 

In "about:support" under Firefox Features, the "TLS 1.3 gradual roll-out fallback-limit" is displayed, but "security.tls.version.fallback-limit" pref is set to 3, we tried 20+ times and the results were the same, the preference is not switched to 4.results were the same.

Please let me know if I'm doing something wrong.
Thanks.

Comment 19

[Stefan [:StefanG_QA]]

With the addon installed, when visit the both pages I can see the follow: 

- www.allizom.org - Connection Encrypted (TLS_AES_128_GCM_SHA256, 128 bit keys, TLS 1.3)

- www.google.com - Connection Encrypted (TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, 128 bit keys, TLS 1.2)

Comment 20

[Liz Henry (:lizzard) (relman/hg->git project)]

Sounds like this is ready to go, and testing shows us that it functions as expected, it's just hard to hit the right "cohort" on testing. rhelmer confirmed on irc so I think we're on track to launch this on beta 60.

Comment 21

[Liz Henry (:lizzard) (relman/hg->git project)]

This is set up in balrog and signed off so it should be rolling out shortly.

Comment 22

[Robert Helmer [:rhelmer]]

Attachment: TLS 1.3 fallback-limit roll-out SAO v2 (unsigned), 50%

Could you please sign this system add-on update? Thanks!

Comment 23

[:wezhou]

Attachment: TLS 1.3 fallback-limit roll-out SAO v2 (signed), 50%

Signed file attached. Please test.

Comment 24

[Robert Helmer [:rhelmer]]

Comment on attachment 8969128 [details]
TLS 1.3 fallback-limit roll-out SAO v2 (signed), 50%

Rehan, could you please put this up on the beta-sysaddon test channel and also stage for beta? Thanks!

Comment 25

[Rehan Dalal (Inactive)]

This is live on beta-sysaddon and pending signoff on beta.

Comment 26

[Eric Rescorla (:ekr)]

Before we roll out, can we please get this tested to verify it's actually working? It's set for 50%, so you should have no trouble getting it to trigger, if you try 2-4 separate profiles.

Comment 27

[Stefan [:StefanG_QA]]

We have tested this on Windows 10 x64, Windows 7 x86 and OS X 10.13 using Firefox 60.0b14 on "beta-sysaddon" channel. When starting Firefox with a new clean profile, the "security.tls.version.fallback-limit" pref is set by default to "3". Then we run backgroundUpdateCheck in the Browser Console.

In "about:support" under Firefox Features, the "TLS 1.3 gradual roll-out fallback-limit" is displayed and "security.tls.version.fallback-limit" pref is set to 4 by default. Restarting the browser does NOT revert back the pref. The pref keeps the default value to 4.

With the addon installed, when visiting the below pages we can see the following: 

- www.allizom.org - Connection Encrypted (TLS_AES_128_GCM_SHA256, 128 bit keys, TLS 1.3)

- www.google.com - Connection Encrypted (TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, 128 bit keys, TLS 1.2)

Comment 28

[Rehan Dalal (Inactive)]

This is all set up on my end. Once we are ready to roll out this needs relman sign off in Balrog and it will go out to users.

Comment 29

[Eric Rescorla (:ekr)]

I have checked the numbers for the beta rollout and I think this is ready to go once relman is happy.

Comment 30

[Rehan Dalal (Inactive)]

Liz could you please sign off on the rule #794 for the beta channel? There is only one rule for this bug.

Comment 31

[Liz Henry (:lizzard) (relman/hg->git project)]

No problem. I signed off in balrog just now.

Comment 32

[Robert Helmer [:rhelmer]]

This is done for 60.

For 61 we're going to test out Normandy for pref roll-out (bug 1462164)