Closed Bug 1462099 Opened 6 years ago Closed 6 years ago

Gradual roll-out of TLS fallback-limit to 1.3 on release channel (60)

Categories

(Core :: Security: PSM, enhancement, P1)

enhancement

Tracking

()

RESOLVED FIXED
Tracking Status
firefox60 + fixed

People

(Reporter: rhelmer, Assigned: rhelmer)

Details

(Whiteboard: [psm-assigned])

Attachments

(8 files)

TLS 1.3 is already enabled on Release (currently version 60), we'd like to now do a gradual roll-out of the fallback-limit pref.

This is controlled by the "security.tls.version.fallback-limit" pref, which is currently set to 3 (TLS 1.2) on Release. The value we wish to roll out is 4 (TLS 1.3)

The plan is to use a system add-on (SAO) update, and to initially roll out to 1% of release users.
Please sign this system add-on update. Thanks!
Flags: needinfo?(wezhou)
Flags: needinfo?(jthomas)
Signed file attached. Please test.
Flags: needinfo?(wezhou)
Flags: needinfo?(jthomas)
Attachment #8976285 - Attachment description: TLS 1.3 fallback-limit roll-out SAO v1 for Fx 60 (unsigned), 1% of users → TLS 1.3 fallback-limit roll-out SAO v2 for Fx 60 (unsigned), 1% of users
Attachment #8976285 - Attachment filename: tls13-version-fallback-rollout-bug1448176@mozilla.org-v1.0.xpi → tls13-version-fallback-rollout-bug1448176@mozilla.org-v2.0.xpi
Comment on attachment 8976306 [details]
TLS 1.3 fallback-limit roll-out SAO v2 for Fx 60 (signed), 1% of users

Rehan, could you please stage this on the test channel, and also set it up pending relman sign-off on the release channel? Thanks!
Attachment #8976306 - Attachment description: signed.8976285.xpi → TLS 1.3 fallback-limit roll-out SAO v2 for Fx 60 (signed), 1% of users
Attachment #8976306 - Attachment filename: signed.8976285.xpi → tls13-version-fallback-rollout-bug1448176@mozilla.org-v2.0.xpi
Flags: needinfo?(rdalal)
This is now live on the test channel and pending sign-off on the release channel.
Flags: needinfo?(rdalal)
(In reply to Rehan Dalal [:rehan, :rdalal] from comment #4)
> This is now live on the test channel and pending sign-off on the release
> channel.

I've tested this using 60.0.1 on macOS using the release-sysaddon channel, I see TLS 1.3 gradual roll-out fallback-limit v2.0 (addon ID bug1462099@mozilla.org">tls13-version-fallback-rollout-bug1462099@mozilla.org) installed in about:support

I tried a few new profiles but didn't get in the lucky 1% (which at least shows there isn't a bug where we're flipping it for a very high percentage of profiles at least... since this code was already tested I'd be quite surprised to find such a thing)

This is ready for sign-off in balrog, please let me know if we require any additional testing etc.

Thanks!
Flags: needinfo?(jcristau)
Please hold this until we have figured out https://bugzilla.mozilla.org/show_bug.cgi?id=1462303
(In reply to Eric Rescorla (:ekr) from comment #6)
> Please hold this until we have figured out
> https://bugzilla.mozilla.org/show_bug.cgi?id=1462303
Flags: needinfo?(jcristau)
I think it's now safe to try this at 1%.
Signed off in balrog, this is live on release.
Please sign this system add-on update. Thanks!
Flags: needinfo?(wezhou)
Flags: needinfo?(jthomas)
Signed file attached. Please test.
Flags: needinfo?(wezhou)
Flags: needinfo?(jthomas)
Things look good on Spark, so once this has been tested, please ship.
Attachment #8981587 - Attachment description: TLS 1.3 fallback-limit roll-out SAO v2 for Fx 60 (unsigned), 5% of users → TLS 1.3 fallback-limit roll-out SAO v3 for Fx 60 (unsigned), 5% of users
Comment on attachment 8981640 [details]
TLS 1.3 fallback-limit roll-out SAO v3 for Fx 60 (signed), 5% of users

Rehan, could you please set this up on the release-sysaddon channel and on the release channel pending relman sign-off? Thanks!
Attachment #8981640 - Attachment description: signed.8981587.xpi → TLS 1.3 fallback-limit roll-out SAO v3 for Fx 60 (signed), 5% of users
Attachment #8981640 - Attachment filename: signed.8981587.xpi → tls13-version-fallback-rollout-bug1462099@mozilla.org-v3.0-signed.xpi
Flags: needinfo?(rdalal)
This has been set up on release-sysaddon and staged for sign off on release.
Flags: needinfo?(rdalal)
Julien, this is ready for sign-off. Do you want any additional testing? Below is what I've checked:

1) installs on macOS with release-sysaddon channel on latest Firefox 60 (did not change the TLS flag for me which is expected since I only have a 5% chance)

2) the only difference between v2 and v3 on https://archive.mozilla.org/pub/system-addons/tls13-version-fallback-rollout-bug1462099/ are the signatures and this diff:

diff -u -r bug1462099@mozilla.org-2.0-signed">tls13-version-fallback-rollout-bug1462099@mozilla.org-2.0-signed/bootstrap.js bug1462099@mozilla.org-3.0-signed">tls13-version-fallback-rollout-bug1462099@mozilla.org-3.0-signed/bootstrap.js
--- bug1462099@mozilla.org-2.0-signed">tls13-version-fallback-rollout-bug1462099@mozilla.org-2.0-signed/bootstrap.js	2018-05-16 12:25:43.000000000 -0700
+++ bug1462099@mozilla.org-3.0-signed">tls13-version-fallback-rollout-bug1462099@mozilla.org-3.0-signed/bootstrap.js	2018-05-29 11:39:56.000000000 -0700
@@ -12,10 +12,10 @@
 
 Cu.importGlobalProperties(["crypto", "TextEncoder"]);
 
-// initially roll out to 10%, we want to control this on the client rather than
-// depending on server-side throttling, as throttling cannot account for any
-// other concurrent gradual roll-outs.
-const ENABLE_PROB = 0.01;
+// we want to control this on the client rather than
+// depending on server-side throttling, as throttling
+// cannot account for any other concurrent gradual roll-outs.
+const ENABLE_PROB = 0.05;
 const DEBUG = false;
 const VERSION_FALLBACK_LIMIT_PREF = "security.tls.version.fallback-limit";
 
diff -u -r bug1462099@mozilla.org-2.0-signed">tls13-version-fallback-rollout-bug1462099@mozilla.org-2.0-signed/install.rdf bug1462099@mozilla.org-3.0-signed">tls13-version-fallback-rollout-bug1462099@mozilla.org-3.0-signed/install.rdf
--- bug1462099@mozilla.org-2.0-signed">tls13-version-fallback-rollout-bug1462099@mozilla.org-2.0-signed/install.rdf	2018-05-16 12:26:58.000000000 -0700
+++ bug1462099@mozilla.org-3.0-signed">tls13-version-fallback-rollout-bug1462099@mozilla.org-3.0-signed/install.rdf	2018-05-29 11:40:25.000000000 -0700
@@ -2,7 +2,7 @@
 <RDF xmlns="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:em="http://www.mozilla.org/2004/em-rdf#">
   <Description about="urn:mozilla:install-manifest">
     <em:id>bug1462099@mozilla.org">tls13-version-fallback-rollout-bug1462099@mozilla.org</em:id>
-    <em:version>2.0</em:version>
+    <em:version>3.0</em:version>
     <em:type>2</em:type>
     <em:bootstrap>true</em:bootstrap>
     <em:unpack>false</em:unpack>
Flags: needinfo?(jcristau)
Checked the blob diff:

$ diff -u <(curl -s https://aus-api.mozilla.org/api/v1/releases/Superblob-d90be28c143626d753568ed13e7db61b06820aa66d2d0f34b87256928597397f | jq .) <(curl -s https://aus-api.mozilla.org/api/v1/releases/Superblob-7b0041b2140d6b114b26abd52725a25c30cce1b8bb856fef64e66af89b78ba06 | jq .)
--- /dev/fd/63	2018-05-31 09:41:21.217680802 +0200
+++ /dev/fd/62	2018-05-31 09:41:21.217680802 +0200
@@ -1,7 +1,7 @@
 {
   "blobs": [
-    "bug1462099@mozilla.org-2.0">tls13-version-fallback-rollout-bug1462099@mozilla.org-2.0"
+    "bug1462099@mozilla.org-3.0">tls13-version-fallback-rollout-bug1462099@mozilla.org-3.0"
   ],
-  "name": "Superblob-d90be28c143626d753568ed13e7db61b06820aa66d2d0f34b87256928597397f",
+  "name": "Superblob-7b0041b2140d6b114b26abd52725a25c30cce1b8bb856fef64e66af89b78ba06",
   "schema_version": 4000
 }

and signed off.
Flags: needinfo?(jcristau)
Please sign this system add-on update. Thanks!
Flags: needinfo?(wezhou)
Flags: needinfo?(jthomas)
Signed file attached. Please test.
Flags: needinfo?(wezhou)
Flags: needinfo?(jthomas)
Comment on attachment 8984202 [details]
TLS 1.3 fallback-limit roll-out SAO v4 for Fx 60 (signed), 20% of users

Rehan, can you please put this on the test channel and queue up for release sign-off? Thanks!
Attachment #8984202 - Attachment description: signed.8984024.xpi → TLS 1.3 fallback-limit roll-out SAO v4 for Fx 60 (signed), 20% of users
Attachment #8984202 - Attachment filename: signed.8984024.xpi → tls13-version-fallback-rollout-bug1462099@mozilla.org-v4.0-signed.xpi
Flags: needinfo?(rdalal)
This is on the test channel and queued for sign off on release.
Flags: needinfo?(rdalal)
ekr, could you please let us know when you'd like to go to 20% for release channel? Thanks!
Flags: needinfo?(ekr)
Julien, I believe this is ready for sign-off (I spoke to ekr earlier this evening)

Below is what I've checked:
 
1) installs on macOS with release-sysaddon channel on latest Firefox 60 (I *did* see it change the TLS flag for me one out of two times, now that it's at 20%)

2) the only difference I see between v3 and v4 on
https://archive.mozilla.org/pub/system-addons/tls13-version-fallback-rollout-bug1462099/ are the signatures and this diff:

diff -r bug1462099@mozilla.org-3.0">tls13-version-fallback-rollout-bug1462099@mozilla.org-3.0/bootstrap.js bug1462099@mozilla.org-4.0">tls13-version-fallback-rollout-bug1462099@mozilla.org-4.0/bootstrap.js
18c18
< const ENABLE_PROB = 0.05;
---
> const ENABLE_PROB = 0.20;
diff -r bug1462099@mozilla.org-3.0">tls13-version-fallback-rollout-bug1462099@mozilla.org-3.0/install.rdf bug1462099@mozilla.org-4.0">tls13-version-fallback-rollout-bug1462099@mozilla.org-4.0/install.rdf
5c5
<     <em:version>3.0</em:version>
---
>     <em:version>4.0</em:version>
Flags: needinfo?(ekr) → needinfo?(jcristau)
This is now live.
Flags: needinfo?(jcristau)
Please sign this system add-on update - thanks!
Flags: needinfo?(wezhou)
Flags: needinfo?(jthomas)
Signed file attached. Please test.
Flags: needinfo?(jthomas)
Flags: needinfo?(wezhou)
This has been updated on the testing channel and is pending sign off on release.
Attachment #8986623 - Attachment description: signed.8986549.xpi → TLS 1.3 fallback-limit roll-out SAO v5 for Fx 60 (unsigned), 95% of users
Attachment #8986623 - Attachment filename: signed.8986549.xpi → tls13-version-fallback-rollout-bug1462099@mozilla.org-v5.0-signed.xpi
Things look good on Telemetry.

Martin, I see bug 1468892, but that isn't a fallback condition anyway. so I think we should move ahead.
Flags: needinfo?(martin.thomson)
Yep.  I think that the next step here is to flip the default.  Is that something we want to do in 61?
Flags: needinfo?(martin.thomson)
Attachment #8986623 - Attachment description: TLS 1.3 fallback-limit roll-out SAO v5 for Fx 60 (unsigned), 95% of users → TLS 1.3 fallback-limit roll-out SAO v5 for Fx 60 (signed), 95% of users
(In reply to Martin Thomson [:mt:] from comment #28)
> Yep.  I think that the next step here is to flip the default.  Is that
> something we want to do in 61?

Release management (specifically RyanVM who is the relman owner for 61) had some concerns about this since we've only rolled out to 10% on 61 Beta so far (this is really blocked on migrating our telemetry analysis over to Normandy, per bug 1462164, which I expect to be resolved soon).

I'll start an intent-to-ship email to release-drivers and cc: you.
rehan, is something blocking shipping this extension?
Flags: needinfo?(rdalal)
It should be live on the test channel (release-sysaddon).

It is still pending relman signoff on the release channel. Once relman has signed off it should go out to the everyone on the release channel.

Perhaps rhelmer has more information on why there might be a hold up on sign off?
Flags: needinfo?(rdalal) → needinfo?(rhelmer)
diff -Nr 4/bootstrap.js 5/bootstrap.js
18c18
< const ENABLE_PROB = 0.20;
---
> const ENABLE_PROB = 0.95;
diff -Nr 4/install.rdf 5/install.rdf
5c5
<     <em:version>4.0</em:version>
---
>     <em:version>5.0</em:version>

Signed off, 5.0 is now live.
Flags: needinfo?(rhelmer)
I guess we're done here for 60?
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: