[telemetry-experiment] Determine how many users are behind a TLS MITM inspection portal

RESOLVED FIXED

Status

()

Core
Security: PSM
P1
enhancement
RESOLVED FIXED
10 months ago
25 days ago

People

(Reporter: jcj, Assigned: keeler)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [psm-assigned])

Attachments

(1 attachment, 3 obsolete attachments)

(Reporter)

Description

10 months ago
One of the challenges we know about is that some of our users are affected by logical network appliances that man-in-the-middle (MITM) all of Firefox's connections. If their MITM appliance uses SHA-1, any action we take on SHA-1 as a whole will affect all their browsing.

This bit us last January, and seems to us to necessitate a careful approach. Right now we have no information about how common a state this is, so we're considering that maybe this is an opportunity for a Telemetry Experiment, where we write a simple addon that connects to a Mozilla HTTPS site and evaluates whether the certificate received is A) the one we expect to be there, and if not B) whether the MITM certificate is using SHA-1, and thus will eventually cause user-breakage.

I believe we can get the information we need without transmitting any information about the certificate we receive, so this should be anonymous, breaking users down into 4 buckets:

    Users who are behind a MITM proxy using SHA-1 with an imported root
    Users who are behind a MITM proxy using SHA-1 with a built-in root
    Users who are behind a MITM proxy not using SHA-1
    Users who are not behind a MITM proxy

To determine if we are behind a MITM proxy, I think we should be OK to fetch some mozilla.org website, though I do not know what precisely to load.
(Reporter)

Comment 1

10 months ago
Note: bsmedberg has approved this experiment's additional collection via email.
(Assignee)

Comment 2

10 months ago
For tracking purposes, I'm just going to assign this to JC for now.
Assignee: nobody → jjones
Priority: -- → P1
Whiteboard: [psm-assigned]
(Reporter)

Comment 3

10 months ago
Assigning to Keeler for now. The instructions on how Telemetry Experiments work are here: https://wiki.mozilla.org/Telemetry/Experiments

I'm hoping that the code used for this experiment can be rev'd to v2 to also adjust preferences and used as a hotfix. That'll be a separate bug, though.
Assignee: jjones → dkeeler
(Assignee)

Comment 4

10 months ago
Created attachment 8806885 [details] [diff] [review]
1311479-mitm-prevalence.diff

This is what I have so far - thoughts?
Attachment #8806885 - Flags: feedback?(jjones)
(Reporter)

Comment 5

10 months ago
Comment on attachment 8806885 [details] [diff] [review]
1311479-mitm-prevalence.diff

Review of attachment 8806885 [details] [diff] [review]:
-----------------------------------------------------------------

This looks like the right way to go. I'd r+ it.

::: experiments/mitm-prevalence-beta/code/bootstrap.js
@@ +118,5 @@
> +      let securityInfo = evt.target.channel.securityInfo
> +                           .QueryInterface(Ci.nsITransportSecurityInfo);
> +      if (securityInfo.securityState &
> +          Ci.nsIWebProgressListener.STATE_CERT_USER_OVERRIDDEN) {
> +        resolve({ error: "user-added certificate error override" });

If we see any of these, I am going to be amazed. It'll be amazing.

@@ +145,1 @@
>    console.log(result);

Think we should identify here what got logged, for the curious user? `"mitm-prevalence-beta experiment result: " + result` ?
Attachment #8806885 - Flags: feedback?(jjones) → feedback+
(Assignee)

Comment 6

10 months ago
Created attachment 8808340 [details] [diff] [review]
1311479-mitm-prevalence.diff (v2)

For context, this is based on the TLS 1.3 experiment (bug 1310338). One known issue is I'm not sure what to put for start/stop/duration times, and if the versions are correct.
Attachment #8806885 - Attachment is obsolete: true
Attachment #8808340 - Flags: review?(benjamin)
(Assignee)

Comment 7

10 months ago
(In reply to J.C. Jones [:jcj] from comment #5)
> This looks like the right way to go. I'd r+ it.

Thanks!

> ::: experiments/mitm-prevalence-beta/code/bootstrap.js
> @@ +118,5 @@
> 
> If we see any of these, I am going to be amazed. It'll be amazing.

Indeed.

> @@ +145,1 @@
> >    console.log(result);
> 
> Think we should identify here what got logged, for the curious user?
> `"mitm-prevalence-beta experiment result: " + result` ?

I added an identifier line of output (the other change to the patch was I updated some documentation).
(Reporter)

Comment 8

10 months ago
Note: There's some additional context for what we're thinking of doing After The Experiment here:

https://wiki.mozilla.org/Security/CryptoEngineering/SHA-1

Comment 9

10 months ago
Comment on attachment 8808340 [details] [diff] [review]
1311479-mitm-prevalence.diff (v2)

I'm going to review the following bits only:

the data-review, README.md primarily

The description of `error` is oddly vague. Is this a string value? Will that string contain "The connection to telemetry.mozilla.org timed out"? That seems very verbose. The rest of the items listed are clearly not verbatim. I'd like to be able to reconstruct an actual payload from the docs.

data-review=me with that clarified and cleaned up.

I also reviewed the manifest:

please change the ID to "mitm-prevalence-beta51@experiments.mozilla.org"

I think you need new start/end dates?

I did not review bootstrap.js and do not think I am the appropriate person to do so. Please find another reviewer who can review your NSS/algorithmic code. That person should also make sure that your actual telemetry ping matches the docs that I reviewed.
Attachment #8808340 - Flags: review?(benjamin) → feedback+
(Assignee)

Comment 10

10 months ago
Created attachment 8809515 [details] [diff] [review]
1311479-mitm-prevalence.diff (v3)

Thanks for the review!
I updated the documentation and included the expected results. I also changed the id and updated the dates to be from November 14th-28th. I'll get JC to review the code.
Attachment #8808340 - Attachment is obsolete: true
Attachment #8809515 - Flags: review?(benjamin)
(Assignee)

Updated

10 months ago
Attachment #8809515 - Flags: review?(jjones)
(Reporter)

Comment 11

10 months ago
Comment on attachment 8809515 [details] [diff] [review]
1311479-mitm-prevalence.diff (v3)

Review of attachment 8809515 [details] [diff] [review]:
-----------------------------------------------------------------

The telemetry ping code looks good to me, and it matches the technical description in the readme. r=jcj
Attachment #8809515 - Flags: review?(jjones) → review+
(Reporter)

Comment 12

9 months ago
Note: I think this is good to go w/o code review from bsmedberg. Seems like you can cancel his review.
Comment on attachment 8809515 [details] [diff] [review]
1311479-mitm-prevalence.diff (v3)

(In reply to Benjamin Smedberg [:bsmedberg] from comment #9)
...
> data-review=me with that clarified and cleaned up.
...

(In reply to J.C. Jones [:jcj] from comment #12)
> Note: I think this is good to go w/o code review from bsmedberg. Seems like
> you can cancel his review.

Ok - sounds good.
Attachment #8809515 - Flags: review?(benjamin)
(Assignee)

Updated

9 months ago
Depends on: 1317535
Created attachment 8810996 [details] [diff] [review]
patch with signed xpi
Attachment #8809515 - Attachment is obsolete: true
Attachment #8810996 - Flags: review+
From my understanding of https://wiki.mozilla.org/Telemetry/Experiments and https://wiki.mozilla.org/QA/Telemetry/Developing_a_Telemetry_Experiment the next step is to push this to staging for QA sign-off. Hence: https://hg.mozilla.org/webtools/telemetry-experiment-server/rev/475a4d815e0ace87c47cefdde6b48012d31fdab1
https://wiki.mozilla.org/Telemetry/Experiments says I'm supposed to refer to https://wiki.mozilla.org/QA/Telemetry to request QA sign-off, but I can't seem to find the details I'm looking for there, so ni? to RyanVM for next steps (should I just file another bug or does this one work?)
Flags: needinfo?(ryanvm)
Michelle, can your please take this?
Flags: needinfo?(ryanvm) → needinfo?(mfunches)
SV Vegas should be able to accommodate this, I will follow up with :keeler for requirements and priorities.
Flags: needinfo?(mfunches)
Kanchan will be taking over on the testing here. A signoff should be sent in sometime tomorrow.
Flags: needinfo?(kkumari)

Comment 20

9 months ago
Pre sign-off email for this experiment has been sent to Keeler. I will be sending formal sign-off after his review/feedback on that.
Flags: needinfo?(kkumari)
Hi Kanchan, I received your email, but I thought it might be best if you also comment here in the bug noting QA sign-off. Thanks!
Flags: needinfo?(kkumari)

Comment 22

9 months ago
QA completed testing of the Telemetry Experiment: "MitM Prevalence" - TLS MITM Inspection Portal using the Beta 51.0b2 on the following platforms:
1. Mac OS X 10.11
2. Windows 7
3. Windows 10
4. Windows 8.1
5. Ubuntu 16.04

Testing focused on following scenarios:  
1. Users who are not behind a MitM proxy
2. Users using hosts file for website block and redirect
3. User behind MitM, Root Trusted
4. Users behind MitM Root Untrusted
5. Users behind MitM Root Untrusted with Override (exception)

Document illustrating test scenarios and execution status can be found at: https://docs.google.com/spreadsheets/d/1meLmF35TUzbwv9jkIb28M9UCcCw1M7ikd2V0xiLbW2w/edit?ts=582f8a14#gid=1140829424. 

No bugs were logged and data is being accurately collected. 

QA signs off on this Telemetry Experiment. Thanks!
Flags: needinfo?(kkumari)
Thanks!
I had to update the start/end times of the experiment. Also, we wanted to get as complete a picture of the ecosystem as possible, so I also increased the sample rate to 100%:

https://hg.mozilla.org/webtools/telemetry-experiment-server/rev/08b7d157b12fca49d3d9f37bc604976c1ea1048b
(Reporter)

Updated

9 months ago
Blocks: 1321114
I filed bug 1322278 on deploying this.
Depends on: 1322278
Now that this has completed, how do I actually access the data collected?
Flags: needinfo?(benjamin)

Comment 26

8 months ago
Sent email introductions.
Flags: needinfo?(benjamin)
Depends on: 1323851
Well, I think we can call this a success.
Status: NEW → RESOLVED
Last Resolved: 8 months ago
Resolution: --- → FIXED
See Also: → bug 1384782
You need to log in before you can comment on or make changes to this bug.