1346017 - develop system add-on to check the deployments of various mozilla properties

Status: RESOLVED FIXED

(Core :: Security: PSM, defect, P1)

Description

[Dana Keeler (she/her) (use needinfo) [:keeler]]

We should develop and deploy a system add-on to verify that users are encountering Mozilla sites as Mozilla has deployed them (from the perspective of which certificates we're using, that is).

Comment 1

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Attachment: bug 1346017 - develop system add-on to check the deployments of various mozilla properties data-review=bsmedberg

Review commit: https://reviewboard.mozilla.org/r/118786/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/118786/

Comment 2

[J.C. Jones [:jcj] (he/they)]

Comment on attachment 8845651 [details]
bug 1346017 - develop system add-on to check the deployments of various mozilla properties data-review=bsmedberg

https://reviewboard.mozilla.org/r/118786/#review120736

Functionality is correct.

Comment 3

[:Felipe Gomes (needinfo for replies!)]

Comment on attachment 8845651 [details]
bug 1346017 - develop system add-on to check the deployments of various mozilla properties data-review=bsmedberg

https://reviewboard.mozilla.org/r/118786/#review120792

Comment 4

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Comment on attachment 8845651 [details]
bug 1346017 - develop system add-on to check the deployments of various mozilla properties data-review=bsmedberg

Privacy/data review to :bsmedberg.

Comment 5

[Benjamin Smedberg]

Comment on attachment 8845651 [details]
bug 1346017 - develop system add-on to check the deployments of various mozilla properties data-review=bsmedberg

https://reviewboard.mozilla.org/r/118786/#review121088

data-r=me with the doc nit fixed.

::: browser/extensions/deployment-checker/README.md:39
(Diff revision 1)
> +* mismatches -- a list of objects with the properties:
> +  * hostname -- the host for which a mismatch was detected
> +  * chain -- a list of base64-encoded strings representing the bytes of the
> +    certificates in the chain
> +
> +For example, if the add-on determined that the hosts "example1.mozilla.org" and

Please in this data doc include the ping name, which is deployment-checker. Also include whether this ping includes the client ID (it doesn't, which is fine if you don't need it).

::: browser/extensions/deployment-checker/bootstrap.js:235
(Diff revision 1)
> +  }
> +  console.log("deployment-checker results:");
> +  console.log(results);
> +  console.log("deployment-checker payload:");
> +  console.log(payload);
> +  return TelemetryController.submitExternalPing("deployment-checker", payload,

Please confirm with mreid that the pipeline is prepared to receive this ping type.

Comment 6

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Comment on attachment 8845651 [details]
bug 1346017 - develop system add-on to check the deployments of various mozilla properties data-review=bsmedberg

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/118786/diff/1-2/

Comment 7

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Thanks for the reviews!

:mreid, is the pipeline prepared to receive this ping type? (see attachment 8845651 [details] for details)

Comment 8

[J.C. Jones [:jcj] (he/they)]

Note: mreid provided approval to continue via email ("Telemetry Pings and Data Sizes")  on 1 March, but I'll let him confirm here.

Comment 9

[Mark Reid [:mreid]]

To summarize the email conversation, the discussion was about increasing a small fraction of the disableSHA1rollout pings[1] in size by up to 10kb per ping.

This sounds fine to me w.r.t. the ingestion pipeline.

[1] https://pipeline-cep.prod.mozaws.net/dashboard_output/graphs/analysis.moz_telemetry_doctype_error_ratio_monitor.disableSHA1rollout.html

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8245e6ca72d4
develop system add-on to check the deployments of various mozilla properties data-review=bsmedberg r=bsmedberg,Felipe,jcj

Comment 11

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/8245e6ca72d4

Comment 12

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Attachment: [e-mail]

Jason, could I get this signed so we can QA this? Thanks!
(note for folks following along: this includes the changes from bug 1347217 so as to not debug spew over everyone's consoles)

Comment 13

[Jason Thomas [:jason]]

:whd can you add 'deployment-checker' doctype to schema configuration?

Comment 14

[Jason Thomas [:jason]]

Attachment: [e-mail] signed

Please see attached.

Comment 15

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Thanks!

Justin, could you confirm that this runs on release/52? The expected behavior is that it sends a telemetry ping with the name "deployment-checker" with the data '{ "version": "1.0", "mismatches": [] }'. In the browser console, you should also see the following output (not necessarily in this order) :

deployment-checker results:
incoming.telemetry.mozilla.org sends the expected certificate chain
telemetry.mozilla.org sends the expected certificate chain
addons.mozilla.org sends the expected certificate chain
services.addons.mozilla.org sends the expected certificate chain
aus5.mozilla.org sends the expected certificate chain
versioncheck.addons.mozilla.org sends the expected certificate chain
support.mozilla.org sends the expected certificate chain
ftp.mozilla.org sends the expected certificate chain
mozilla.org sends the expected certificate chain
bugzilla.mozilla.org sends the expected certificate chain
crash-reports.mozilla.com sends the expected certificate chain
releases.mozilla.com sends the expected certificate chain
download-installer.cdn.mozilla.net sends the expected certificate chain
firefox.settings.services.mozilla.com sends the expected certificate chain
push.services.mozilla.com sends the expected certificate chain
token.services.mozilla.com sends the expected certificate chain
shavar.services.mozilla.com sends the expected certificate chain
search.services.mozilla.com sends the expected certificate chain

Comment 16

[Justin [:JW_SoftvisionQA]]

The following are my findings from my machine:
 	Mozilla/5.0 (Windows NT 6.3; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0

Telemetry Ping:

"[e-mail]": {
          "blocklisted": false,
          "description": "Check that Users Encounter Mozilla Sites as Deployed by Mozilla",
          "name": "Site Deployment Checker",
          "userDisabled": false,
          "appDisabled": false,
          "version": "1.0",
          "scope": 1,
          "type": "extension",
          "foreignInstall": false,
          "hasBinaryComponents": false,
          "installDay": 17240,
          "updateDay": 17240,
          "signedState": 3,
          "isSystem": false

Browser Console:

deployment-checker results:
incoming.telemetry.mozilla.org sends expected certificate chain
telemetry.mozilla.org sends expected certificate chain
addons.mozilla.org sends expected certificate chain
services.addons.mozilla.org sends expected certificate chain
aus5.mozilla.org sends expected certificate chain
versioncheck.addons.mozilla.org sends expected certificate chain
support.mozilla.org sends expected certificate chain
ftp.mozilla.org sends expected certificate chain
mozilla.org sends expected certificate chain
bugzilla.mozilla.org sends expected certificate chain
crash-reports.mozilla.com sends expected certificate chain
releases.mozilla.com sends expected certificate chain
download-installer.cdn.mozilla.net sends expected certificate chain
firefox.settings.services.mozilla.com sends expected certificate chain
push.services.mozilla.com sends expected certificate chain
token.services.mozilla.com sends expected certificate chain
shavar.services.mozilla.com sends expected certificate chain
search.services.mozilla.com sends expected certificate chain

Keeler: Does the telemetry ping look alright?

Comment 17

[Dana Keeler (she/her) (use needinfo) [:keeler]]

The console output looks good. I think the telemetry ping that data comes from is a different ping that includes details about what's installed. I think there should be a different archived ping in the drop-down called "deployment-checker"?

Comment 18

[Justin [:JW_SoftvisionQA]]

This is the drop-down titled "deployment-checker". The only instance of "deployment-checker" reads:

 "addons": "%7B972ce4c6-7e08-4474-a285-3208198ce6fd%7D:52.0,jid0-edalmuivkozlouyij0lpdx548bc%40jetpack:2.0.31,tls13-compat-ff51%40mozilla.org:1.0.4,aushelper%40mozilla.org:2.0,e10srollout%40mozilla.org:1.9,firefox%40getpocket.com:1.0.5,webcompat%40mozilla.org:1.0,deployment-checker%40mozilla.org:1.0"

If I am sure this is showing what is installed as my current flash version is shown after this.

Comment 19

[Justin [:JW_SoftvisionQA]]

adding ni

Comment 20

[J.C. Jones [:jcj] (he/they)]

Attachment: finding the telemetry payload

This is where I've found the telemetry payload, Justin.

Comment 21

[Justin [:JW_SoftvisionQA]]

Perfect. Thank you. I can verify this works as expected.

Comment 22

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Great - thanks!
Cory - we'd like to coordinate getting this ready to ship, pending approval from release drivers.

Comment 23

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Julien, what's the best way to get release approval on this? (and do we have to land in mozilla-release first?) Thanks!

Comment 24

[Julien Cristau [:jcristau]]

Yes, please request uplift to aurora/beta/release.

Also, has there been any testing for the case where the set of mismatches isn't (or shouldn't be) empty?

Comment 25

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Attachment: expected-mismatch-results.png

That's probably a good idea.

Here's a test plan:

1. Install and start OWASP ZAP ( https://github.com/zaproxy/zaproxy/wiki/Downloads )
2. Export its root certificate ( Tools -> Options -> Dynamic SSL Certificates -> Save )
3. Start a debug build of Firefox (this won't work on a non-debug build)
4. Import the root certificate and trust it for websites/SSL ( about:preferences -> Advanced -> Certificates -> View Certificates -> Authorities -> Import ) (make a note of the certificate's SHA-256 hash when doing this - it should be a long upper-case hexadecimal string delimited by ':')
5. Configure Firefox to use the proxy ( about:preferences -> Advanced -> Network -> Settings -> Manual proxy configuration -> localhost:8080, check "use this proxy server for all protocols" )
6. In about:config, add a string preference "security.test.built_in_root_hash" and give it the value of the root certificate's hash from step 4 (note that you have to unset and reset this preference if you close and reopen Firefox, unfortunately)
7. Also in about:config, change the value of the preference "security.pki.name_matching_mode" to 0 (this works around a ZAP bug)
8. Install the add-on from this bug (attachment 8847747 [details])
9. Hopefully you'll get something that looks like the the attached screenshot

Justin - if you could give this a go, that would be great. Let me know if I should elaborate on any of these steps. Thanks!

Comment 26

[Justin [:JW_SoftvisionQA]]

Attachment: log1.png

Here is a screenshot of my logs.

Comment 27

[Justin [:JW_SoftvisionQA]]

Attachment: log2.png

Comment 28

[Dana Keeler (she/her) (use needinfo) [:keeler]]

attachment 8848185 [details] looks good. How about the telemetry ping itself?

Comment 29

[Justin [:JW_SoftvisionQA]]

Attachment: Raw_Payload.txt

The telemetry looks just like yours. Everything looks great.

Comment 30

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Awesome - thanks!

Comment 31

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Comment on attachment 8845651 [details]
bug 1346017 - develop system add-on to check the deployments of various mozilla properties data-review=bsmedberg

(it looks like this applies fine to aurora and beta as it landed - a separate patch is needed for release that I'll upload shortly)

Approval Request Comment
[Feature/Bug causing the regression]: n/a
[User impact if declined]: we want to confirm users are encountering the right certificates on Mozilla sites
[Is this code covered by automated tests?]: no
[Has the fix been verified in Nightly?]: yes
[Needs manual test from QE? If yes, steps to reproduce]: Already has been QA'd
[List of other uplifts needed for the feature/fix]: bug 1347217
[Is the change risky?]: not very
[Why is the change risky/not risky?]: this add-on causes Firefox to connect 18 Mozilla hosts once and send a telemetry ping
[String changes made/needed]: none

Comment 32

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Attachment: 1346017-deployment-checker-release.diff

Approval Request Comment
[Feature/Bug causing the regression]: n/a
[User impact if declined]: we want to confirm users are encountering the right certificates on Mozilla sites
[Is this code covered by automated tests?]: no
[Has the fix been verified in Nightly?]: yes
[Needs manual test from QE? If yes, steps to reproduce]: Already has been QA'd
[List of other uplifts needed for the feature/fix]: bug 1347217
[Is the change risky?]: not very
[Why is the change risky/not risky?]: this add-on causes Firefox to connect 18 Mozilla hosts once and send a telemetry ping
[String changes made/needed]: none

Comment 33

[Gerry Chang [:gchang]]

Comment on attachment 8845651 [details]
bug 1346017 - develop system add-on to check the deployments of various mozilla properties data-review=bsmedberg

A new system addon to check the deployments of various mozilla properties. Beta53+ & Aurora54+.

Comment 34

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/6ab8262a211b

Comment 35

[Julien Cristau [:jcristau]]

Comment on attachment 8848252 [details] [diff] [review]
1346017-deployment-checker-release.diff

thanks for the extra testing.  let's land this new system add-on on release.

Comment 36

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/releases/mozilla-beta/rev/81b3ab51eafe

Comment 37

[Cory Price [:ckprice] (bugmail disabled, NI me!)]

(In reply to David Keeler [:keeler] (use needinfo?) from comment #22)
> Great - thanks!
> Cory - we'd like to coordinate getting this ready to ship, pending approval
> from release drivers.
Looks like you have a sign off from RelMan and QA. Could you package the XPI, and attach to a comment NI'ing :jason for a signed XPI. Example: Bug 1344345 comment 5

Could you also please socialize this to [e-mail] and [e-mail] in the form of an "Intent to Ship" email[0]? You can note in it that you've already received RelMan/QA approval.

[0] https://wiki.mozilla.org/Firefox/Go_Faster/Process#.E2.80.9CIntent_to_Ship.E2.80.9D_email

Comment 38

[Cory Price [:ckprice] (bugmail disabled, NI me!)]

(In reply to Cory Price [:ckprice] from comment #37)
> [0] https://wiki.mozilla.org/Firefox/Go_Faster/Process#.E2.80.9CIntent_to_Ship. E2.80.9D_email

Better link: https://wiki.mozilla.org/Firefox/Go_Faster/System_Add-ons/Process#Intent_to_Ship_and_RelMan_Approval

Comment 39

[J.C. Jones [:jcj] (he/they)]

Already done, see email "Intent to ship: Telemetry experiment to identify..." dated 6 March.

Comment 40

[Dana Keeler (she/her) (use needinfo) [:keeler]]

(Also, the packaged, signed add-on is attachment 8847747 [details])

Comment 41

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-release/rev/2d1bf6984dde

Comment 42

[J.C. Jones [:jcj] (he/they)]

Cory, per comment #39 and #40, do you need anything else?

Thanks!

Comment 43

[Cory Price [:ckprice] (bugmail disabled, NI me!)]

This is on release.

Comment 44

[J.C. Jones [:jcj] (he/they)]

We need to start backing this out of all trees; I'm not sure how to make that happen myself.

Comment 45

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Filed bug 1353789.