develop system add-on to check the deployments of various mozilla properties

RESOLVED FIXED in Firefox 52

Status

()

defect
P1
normal
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: keeler, Assigned: keeler)

Tracking

unspecified
mozilla55
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox52 fixed, firefox53 fixed, firefox54 fixed, firefox55 fixed)

Details

(Whiteboard: [psm-assigned][go-faster-system-addon])

Attachments

(9 attachments)

We should develop and deploy a system add-on to verify that users are encountering Mozilla sites as Mozilla has deployed them (from the perspective of which certificates we're using, that is).
Whiteboard: [psm-assigned] → [psm-assigned][go-faster-system-addon]

Comment 2

2 years ago
mozreview-review
Comment on attachment 8845651 [details]
bug 1346017 - develop system add-on to check the deployments of various mozilla properties data-review=bsmedberg

https://reviewboard.mozilla.org/r/118786/#review120736

Functionality is correct.
Attachment #8845651 - Flags: review?(jjones) → review+

Comment 3

2 years ago
mozreview-review
Comment on attachment 8845651 [details]
bug 1346017 - develop system add-on to check the deployments of various mozilla properties data-review=bsmedberg

https://reviewboard.mozilla.org/r/118786/#review120792
Attachment #8845651 - Flags: review?(felipc) → review+
Comment on attachment 8845651 [details]
bug 1346017 - develop system add-on to check the deployments of various mozilla properties data-review=bsmedberg

Privacy/data review to :bsmedberg.
Attachment #8845651 - Flags: review?(benjamin)

Comment 5

2 years ago
mozreview-review
Comment on attachment 8845651 [details]
bug 1346017 - develop system add-on to check the deployments of various mozilla properties data-review=bsmedberg

https://reviewboard.mozilla.org/r/118786/#review121088

data-r=me with the doc nit fixed.

::: browser/extensions/deployment-checker/README.md:39
(Diff revision 1)
> +* mismatches -- a list of objects with the properties:
> +  * hostname -- the host for which a mismatch was detected
> +  * chain -- a list of base64-encoded strings representing the bytes of the
> +    certificates in the chain
> +
> +For example, if the add-on determined that the hosts "example1.mozilla.org" and

Please in this data doc include the ping name, which is deployment-checker. Also include whether this ping includes the client ID (it doesn't, which is fine if you don't need it).

::: browser/extensions/deployment-checker/bootstrap.js:235
(Diff revision 1)
> +  }
> +  console.log("deployment-checker results:");
> +  console.log(results);
> +  console.log("deployment-checker payload:");
> +  console.log(payload);
> +  return TelemetryController.submitExternalPing("deployment-checker", payload,

Please confirm with mreid that the pipeline is prepared to receive this ping type.
Attachment #8845651 - Flags: review?(benjamin) → review+
Comment hidden (mozreview-request)
Thanks for the reviews!

:mreid, is the pipeline prepared to receive this ping type? (see attachment 8845651 [details] for details)
Flags: needinfo?(mreid)
Note: mreid provided approval to continue via email ("Telemetry Pings and Data Sizes")  on 1 March, but I'll let him confirm here.
To summarize the email conversation, the discussion was about increasing a small fraction of the disableSHA1rollout pings[1] in size by up to 10kb per ping.

This sounds fine to me w.r.t. the ingestion pipeline.

[1] https://pipeline-cep.prod.mozaws.net/dashboard_output/graphs/analysis.moz_telemetry_doctype_error_ratio_monitor.disableSHA1rollout.html
Flags: needinfo?(mreid)

Comment 10

2 years ago
Pushed by dkeeler@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/8245e6ca72d4
develop system add-on to check the deployments of various mozilla properties data-review=bsmedberg r=bsmedberg,Felipe,jcj

Comment 11

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/8245e6ca72d4
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla55
Jason, could I get this signed so we can QA this? Thanks!
(note for folks following along: this includes the changes from bug 1347217 so as to not debug spew over everyone's consoles)
Flags: needinfo?(jthomas)
Flags: needinfo?(jthomas)
:whd can you add 'deployment-checker' doctype to schema configuration?
Please see attached.
Thanks!

Justin, could you confirm that this runs on release/52? The expected behavior is that it sends a telemetry ping with the name "deployment-checker" with the data '{ "version": "1.0", "mismatches": [] }'. In the browser console, you should also see the following output (not necessarily in this order) :

deployment-checker results:
incoming.telemetry.mozilla.org sends the expected certificate chain
telemetry.mozilla.org sends the expected certificate chain
addons.mozilla.org sends the expected certificate chain
services.addons.mozilla.org sends the expected certificate chain
aus5.mozilla.org sends the expected certificate chain
versioncheck.addons.mozilla.org sends the expected certificate chain
support.mozilla.org sends the expected certificate chain
ftp.mozilla.org sends the expected certificate chain
mozilla.org sends the expected certificate chain
bugzilla.mozilla.org sends the expected certificate chain
crash-reports.mozilla.com sends the expected certificate chain
releases.mozilla.com sends the expected certificate chain
download-installer.cdn.mozilla.net sends the expected certificate chain
firefox.settings.services.mozilla.com sends the expected certificate chain
push.services.mozilla.com sends the expected certificate chain
token.services.mozilla.com sends the expected certificate chain
shavar.services.mozilla.com sends the expected certificate chain
search.services.mozilla.com sends the expected certificate chain
Flags: needinfo?(jwilliams)
The following are my findings from my machine:
 	Mozilla/5.0 (Windows NT 6.3; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0

Telemetry Ping:

"deployment-checker@mozilla.org": {
          "blocklisted": false,
          "description": "Check that Users Encounter Mozilla Sites as Deployed by Mozilla",
          "name": "Site Deployment Checker",
          "userDisabled": false,
          "appDisabled": false,
          "version": "1.0",
          "scope": 1,
          "type": "extension",
          "foreignInstall": false,
          "hasBinaryComponents": false,
          "installDay": 17240,
          "updateDay": 17240,
          "signedState": 3,
          "isSystem": false

Browser Console:

deployment-checker results:
incoming.telemetry.mozilla.org sends expected certificate chain
telemetry.mozilla.org sends expected certificate chain
addons.mozilla.org sends expected certificate chain
services.addons.mozilla.org sends expected certificate chain
aus5.mozilla.org sends expected certificate chain
versioncheck.addons.mozilla.org sends expected certificate chain
support.mozilla.org sends expected certificate chain
ftp.mozilla.org sends expected certificate chain
mozilla.org sends expected certificate chain
bugzilla.mozilla.org sends expected certificate chain
crash-reports.mozilla.com sends expected certificate chain
releases.mozilla.com sends expected certificate chain
download-installer.cdn.mozilla.net sends expected certificate chain
firefox.settings.services.mozilla.com sends expected certificate chain
push.services.mozilla.com sends expected certificate chain
token.services.mozilla.com sends expected certificate chain
shavar.services.mozilla.com sends expected certificate chain
search.services.mozilla.com sends expected certificate chain

Keeler: Does the telemetry ping look alright?
Flags: needinfo?(jwilliams) → needinfo?(dkeeler)
The console output looks good. I think the telemetry ping that data comes from is a different ping that includes details about what's installed. I think there should be a different archived ping in the drop-down called "deployment-checker"?
Flags: needinfo?(dkeeler)
This is the drop-down titled "deployment-checker". The only instance of "deployment-checker" reads:

 "addons": "%7B972ce4c6-7e08-4474-a285-3208198ce6fd%7D:52.0,jid0-edalmuivkozlouyij0lpdx548bc%40jetpack:2.0.31,tls13-compat-ff51%40mozilla.org:1.0.4,aushelper%40mozilla.org:2.0,e10srollout%40mozilla.org:1.9,firefox%40getpocket.com:1.0.5,webcompat%40mozilla.org:1.0,deployment-checker%40mozilla.org:1.0"

If I am sure this is showing what is installed as my current flash version is shown after this.
adding ni
Flags: needinfo?(dkeeler)
This is where I've found the telemetry payload, Justin.
Flags: needinfo?(dkeeler) → needinfo?(jwilliams)
Perfect. Thank you. I can verify this works as expected.
Flags: needinfo?(jwilliams)
Great - thanks!
Cory - we'd like to coordinate getting this ready to ship, pending approval from release drivers.
Flags: needinfo?(cprice)
Julien, what's the best way to get release approval on this? (and do we have to land in mozilla-release first?) Thanks!
Flags: needinfo?(jcristau)
Yes, please request uplift to aurora/beta/release.

Also, has there been any testing for the case where the set of mismatches isn't (or shouldn't be) empty?
Flags: needinfo?(jcristau) → needinfo?(dkeeler)
That's probably a good idea.

Here's a test plan:

1. Install and start OWASP ZAP ( https://github.com/zaproxy/zaproxy/wiki/Downloads )
2. Export its root certificate ( Tools -> Options -> Dynamic SSL Certificates -> Save )
3. Start a debug build of Firefox (this won't work on a non-debug build)
4. Import the root certificate and trust it for websites/SSL ( about:preferences -> Advanced -> Certificates -> View Certificates -> Authorities -> Import ) (make a note of the certificate's SHA-256 hash when doing this - it should be a long upper-case hexadecimal string delimited by ':')
5. Configure Firefox to use the proxy ( about:preferences -> Advanced -> Network -> Settings -> Manual proxy configuration -> localhost:8080, check "use this proxy server for all protocols" )
6. In about:config, add a string preference "security.test.built_in_root_hash" and give it the value of the root certificate's hash from step 4 (note that you have to unset and reset this preference if you close and reopen Firefox, unfortunately)
7. Also in about:config, change the value of the preference "security.pki.name_matching_mode" to 0 (this works around a ZAP bug)
8. Install the add-on from this bug (attachment 8847747 [details])
9. Hopefully you'll get something that looks like the the attached screenshot

Justin - if you could give this a go, that would be great. Let me know if I should elaborate on any of these steps. Thanks!
Flags: needinfo?(dkeeler) → needinfo?(jwilliams)
Posted image log1.png
Here is a screenshot of my logs.
Flags: needinfo?(jwilliams)
Posted image log2.png
attachment 8848185 [details] looks good. How about the telemetry ping itself?
Flags: needinfo?(jwilliams)
Posted file Raw_Payload.txt
The telemetry looks just like yours. Everything looks great.
Flags: needinfo?(jwilliams)
Comment on attachment 8845651 [details]
bug 1346017 - develop system add-on to check the deployments of various mozilla properties data-review=bsmedberg

(it looks like this applies fine to aurora and beta as it landed - a separate patch is needed for release that I'll upload shortly)

Approval Request Comment
[Feature/Bug causing the regression]: n/a
[User impact if declined]: we want to confirm users are encountering the right certificates on Mozilla sites
[Is this code covered by automated tests?]: no
[Has the fix been verified in Nightly?]: yes
[Needs manual test from QE? If yes, steps to reproduce]: Already has been QA'd
[List of other uplifts needed for the feature/fix]: bug 1347217
[Is the change risky?]: not very
[Why is the change risky/not risky?]: this add-on causes Firefox to connect 18 Mozilla hosts once and send a telemetry ping
[String changes made/needed]: none
Attachment #8845651 - Flags: approval-mozilla-beta?
Attachment #8845651 - Flags: approval-mozilla-aurora?
Approval Request Comment
[Feature/Bug causing the regression]: n/a
[User impact if declined]: we want to confirm users are encountering the right certificates on Mozilla sites
[Is this code covered by automated tests?]: no
[Has the fix been verified in Nightly?]: yes
[Needs manual test from QE? If yes, steps to reproduce]: Already has been QA'd
[List of other uplifts needed for the feature/fix]: bug 1347217
[Is the change risky?]: not very
[Why is the change risky/not risky?]: this add-on causes Firefox to connect 18 Mozilla hosts once and send a telemetry ping
[String changes made/needed]: none
Attachment #8848252 - Flags: review+
Attachment #8848252 - Flags: approval-mozilla-release?
Comment on attachment 8845651 [details]
bug 1346017 - develop system add-on to check the deployments of various mozilla properties data-review=bsmedberg

A new system addon to check the deployments of various mozilla properties. Beta53+ & Aurora54+.
Attachment #8845651 - Flags: approval-mozilla-beta?
Attachment #8845651 - Flags: approval-mozilla-beta+
Attachment #8845651 - Flags: approval-mozilla-aurora?
Attachment #8845651 - Flags: approval-mozilla-aurora+
Comment on attachment 8848252 [details] [diff] [review]
1346017-deployment-checker-release.diff

thanks for the extra testing.  let's land this new system add-on on release.
Attachment #8848252 - Flags: approval-mozilla-release? → approval-mozilla-release+
(In reply to David Keeler [:keeler] (use needinfo?) from comment #22)
> Great - thanks!
> Cory - we'd like to coordinate getting this ready to ship, pending approval
> from release drivers.
Looks like you have a sign off from RelMan and QA. Could you package the XPI, and attach to a comment NI'ing :jason for a signed XPI. Example: Bug 1344345 comment 5

Could you also please socialize this to release-drivers@mozilla.org and gofaster@mozilla.org in the form of an "Intent to Ship" email[0]? You can note in it that you've already received RelMan/QA approval.

[0] https://wiki.mozilla.org/Firefox/Go_Faster/Process#.E2.80.9CIntent_to_Ship.E2.80.9D_email
Flags: needinfo?(cprice) → needinfo?(dkeeler)
Already done, see email "Intent to ship: Telemetry experiment to identify..." dated 6 March.
Flags: needinfo?(dkeeler)
(Also, the packaged, signed add-on is attachment 8847747 [details])
Cory, per comment #39 and #40, do you need anything else?

Thanks!
Flags: needinfo?(cprice)
This is on release.
Flags: needinfo?(cprice)
Blocks: 1353364
See Also: → 1350430
We need to start backing this out of all trees; I'm not sure how to make that happen myself.
Flags: needinfo?(dkeeler)
Filed bug 1353789.
Flags: needinfo?(dkeeler)
Duplicate of this bug: 1343609
You need to log in before you can comment on or make changes to this bug.