Closed Bug 1529437 Opened 5 years ago Closed 5 years ago

[Shield] DNS Over HTTPS V6 (TRR), stable perf study, release 66

Categories

(Shield :: Shield Study, enhancement)

enhancement
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: jkt, Assigned: jkt)

References

()

Details

Attachments

(8 files, 7 obsolete files)

52 bytes, patch
johannh
: review+
Details | Diff | Splinter Review
24.62 KB, application/json
valentin
: review+
tdsmith
: review+
Details
30.12 KB, application/json
Details
3.83 KB, text/plain
chutten
: data-review+
Details
144.59 KB, application/x-xpinstall
Details
139.69 KB, application/zip
Details
144.60 KB, application/x-xpinstall
Details
144.58 KB, application/x-xpinstall
Details

Second perf study for DOH

Group: client-services-security
Group: mozilla-employee-confidential
Attached file doh-beta-data-review-2.md (obsolete) —
Attachment #9045499 - Flags: data-review?

This is an example payload that would be sent to Shield. Is there any changes we want to gather and send back that is different.

This is an example payload of a user with a FB auth cookie. Without the cookie the facebook entries would have blank arrays for those URLs.

Attachment #9045504 - Flags: review?(valentin.gosu)
Attachment #9045504 - Flags: review?(tdsmith)
Group: client-services-security
Attachment #9045499 - Flags: data-review? → data-review?(chutten)

(In reply to Jonathan Kingston [:jkt] from comment #3)

Created attachment 9045504 [details]
shield-telemetry-payload.json

This is an example payload that would be sent to Shield. Is there any changes we want to gather and send back that is different.

This is an example payload of a user with a FB auth cookie. Without the cookie the facebook entries would have blank arrays for those URLs.

To start thinking about this, I opened a PR to try and document the schema of the fields: https://github.com/jonathanKingston/http-dns/pull/34

I took my best guess at what each of the fields is. Does this look right? Anywhere there's a ? I'd like to know more about the range of values the field can take and what they mean.

Summary: [Shield] DNS Over HTTPS V5 (TRR), stable perf study, release 65 → [Shield] DNS Over HTTPS V6 (TRR), stable perf study, release 65
Comment on attachment 9045499 [details]
doh-beta-data-review-2.md

Can you please point to "documentation that describes the schema for the ultimate data set available publicly, complete and accurate?". I need that for the Data Review response: https://github.com/mozilla/data-review/blob/master/review.md
Attachment #9045499 - Flags: data-review?(chutten)
Attachment #9045504 - Flags: review?(valentin.gosu) → review+
Attachment #9045504 - Flags: review?(tdsmith) → review+
Summary: [Shield] DNS Over HTTPS V6 (TRR), stable perf study, release 65 → [Shield] DNS Over HTTPS V6 (TRR), stable perf study, release 66

Updated to Release 66.

Attached file data-with-fb-ecs.json

Updated study data to include samples of ecs and reduction of keys.

Attachment #9045499 - Attachment is obsolete: true

I added the link to the document for the schema that :tdsmith worked on https://github.com/jonathanKingston/http-dns/pull/33/files#diff-cd562a6cc04b2136747946c9a83c16ca I also uploaded a sample of data to this bug too: https://bug1529437.bmoattachments.org/attachment.cgi?id=9051954&t=22pdYepGCvqB5vPcEqyv0g

Attachment #9051955 - Flags: data-review?(chutten)
Comment on attachment 9051955 [details]
doh-beta-data-review-2.md

Clearing data-review as :tdsmith mentions some discussions underway.
Attachment #9051955 - Flags: data-review?(chutten)

Removing confidential flag as agreed it isn't required.

Group: mozilla-employee-confidential
Attached file dns_over_https-6.0.0.zip (obsolete) —

Hey :mythmon are you able to sign this addon please?

Flags: needinfo?(mcooper)
Attachment #9045495 - Flags: review?(jhofmann)

Hold fire on signing this I need to change something sorry.

Flags: needinfo?(mcooper)
Attached file dns_over_https-6.0.1.zip (obsolete) —

:mythmon are you able to sign this one please.

Attachment #9053559 - Attachment is obsolete: true
Flags: needinfo?(mcooper)
Comment on attachment 9051955 [details]
doh-beta-data-review-2.md

I think we are ready for review again :)
Attachment #9051955 - Flags: review?(chutten)
Flags: needinfo?(mcooper)

:mythmon I am seeing the following in the console when loading the addon via about:addons

1553688547968 addons.xpi-utils WARN Add-on httpdns@shield.mozilla.org is not correctly signed.

Flags: needinfo?(mcooper)

Since this add-on has not (as far as I can tell) passed QA yet, it was given a preliminary signature for testing purposes. In order to install it in Firefox, you'll need to use an unbranded build and set the preference xpinstall.signatures.dev-root to true to allow the dev add-on root to be trusted. More details can be seen in this document.

Once the add-on has been tested and signed off on, a final signature will be given.

Flags: needinfo?(mcooper)
Comment on attachment 9051955 [details]
doh-beta-data-review-2.md

DATA COLLECTION REVIEW RESPONSE:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes: https://github.com/jonathanKingston/http-dns/pull/33/files#diff-cd562a6cc04b2136747946c9a83c16ca

    Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is a study so can be controlled through Firefox's Preferences.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

No. This collection will expire at the end of the 2-week study.

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction.
(There is a small Cat3 portion, but it has been deemed to have a sufficient mitigation in-place for default-on collection in release).

    Is the data collection request for default-on or default-off?

Default on for a select en-us release study population.

    Does the instrumentation include the addition of any new identifiers?

No.

    Is the data collection covered by the existing Firefox privacy notice?

Yes.

    Does there need to be a check-in in the future to determine whether to renew the data?

No. The study should self-disable.

---
Result: datareview+
Attachment #9051955 - Flags: review?(chutten) → data-review+
Attached file dns_over_https-6.0.2.zip (obsolete) —

Context-fill was dropped from the updated icon, I added it back in. I'm unable to test locally without signing though (hopefully it works with the test signing).

Attachment #9053564 - Attachment is obsolete: true
Flags: needinfo?(mcooper)
Flags: needinfo?(mcooper)
Attached file dns_over_https-6.0.3.zip (obsolete) —

Ugh sorry I just realise Bug 1394579 never landed and instead we depend on the addon id being @mozilla.org

:mythmon do you know if we can launch shield studies without the shield prefix ("httpdns@mozilla.org" instead of "httpdns@shield.mozilla.org")?

Attachment #9053779 - Attachment is obsolete: true
Attachment #9054290 - Attachment is obsolete: true
Attachment #9054292 - Attachment is obsolete: true

I think that it will be OK to leave out the shield prefix. There are some analysis that depend on this, and so we'll mess those up. However, I don't think it will be a blocker to launching the study, and any data analysis will still apply if we keep this in mind.

6.0.3 should be ok then assuming we don't need the analysis you mentioned. Would you be able to sign that one instead?
Thanks

Flags: needinfo?(mcooper)
Flags: needinfo?(mcooper)

DNS Over HTTPS v6 - 6.0.3 build

Firefox Release 66.0.2

We have finished testing the DNS Over HTTPS v6 (6.0.3 build) experiment.

We have found the following issue which we consider blocker for the experiment:
#41 - Telemetry perf pings are no longer generated after restarting the browser with the PerfInterval pref modified to 60000

QA’s recommendation: RED - DON’T SHIP

Reasoning:

  • Given the fact that the experiment gathers performance data through telemetry probes, restarting the browser seems to block the pings generation, hence, there will be no data collected from the end users.

  • Another concern is that one of the links displayed in the experiment’s description page redirects the users to a “Not found” blog page (issue no. #40).

Testing Summary:

Tested Platforms:

  • Windows 10 x64
  • MacOS 10.14
  • Ubuntu 16.04 x64

Tested Firefox versions:

  • Unbranded Firefox Release 66.0.2
Flags: shield-qa-

Hey would you be able to sign this again please.

Attachment #9054294 - Attachment is obsolete: true
Flags: needinfo?(mcooper)
Flags: needinfo?(mcooper)

DNS Over HTTPS v6 - 6.0.4 build
Firefox Release 66.0.2

We have finished testing the DNS Over HTTPS v6 (6.0.4 build) experiment.

QA’s recommendation: GREEN - SHIP IT

Reasoning:

  • The blocker mentioned in the previous sign-off (#41) was fixed and verified.
  • There were no new issues found during testing.

Testing Summary:

  • Verified and confirmed the fix for the blocker issue.
  • Performed regression testing to ensure that the fix doesn’t affect the user-facing behavior and telemetry probes.

Tested Platforms:

  • Windows 10 x64
  • MacOS 10.14
  • Ubuntu 16.04 x64

Tested Firefox versions:

  • Unbranded Firefox Release 66.0.2
Flags: shield-qa- → shield-qa+
Comment on attachment 9045495 [details] [diff] [review]
GitHub Pull Request

Review of attachment 9045495 [details] [diff] [review]:
-----------------------------------------------------------------

I had a few comments, but overall r=me.

Thanks!
Attachment #9045495 - Attachment is patch: true
Attachment #9045495 - Attachment mime type: text/x-github-pull-request → text/plain
Attachment #9045495 - Flags: review?(jhofmann) → review+

I've signed version 6.04 for release.

The experiment is complete; followup analysis work at bug 1525632.

Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.