Add Telemetry for (schemeless) HTTPS-First
Categories
(Core :: DOM: Security, task, P2)
Tracking
()
Tracking | Status | |
---|---|---|
firefox125 | --- | fixed |
People
(Reporter: maltejur, Assigned: maltejur)
References
(Blocks 2 open bugs)
Details
(Whiteboard: [domsecurity-active])
Attachments
(2 files, 5 obsolete files)
48 bytes,
text/x-phabricator-request
|
Details | Review | |
6.18 KB,
text/plain
|
chutten
:
data-review+
|
Details |
As schemeless HTTPS-First (https as the default scheme for address bar inputs without a scheme, with a fallback to http) has recently landed in Nightly (Bug 1863281), before we proceed to enable it by default in release, we want to add telemetry for HTTPS-First. We want to measure the benefit this feature brings vs in how much load time overhead and annoyance it results in if a upgrade is not possible. More specifically, we want to answer these questions:
- Which percentage and amount of url bar inputs get marked as "schemeless" and thus get marked for a potential HTTPS-First upgrade?
- Which percentage of HTTPS-First upgrades succeed and which percentage gets downgraded again?
- If a upgrade isn't possible, which percentage of downgrades happen because the https request failed and which percentage happens because the extra http request fired after 3s got a response first?
- Of the downgrades that happen because the https request failed, what is the average time overhead / what is the time between the first load start and the downgrade?
Updated•7 months ago
|
Assignee | ||
Comment 1•7 months ago
|
||
Assignee | ||
Comment 2•7 months ago
|
||
Comment hidden (obsolete) |
Assignee | ||
Updated•7 months ago
|
Assignee | ||
Comment 4•7 months ago
•
|
||
Freddy, can you take a look at my data review draft? Open things from me:
a. I am unsure what to write for how we are going to analyze the data (10. and 11.). Should I just write that we are going to create a dashboard for internal team use to look at the data?
b. Should I add you to the list of people responsible for the collection (7.)?
Assignee | ||
Comment 5•6 months ago
|
||
Comment 6•5 months ago
|
||
Comment on attachment 9372782 [details] data-review.txt Looks great. Some minor comments inline. >DATA REVIEW REQUEST > >(Context) > >HTTPS-First is a mechanism enabled in private browsing that upgrades all top level loads to HTTPS, with a mechanism to fall back to HTTP if the HTTPS site can't be loaded. > >Schemeless HTTPS-First means that HTTPS-First will be enabled for urlbar inputs that are being fixed up to URI and don't have a scheme, essentially making HTTPS the default scheme for the urlbar with a fallback to HTTP. > How about removing the first sentence and trimming away some jargon. Maybe like so? "We intend to upgrade loads from the address bar to HTTPS, when the explicit URL scheme is missing (i.e., "schemeless"). This upgrade happens with a fallback to HTTP, when a secure channel is not available. To better evaluate the performance and security impact of this new mechanism, we want to measure the number of affected requests as well as added overhead of our request upgrades" >1. What questions will you answer with this data? > >- Which percentage and amount of url bar inputs get marked as "schemeless" and thus get marked for a potential HTTPS-First upgrade? >- Which percentage of HTTPS-First upgrades succeed and which percentage gets downgraded again? >- If a upgrade isn't possible, which percentage of downgrades happen because the extra http request fired after 3s got a response first? >- If a upgrade isn't possible, what is the average time overhead caused by HTTPS-First? > >2. Why does Mozilla need to answer these questions? Are there benefits for users? > Do we need this information to address product or business requirements? > >Currently, schemeless HTTPS-First is enabled in Nightly and early beta. Before we ship it to release, we want to make sure it does not significantly worsen the page load times for most users on sites not supporting HTTP. In addition to that, we s/HTTP/HTTPS/. It also looks like the last sentence is incomplete? > >3. What alternative methods did you consider to answer these questions? > Why were they not sufficient? > >I already did testing for all of these questions locally on some sites, but since HTTPS-First can behave differently on different sites and network conditions, my testing isn't an adequate answer to these questions. > >4. Can current instrumentation answer these questions? > >The first question can be answered by combining the existing urlbar telemetry and the new `httpsfirst.upgraded_schemeless` metric. The relevant urlbar telemetry can be found here: https://sql.telemetry.mozilla.org/queries/96161#237499 > >5. List all proposed measurements and indicate the category of data collection for each > measurement, using the Firefox data collection categories found on the Mozilla wiki. > >Measurement Name | Measurement Description | Data Collection Category | Tracking Bug >---------------- | ----------------------- | ------------------------ | ------------ >`httpsfirst.downgrade_time` | If a HTTPS-First upgrade isn't successful, measures the timespan between the navigation start and the downgrade. This is essentially the overhead caused by HTTPS-First if a site does not support HTTPS. | Technical data | https://bugzilla.mozilla.org/show_bug.cgi?id=1868380 >`httpsfirst.downgraded` | How many of the HTTPS-First upgrades get downgraded again. | Technical data | https://bugzilla.mozilla.org/show_bug.cgi?id=1868380 >`httpsfirst.downgraded_faster_http_request` | How many of the HTTPS-First upgrades get downgraded again because the HTTP request fired after 3s received a answer faster than the HTTPS request. | Technical data | https://bugzilla.mozilla.org/show_bug.cgi?id=1868380 >`httpsfirst.upgraded` | Counts how often a load is marked to be upgraded to HTTPS because of HTTPS-First. | Technical data | https://bugzilla.mozilla.org/show_bug.cgi?id=1868380 >`httpsfirst.upgraded_schemeless` | How many of the HTTPS-First upgrades are happening while the load is marked as schemeless. | Technical data | https://bugzilla.mozilla.org/show_bug.cgi?id=1868380 > >6. Please provide a link to the documentation for this data collection which > describes the ultimate data set in a public, complete, and accurate way. > >This collection is Glean so is documented [in the Glean Dictionary](https://dictionary.telemetry.mozilla.org). > ^--- grammar. >7. How long will this data be collected? > >This collection will be collected permanently. >seceng-telemetry@mozilla.com and mjurgens@mozilla.com will be responsible for the permanent collections. > ...collected permanently until we succeeded in shipping the feature for all of our users I think we intend to remove it when we are confident in its robustness, don't we? >8. What populations will you measure? > >All channels, countries, and locales. No filters. > >9. If this data collection is default on, what is the opt-out mechanism for users? > >These collections are Glean. The opt-out can be found in the product's preferences. "are using glean"? > >10. Please provide a general description of how you will analyze this data. > >We are going to create an internal dashboard to evaluate the data. Depending on the results, we will either continue with shipping schemeless HTTPS-First, or add experiments to improve the behavior of schemeless HTTPS-First. > >11. Where do you intend to share the results of your analysis? > >This is primarily meant for internally evaluating the effectivenes of schemeless HTTPS-First. If we successfully ship schemeless HTTPS-First though, we may publish some of the data publicly, for example as part of a blog post. typo: effectiveness. "some of the data" -> "our aggregate findings"? > >12. Is there a third-party tool (i.e. not Glean or Telemetry) that you > are proposing to use for this data collection? > >No.
Assignee | ||
Comment 7•5 months ago
•
|
||
Thanks.
How about removing the first sentence and trimming away some jargon. Maybe like so? "We intend to upgrade loads from the address bar to HTTPS, when the explicit URL scheme is missing (i.e., "schemeless"). This upgrade happens with a fallback to HTTP, when a secure channel is not available. To better evaluate the performance and security impact of this new mechanism, we want to measure the number of affected requests as well as added overhead of our request upgrades"
Sounds good
It also looks like the last sentence is incomplete?
Oh right, I've removed that sentence now. Sorry about that.
^--- grammar.
"are using glean"?
These sentences are automatically generated by Glean, so I suppose they are supposed to be like that? :D
...collected permanently until we succeeded in shipping the feature for all of our users
I think we intend to remove it when we are confident in its robustness, don't we?
But we also want to keep this telemetry for shipping HTTPS-First itself, right? I've changed the sentence to "This collection will be collected permanently until we succeeded in shipping HTTPS-First for all of our users." for now.
"some of the data" -> "our aggregate findings"?
Yes, that does sound better.
Comment 8•5 months ago
|
||
Can we clarify in the context section that this is for HTTPS first including schemeless and clarify which measurements apply to what. Currently, it's not clear to me which of these will be triggered by schemeless upgrades, HTTPS first upgrade and HTTPS only upgrades.
Assignee | ||
Comment 9•5 months ago
|
||
Thanks Simon, I added a short paragraph to the context, explaining that a bit better.
Updated•5 months ago
|
Comment 10•5 months ago
|
||
Comment on attachment 9376017 [details]
data-review.txt
DATA COLLECTION REVIEW RESPONSE:
Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?
Yes.
Is there a control mechanism that allows the user to turn the data collection on and off?
Yes. This collection can be controlled through the product's preferences.
If the request is for permanent data collection, is there someone who will monitor the data over time?
Yes, seceng-telemetry@mozilla.com and mjurgens@mozilla.com will be responsible for the permanent collections.
Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?
Category 1, Technical.
Is the data collection request for default-on or default-off?
Default on for all channels.
Does the instrumentation include the addition of any new identifiers?
No.
Is the data collection covered by the existing Firefox privacy notice?
Yes.
Does the data collection use a third-party collection tool?
No.
Result: datareview+
Updated•5 months ago
|
Assignee | ||
Updated•5 months ago
|
Assignee | ||
Comment 11•4 months ago
|
||
Re-requesting data-review as the measurements have changed (only section 5 is different).
Assignee | ||
Updated•4 months ago
|
Comment 12•4 months ago
|
||
Comment on attachment 9388989 [details]
data-review.txt
DATA COLLECTION REVIEW RESPONSE:
Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?
Yes.
Is there a control mechanism that allows the user to turn the data collection on and off?
Yes. This collection can be controlled through the product's preferences.
If the request is for permanent data collection, is there someone who will monitor the data over time?
seceng-telemetry@mozilla.com and mjurgens@mozilla.com will be responsible for the permanent collections.
Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?
Category 1, Technical.
Is the data collection request for default-on or default-off?
Default on for all channels.
Does the instrumentation include the addition of any new identifiers?
No.
Is the data collection covered by the existing Firefox privacy notice?
Yes.
Does the data collection use a third-party collection tool?
No.
Result: datareview+
Comment 13•4 months ago
|
||
Pushed by mjurgens@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/8c0e44e4d9ae Add glean telemetry for HTTPS-First r=freddyb,simonf
Comment 14•4 months ago
•
|
||
Backed out for causing py3 failures on test_yaml_indices.py
- backout: https://hg.mozilla.org/integration/autoland/rev/e462210bf1b5ec21b23a8224916b5c8b1895397f
- push: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&revision=8c0e44e4d9ae9b49abf96979e8ee9794ba2e394c
- failure log:
[task 2024-03-06T10:14:28.071Z] ============================= test session starts ==============================
[task 2024-03-06T10:14:28.071Z] platform linux -- Python 3.8.10, pytest-7.0.1, pluggy-1.4.0 -- /builds/worker/.mozbuild/srcdirs/gecko-8a5b87fe5d69/_virtualenvs/python-test/bin/python
[task 2024-03-06T10:14:28.071Z] rootdir: /builds/worker/checkouts/gecko, configfile: config/mozunit/mozunit/pytest.ini
[task 2024-03-06T10:14:28.074Z] plugins: mock-3.12.0
[task 2024-03-06T10:14:28.074Z] collecting ... collected 1 item
[task 2024-03-06T10:14:28.074Z]
[task 2024-03-06T10:14:28.074Z] toolkit/components/glean/tests/pytest/test_yaml_indices.py::test_yamls_sorted TEST-UNEXPECTED-FAIL
[task 2024-03-06T10:14:28.074Z]
[task 2024-03-06T10:14:28.074Z] =================================== FAILURES ===================================
[task 2024-03-06T10:14:28.074Z] ______________________________ test_yamls_sorted _______________________________
[task 2024-03-06T10:14:28.074Z] toolkit/components/glean/tests/pytest/test_yaml_indices.py:36: in test_yamls_sorted
[task 2024-03-06T10:14:28.074Z] assert (
[task 2024-03-06T10:14:28.074Z] E AssertionError: gecko_metrics must be be lexicographically sorted.
[task 2024-03-06T10:14:28.074Z] E assert ['browser/base/content/metrics.yaml',\n 'docshell/base/metrics.yaml',\n 'dom/base/use_counter_metrics.yaml',\n 'dom/media/metrics.yaml',\n 'dom/media/webrtc/metrics.yaml',\n 'dom/metrics.yaml',\n 'dom/performance/metrics.yaml',\n 'dom/security/metrics.yaml',\n 'gfx/metrics.yaml',\n 'image/decoders/metrics.yaml',\n 'js/xpconnect/metrics.yaml',\n 'layout/base/metrics.yaml',\n 'mobile/android/actors/metrics.yaml',\n 'mobile/android/modules/geckoview/metrics.yaml',\n 'netwerk/metrics.yaml',\n 'netwerk/protocol/http/metrics.yaml',\n 'security/manager/ssl/metrics.yaml',\n 'toolkit/components/cookiebanners/metrics.yaml',\n 'toolkit/components/extensions/metrics.yaml',\n 'toolkit/components/formautofill/metrics.yaml',\n 'toolkit/components/glean/metrics.yaml',\n 'toolkit/components/passwordmgr/metrics.yaml',\n 'toolkit/components/pdfjs/metrics.yaml',\n 'toolkit/components/processtools/metrics.yaml',\n 'toolkit/components/reportbrokensite/metrics.yaml',\n 'toolkit/components/resistfingerprinting/metrics.yaml',\n 'toolkit/components/translations/metrics.yaml',\n 'toolkit/mozapps/extensions/metrics.yaml',\n 'toolkit/mozapps/handling/metrics.yaml',\n 'toolkit/xre/metrics.yaml',\n 'xpcom/metrics.yaml'] == ['browser/base/content/metrics.yaml',\n 'docshell/base/metrics.yaml',\n 'dom/base/use_counter_metrics.yaml',\n 'dom/media/metrics.yaml',\n 'dom/media/webrtc/metrics.yaml',\n 'dom/security/metrics.yaml',\n 'dom/metrics.yaml',\n 'dom/performance/metrics.yaml',\n 'gfx/metrics.yaml',\n 'image/decoders/metrics.yaml',\n 'js/xpconnect/metrics.yaml',\n 'layout/base/metrics.yaml',\n 'mobile/android/actors/metrics.yaml',\n 'mobile/android/modules/geckoview/metrics.yaml',\n 'netwerk/metrics.yaml',\n 'netwerk/protocol/http/metrics.yaml',\n 'security/manager/ssl/metrics.yaml',\n 'toolkit/components/cookiebanners/metrics.yaml',\n 'toolkit/components/extensions/metrics.yaml',\n 'toolkit/components/formautofill/metrics.yaml',\n 'toolkit/components/glean/metrics.yaml',\n 'toolkit/components/passwordmgr/metrics.yaml',\n 'toolkit/components/pdfjs/metrics.yaml',\n 'toolkit/components/processtools/metrics.yaml',\n 'toolkit/components/reportbrokensite/metrics.yaml',\n 'toolkit/components/resistfingerprinting/metrics.yaml',\n 'toolkit/components/translations/metrics.yaml',\n 'toolkit/mozapps/extensions/metrics.yaml',\n 'toolkit/mozapps/handling/metrics.yaml',\n 'toolkit/xre/metrics.yaml',\n 'xpcom/metrics.yaml']
[task 2024-03-06T10:14:28.074Z] E At index 5 diff: 'dom/metrics.yaml' != 'dom/security/metrics.yaml'
[task 2024-03-06T10:14:28.074Z] E Full diff:
[task 2024-03-06T10:14:28.074Z] E [
[task 2024-03-06T10:14:28.074Z] E 'browser/base/content/metrics.yaml',
[task 2024-03-06T10:14:28.074Z] E 'docshell/base/metrics.yaml',
[task 2024-03-06T10:14:28.074Z] E 'dom/base/use_counter_metrics.yaml',
[task 2024-03-06T10:14:28.074Z] E 'dom/media/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E 'dom/media/webrtc/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E - 'dom/security/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E 'dom/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E 'dom/performance/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E + 'dom/security/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E 'gfx/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E 'image/decoders/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E 'js/xpconnect/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E 'layout/base/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E 'mobile/android/actors/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E 'mobile/android/modules/geckoview/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E 'netwerk/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E 'netwerk/protocol/http/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E 'security/manager/ssl/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E 'toolkit/components/cookiebanners/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E 'toolkit/components/extensions/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E 'toolkit/components/formautofill/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E 'toolkit/components/glean/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E 'toolkit/components/passwordmgr/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E 'toolkit/components/pdfjs/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E 'toolkit/components/processtools/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E 'toolkit/components/reportbrokensite/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E 'toolkit/components/resistfingerprinting/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E 'toolkit/components/translations/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E 'toolkit/mozapps/extensions/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E 'toolkit/mozapps/handling/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E 'toolkit/xre/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E 'xpcom/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E ]
[task 2024-03-06T10:14:28.075Z] ============================== 1 failed in 0.05s ===============================
Comment 15•4 months ago
|
||
[task 2024-03-06T10:37:53.788Z] 10:37:53 INFO - TEST-PASS | dom/security/test/https-first/browser_httpsfirst.js | HTTPS-First disabled; Should not upgrade - true == true -
[task 2024-03-06T10:37:53.788Z] 10:37:53 INFO - Buffered messages finished
[task 2024-03-06T10:37:53.792Z] 10:37:53 INFO - TEST-UNEXPECTED-FAIL | dom/security/test/https-first/browser_httpsfirst.js | No telemetry should have been recorded yet - Got null, expected 5
[task 2024-03-06T10:37:53.792Z] 10:37:53 INFO - Stack trace:
[task 2024-03-06T10:37:53.792Z] 10:37:53 INFO - chrome://mochikit/content/browser-test.js:test_is:1620
[task 2024-03-06T10:37:53.792Z] 10:37:53 INFO - chrome://mochitests/content/browser/dom/security/test/https-first/browser_httpsfirst.js:null:45
[task 2024-03-06T10:37:53.792Z] 10:37:53 INFO - chrome://mochikit/content/browser-test.js:handleTask:1139
[task 2024-03-06T10:37:53.793Z] 10:37:53 INFO - chrome://mochikit/content/browser-test.js:_runTaskBasedTest:1211
[task 2024-03-06T10:37:53.793Z] 10:37:53 INFO - chrome://mochikit/content/browser-test.js:Tester_execTest:1353
[task 2024-03-06T10:37:53.793Z] 10:37:53 INFO - chrome://mochikit/content/browser-test.js:nextTest/<:1122
[task 2024-03-06T10:37:53.794Z] 10:37:53 INFO - chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:SimpleTest.waitForFocus/<:1058
Assignee | ||
Updated•4 months ago
|
Comment 16•4 months ago
|
||
Pushed by mjurgens@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/03e6d6ff8bde Add glean telemetry for HTTPS-First r=freddyb,simonf
Comment 17•4 months ago
|
||
bugherder |
Description
•