Closed Bug 1868380 Opened 7 months ago Closed 4 months ago

Add Telemetry for (schemeless) HTTPS-First

Categories

(Core :: DOM: Security, task, P2)

task

Tracking

()

RESOLVED FIXED
125 Branch
Tracking Status
firefox125 --- fixed

People

(Reporter: maltejur, Assigned: maltejur)

References

(Blocks 2 open bugs)

Details

(Whiteboard: [domsecurity-active])

Attachments

(2 files, 5 obsolete files)

As schemeless HTTPS-First (https as the default scheme for address bar inputs without a scheme, with a fallback to http) has recently landed in Nightly (Bug 1863281), before we proceed to enable it by default in release, we want to add telemetry for HTTPS-First. We want to measure the benefit this feature brings vs in how much load time overhead and annoyance it results in if a upgrade is not possible. More specifically, we want to answer these questions:

  1. Which percentage and amount of url bar inputs get marked as "schemeless" and thus get marked for a potential HTTPS-First upgrade?
  2. Which percentage of HTTPS-First upgrades succeed and which percentage gets downgraded again?
  3. If a upgrade isn't possible, which percentage of downgrades happen because the https request failed and which percentage happens because the extra http request fired after 3s got a response first?
  4. Of the downgrades that happen because the https request failed, what is the average time overhead / what is the time between the first load start and the downgrade?
Severity: -- → N/A
Priority: -- → P2
Whiteboard: [domsecurity-active]
Attached file data-review.txt (obsolete) —
Attachment #9368136 - Attachment is obsolete: true
Attached file data-review.txt (obsolete) —

Freddy, can you take a look at my data review draft? Open things from me:

a. I am unsure what to write for how we are going to analyze the data (10. and 11.). Should I just write that we are going to create a dashboard for internal team use to look at the data?
b. Should I add you to the list of people responsible for the collection (7.)?

Flags: needinfo?(fbraun)
Attached file data-review.txt (obsolete) —
Attachment #9368396 - Attachment is obsolete: true
Comment on attachment 9372782 [details]
data-review.txt

Looks great. Some minor comments inline.

>DATA REVIEW REQUEST
>
>(Context)
>
>HTTPS-First is a mechanism enabled in private browsing that upgrades all top level loads to HTTPS, with a mechanism to fall back to HTTP if the HTTPS site can't be loaded.
>
>Schemeless HTTPS-First means that HTTPS-First will be enabled for urlbar inputs that are being fixed up to URI and don't have a scheme, essentially making HTTPS the default scheme for the urlbar with a fallback to HTTP.
>

How about removing the first sentence and trimming away some jargon. Maybe like so? "We intend to upgrade loads from the address bar to HTTPS, when the explicit URL scheme is missing (i.e., "schemeless"). This upgrade happens with a fallback to HTTP, when a secure channel is not available. To better evaluate the performance and security impact of this new mechanism, we want to measure the number of affected requests as well as added overhead of our request upgrades"


>1. What questions will you answer with this data?
>
>- Which percentage and amount of url bar inputs get marked as "schemeless" and thus get marked for a potential HTTPS-First upgrade?
>- Which percentage of HTTPS-First upgrades succeed and which percentage gets downgraded again?
>- If a upgrade isn't possible, which percentage of downgrades happen because the extra http request fired after 3s got a response first?
>- If a upgrade isn't possible, what is the average time overhead caused by HTTPS-First?
>
>2. Why does Mozilla need to answer these questions? Are there benefits for users?
>   Do we need this information to address product or business requirements?
>
>Currently, schemeless HTTPS-First is enabled in Nightly and early beta. Before we ship it to release, we want to make sure it does not significantly worsen the page load times for most users on sites not supporting HTTP. In addition to that, we

s/HTTP/HTTPS/. It also looks like the last sentence is incomplete?

>
>3. What alternative methods did you consider to answer these questions?
>   Why were they not sufficient?
>
>I already did testing for all of these questions locally on some sites, but since HTTPS-First can behave differently on different sites and network conditions, my testing isn't an adequate answer to these questions.
>
>4. Can current instrumentation answer these questions?
>
>The first question can be answered by combining the existing urlbar telemetry and the new `httpsfirst.upgraded_schemeless` metric. The relevant urlbar telemetry can be found here: https://sql.telemetry.mozilla.org/queries/96161#237499
>

>5. List all proposed measurements and indicate the category of data collection for each
>   measurement, using the Firefox data collection categories found on the Mozilla wiki.
>
>Measurement Name | Measurement Description | Data Collection Category | Tracking Bug
>---------------- | ----------------------- | ------------------------ | ------------
>`httpsfirst.downgrade_time` | If a HTTPS-First upgrade isn't successful, measures the timespan between the navigation start and the downgrade. This is essentially the overhead caused by HTTPS-First if a site does not support HTTPS.  | Technical data | https://bugzilla.mozilla.org/show_bug.cgi?id=1868380
>`httpsfirst.downgraded` | How many of the HTTPS-First upgrades get downgraded again.  | Technical data | https://bugzilla.mozilla.org/show_bug.cgi?id=1868380
>`httpsfirst.downgraded_faster_http_request` | How many of the HTTPS-First upgrades get downgraded again because the HTTP request fired after 3s received a answer faster than the HTTPS request.  | Technical data | https://bugzilla.mozilla.org/show_bug.cgi?id=1868380
>`httpsfirst.upgraded` | Counts how often a load is marked to be upgraded to HTTPS because of HTTPS-First.  | Technical data | https://bugzilla.mozilla.org/show_bug.cgi?id=1868380
>`httpsfirst.upgraded_schemeless` | How many of the HTTPS-First upgrades are happening while the load is marked as schemeless.  | Technical data | https://bugzilla.mozilla.org/show_bug.cgi?id=1868380
>
>6. Please provide a link to the documentation for this data collection which
>   describes the ultimate data set in a public, complete, and accurate way.
>
>This collection is Glean so is documented [in the Glean Dictionary](https://dictionary.telemetry.mozilla.org).
>

^--- grammar.

>7. How long will this data be collected?
>
>This collection will be collected permanently.
>seceng-telemetry@mozilla.com and mjurgens@mozilla.com will be responsible for the permanent collections.
>

...collected permanently until we succeeded in shipping the feature for all of our users
I think we intend to remove it when we are confident in its robustness, don't we?

>8. What populations will you measure?
>
>All channels, countries, and locales. No filters.
>
>9. If this data collection is default on, what is the opt-out mechanism for users?
>
>These collections are Glean. The opt-out can be found in the product's preferences.

"are using glean"?

>
>10. Please provide a general description of how you will analyze this data.
>
>We are going to create an internal dashboard to evaluate the data. Depending on the results, we will either continue with shipping schemeless HTTPS-First, or add experiments to improve the behavior of schemeless HTTPS-First.
>
>11. Where do you intend to share the results of your analysis?
>
>This is primarily meant for internally evaluating the effectivenes of schemeless HTTPS-First. If we successfully ship schemeless HTTPS-First though, we may publish some of the data publicly, for example as part of a blog post.

typo: effectiveness.
"some of the data" -> "our aggregate findings"?

>
>12. Is there a third-party tool (i.e. not Glean or Telemetry) that you
>    are proposing to use for this data collection?
>
>No.
Attached file data-review.txt (obsolete) —

Thanks.

How about removing the first sentence and trimming away some jargon. Maybe like so? "We intend to upgrade loads from the address bar to HTTPS, when the explicit URL scheme is missing (i.e., "schemeless"). This upgrade happens with a fallback to HTTP, when a secure channel is not available. To better evaluate the performance and security impact of this new mechanism, we want to measure the number of affected requests as well as added overhead of our request upgrades"

Sounds good

It also looks like the last sentence is incomplete?

Oh right, I've removed that sentence now. Sorry about that.

^--- grammar.

"are using glean"?

These sentences are automatically generated by Glean, so I suppose they are supposed to be like that? :D

...collected permanently until we succeeded in shipping the feature for all of our users
I think we intend to remove it when we are confident in its robustness, don't we?

But we also want to keep this telemetry for shipping HTTPS-First itself, right? I've changed the sentence to "This collection will be collected permanently until we succeeded in shipping HTTPS-First for all of our users." for now.

"some of the data" -> "our aggregate findings"?

Yes, that does sound better.

Attachment #9372782 - Attachment is obsolete: true

Can we clarify in the context section that this is for HTTPS first including schemeless and clarify which measurements apply to what. Currently, it's not clear to me which of these will be triggered by schemeless upgrades, HTTPS first upgrade and HTTPS only upgrades.

Attached file data-review.txt (obsolete) —

Thanks Simon, I added a short paragraph to the context, explaining that a bit better.

Attachment #9375771 - Attachment is obsolete: true
Attachment #9376017 - Flags: data-review?(chutten)

Comment on attachment 9376017 [details]
data-review.txt

DATA COLLECTION REVIEW RESPONSE:

Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes.

Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection can be controlled through the product's preferences.

If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, seceng-telemetry@mozilla.com and mjurgens@mozilla.com will be responsible for the permanent collections.

Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 1, Technical.

Is the data collection request for default-on or default-off?

Default on for all channels.

Does the instrumentation include the addition of any new identifiers?

No.

Is the data collection covered by the existing Firefox privacy notice?

Yes.

Does the data collection use a third-party collection tool?

No.


Result: datareview+

Attachment #9376017 - Flags: data-review?(chutten) → data-review+
Attachment #9367963 - Attachment description: WIP: Bug 1868380 - Add glean telemetry for HTTPS-First r?freddyb! → Bug 1868380 - Add glean telemetry for HTTPS-First r?freddyb!,simonf!
Flags: needinfo?(fbraun)
Attached file data-review.txt

Re-requesting data-review as the measurements have changed (only section 5 is different).

Attachment #9376017 - Attachment is obsolete: true
Attachment #9388989 - Flags: data-review?(chutten)

Comment on attachment 9388989 [details]
data-review.txt

DATA COLLECTION REVIEW RESPONSE:

Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes.

Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection can be controlled through the product's preferences.

If the request is for permanent data collection, is there someone who will monitor the data over time?

seceng-telemetry@mozilla.com and mjurgens@mozilla.com will be responsible for the permanent collections.

Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 1, Technical.

Is the data collection request for default-on or default-off?

Default on for all channels.

Does the instrumentation include the addition of any new identifiers?

No.

Is the data collection covered by the existing Firefox privacy notice?

Yes.

Does the data collection use a third-party collection tool?

No.


Result: datareview+

Attachment #9388989 - Flags: data-review?(chutten) → data-review+
Pushed by mjurgens@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/8c0e44e4d9ae
Add glean telemetry for HTTPS-First r=freddyb,simonf

Backed out for causing py3 failures on test_yaml_indices.py

[task 2024-03-06T10:14:28.071Z] ============================= test session starts ==============================
[task 2024-03-06T10:14:28.071Z] platform linux -- Python 3.8.10, pytest-7.0.1, pluggy-1.4.0 -- /builds/worker/.mozbuild/srcdirs/gecko-8a5b87fe5d69/_virtualenvs/python-test/bin/python
[task 2024-03-06T10:14:28.071Z] rootdir: /builds/worker/checkouts/gecko, configfile: config/mozunit/mozunit/pytest.ini
[task 2024-03-06T10:14:28.074Z] plugins: mock-3.12.0
[task 2024-03-06T10:14:28.074Z] collecting ... collected 1 item
[task 2024-03-06T10:14:28.074Z] 
[task 2024-03-06T10:14:28.074Z] toolkit/components/glean/tests/pytest/test_yaml_indices.py::test_yamls_sorted TEST-UNEXPECTED-FAIL
[task 2024-03-06T10:14:28.074Z] 
[task 2024-03-06T10:14:28.074Z] =================================== FAILURES ===================================
[task 2024-03-06T10:14:28.074Z] ______________________________ test_yamls_sorted _______________________________
[task 2024-03-06T10:14:28.074Z] toolkit/components/glean/tests/pytest/test_yaml_indices.py:36: in test_yamls_sorted
[task 2024-03-06T10:14:28.074Z]     assert (
[task 2024-03-06T10:14:28.074Z] E   AssertionError: gecko_metrics must be be lexicographically sorted.
[task 2024-03-06T10:14:28.074Z] E   assert ['browser/base/content/metrics.yaml',\n 'docshell/base/metrics.yaml',\n 'dom/base/use_counter_metrics.yaml',\n 'dom/media/metrics.yaml',\n 'dom/media/webrtc/metrics.yaml',\n 'dom/metrics.yaml',\n 'dom/performance/metrics.yaml',\n 'dom/security/metrics.yaml',\n 'gfx/metrics.yaml',\n 'image/decoders/metrics.yaml',\n 'js/xpconnect/metrics.yaml',\n 'layout/base/metrics.yaml',\n 'mobile/android/actors/metrics.yaml',\n 'mobile/android/modules/geckoview/metrics.yaml',\n 'netwerk/metrics.yaml',\n 'netwerk/protocol/http/metrics.yaml',\n 'security/manager/ssl/metrics.yaml',\n 'toolkit/components/cookiebanners/metrics.yaml',\n 'toolkit/components/extensions/metrics.yaml',\n 'toolkit/components/formautofill/metrics.yaml',\n 'toolkit/components/glean/metrics.yaml',\n 'toolkit/components/passwordmgr/metrics.yaml',\n 'toolkit/components/pdfjs/metrics.yaml',\n 'toolkit/components/processtools/metrics.yaml',\n 'toolkit/components/reportbrokensite/metrics.yaml',\n 'toolkit/components/resistfingerprinting/metrics.yaml',\n 'toolkit/components/translations/metrics.yaml',\n 'toolkit/mozapps/extensions/metrics.yaml',\n 'toolkit/mozapps/handling/metrics.yaml',\n 'toolkit/xre/metrics.yaml',\n 'xpcom/metrics.yaml'] == ['browser/base/content/metrics.yaml',\n 'docshell/base/metrics.yaml',\n 'dom/base/use_counter_metrics.yaml',\n 'dom/media/metrics.yaml',\n 'dom/media/webrtc/metrics.yaml',\n 'dom/security/metrics.yaml',\n 'dom/metrics.yaml',\n 'dom/performance/metrics.yaml',\n 'gfx/metrics.yaml',\n 'image/decoders/metrics.yaml',\n 'js/xpconnect/metrics.yaml',\n 'layout/base/metrics.yaml',\n 'mobile/android/actors/metrics.yaml',\n 'mobile/android/modules/geckoview/metrics.yaml',\n 'netwerk/metrics.yaml',\n 'netwerk/protocol/http/metrics.yaml',\n 'security/manager/ssl/metrics.yaml',\n 'toolkit/components/cookiebanners/metrics.yaml',\n 'toolkit/components/extensions/metrics.yaml',\n 'toolkit/components/formautofill/metrics.yaml',\n 'toolkit/components/glean/metrics.yaml',\n 'toolkit/components/passwordmgr/metrics.yaml',\n 'toolkit/components/pdfjs/metrics.yaml',\n 'toolkit/components/processtools/metrics.yaml',\n 'toolkit/components/reportbrokensite/metrics.yaml',\n 'toolkit/components/resistfingerprinting/metrics.yaml',\n 'toolkit/components/translations/metrics.yaml',\n 'toolkit/mozapps/extensions/metrics.yaml',\n 'toolkit/mozapps/handling/metrics.yaml',\n 'toolkit/xre/metrics.yaml',\n 'xpcom/metrics.yaml']
[task 2024-03-06T10:14:28.074Z] E     At index 5 diff: 'dom/metrics.yaml' != 'dom/security/metrics.yaml'
[task 2024-03-06T10:14:28.074Z] E     Full diff:
[task 2024-03-06T10:14:28.074Z] E       [
[task 2024-03-06T10:14:28.074Z] E        'browser/base/content/metrics.yaml',
[task 2024-03-06T10:14:28.074Z] E        'docshell/base/metrics.yaml',
[task 2024-03-06T10:14:28.074Z] E        'dom/base/use_counter_metrics.yaml',
[task 2024-03-06T10:14:28.074Z] E        'dom/media/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E        'dom/media/webrtc/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E     -  'dom/security/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E        'dom/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E        'dom/performance/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E     +  'dom/security/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E        'gfx/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E        'image/decoders/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E        'js/xpconnect/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E        'layout/base/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E        'mobile/android/actors/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E        'mobile/android/modules/geckoview/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E        'netwerk/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E        'netwerk/protocol/http/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E        'security/manager/ssl/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E        'toolkit/components/cookiebanners/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E        'toolkit/components/extensions/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E        'toolkit/components/formautofill/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E        'toolkit/components/glean/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E        'toolkit/components/passwordmgr/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E        'toolkit/components/pdfjs/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E        'toolkit/components/processtools/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E        'toolkit/components/reportbrokensite/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E        'toolkit/components/resistfingerprinting/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E        'toolkit/components/translations/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E        'toolkit/mozapps/extensions/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E        'toolkit/mozapps/handling/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E        'toolkit/xre/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E        'xpcom/metrics.yaml',
[task 2024-03-06T10:14:28.075Z] E       ]
[task 2024-03-06T10:14:28.075Z] ============================== 1 failed in 0.05s ===============================
Flags: needinfo?(mjurgens)

Also failing: TEST-UNEXPECTED-FAIL | dom/security/test/https-first/browser_httpsfirst.js | No telemetry should have been recorded yet - Got null, expected 5

[task 2024-03-06T10:37:53.788Z] 10:37:53     INFO - TEST-PASS | dom/security/test/https-first/browser_httpsfirst.js | HTTPS-First disabled; Should not upgrade - true == true - 
[task 2024-03-06T10:37:53.788Z] 10:37:53     INFO - Buffered messages finished
[task 2024-03-06T10:37:53.792Z] 10:37:53     INFO - TEST-UNEXPECTED-FAIL | dom/security/test/https-first/browser_httpsfirst.js | No telemetry should have been recorded yet - Got null, expected 5
[task 2024-03-06T10:37:53.792Z] 10:37:53     INFO - Stack trace:
[task 2024-03-06T10:37:53.792Z] 10:37:53     INFO - chrome://mochikit/content/browser-test.js:test_is:1620
[task 2024-03-06T10:37:53.792Z] 10:37:53     INFO - chrome://mochitests/content/browser/dom/security/test/https-first/browser_httpsfirst.js:null:45
[task 2024-03-06T10:37:53.792Z] 10:37:53     INFO - chrome://mochikit/content/browser-test.js:handleTask:1139
[task 2024-03-06T10:37:53.793Z] 10:37:53     INFO - chrome://mochikit/content/browser-test.js:_runTaskBasedTest:1211
[task 2024-03-06T10:37:53.793Z] 10:37:53     INFO - chrome://mochikit/content/browser-test.js:Tester_execTest:1353
[task 2024-03-06T10:37:53.793Z] 10:37:53     INFO - chrome://mochikit/content/browser-test.js:nextTest/<:1122
[task 2024-03-06T10:37:53.794Z] 10:37:53     INFO - chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:SimpleTest.waitForFocus/<:1058
Flags: needinfo?(mjurgens)
Pushed by mjurgens@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/03e6d6ff8bde
Add glean telemetry for HTTPS-First r=freddyb,simonf
Status: ASSIGNED → RESOLVED
Closed: 4 months ago
Resolution: --- → FIXED
Target Milestone: --- → 125 Branch
Regressions: 1885893
Regressions: 1891185
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: