Closed Bug 1547140 Opened 5 years ago Closed 5 years ago

expose HttpTrafficAnalyzer information in webrequest

Categories

(WebExtensions :: Request Handling, enhancement, P1)

Product:
WebExtensions
Component:
Request Handling
Type:
enhancement

Tracking

(firefox70 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla70
Tracking Flags:
Tracking Status
firefox70 --- fixed

People

(Reporter: mixedpuppy, Assigned: mixedpuppy)

References

Details

Attachments

(1 file, 1 obsolete file)

Bug 1547140 add classification data to webRequest API
47 bytes, text/x-phabricator-request
Details | Review
Depends on: 1547141
Type: defect → enhancement
Priority: -- → P2

expose classification flags to webrequest and proxy events.

https://searchfox.org/mozilla-central/source/netwerk/protocol/http/nsIHttpChannel.idl#510

Priority: P2 → P1
Assignee: nobody → mixedpuppy

The patch has a rough proposed schema for the classification api in webrequest (and proxy) api. You can look at it for a more detailed view of the api addition. I might remove the "optional" in schema so that one does not need to check for its existence as code below illustrates. Here is a rough sample of how one might use it:

browser.webRequest.onBeforeRequest.addListener(details => {
  if (details.classification.firstParty.includes("tracking_ad")) { ... do something ... }
  if (details.classification.thirdParty.includes("tracking_ad")) { ... do something ... }
}, ....);

In nsIHttpChannel.idl there is a bit more going on, such is isTrackingResource and isThirdPartyTrackingResource. I don't feel those are necessary as that can be checked via "details.classification.firstParty.length > 0".

If we want a higher level indication of any tracking classification, we could leave "details.classification" undefined if there is no classification.

Flags: needinfo?(honzab.moz)
Flags: needinfo?(dd.mozilla)

I honestly don't have an opinion on this; looks good to me generally, tho. I think better to ask is either :baku or ehsan.

Flags: needinfo?(honzab.moz)

ni? for comment 3

Flags: needinfo?(amarchesini)

In nsIHttpChannel.idl there is a bit more going on, such is isTrackingResource and isThirdPartyTrackingResource. I don't feel those are necessary as that can be checked via "details.classification.firstParty.length > 0".

Right! We don't need to expose IsTrackingResource()/IsThirdPartyTrackingResource() if we expose classification flags. These 2 methods are helpers to know what must be consider a tracker, based on the current anti-tracking pref status. This doesn't need to be exposed to web-extensions.

Related to classification flags: we are planning to add more flags. What's the right way to keep web-extension code and nsIHttpChannel.classificationFlags in sync? Should I simply CC you to those bugs? The latest one is bug 1560040.

Flags: needinfo?(amarchesini) → needinfo?(mixedpuppy)

If these need to be kept in sync I'd recommend a unit test that ensures it.

I also do not have an opinion about this.

Flags: needinfo?(dd.mozilla)

(In reply to Andrea Marchesini [:baku] from comment #6)

Related to classification flags: we are planning to add more flags. What's the right way to keep web-extension code and nsIHttpChannel.classificationFlags in sync? Should I simply CC you to those bugs? The latest one is bug 1560040.

I'd just add them where apropriate and r? the right people. See the latest patch. Any addition would require changes ChannelWrapper.webidl/cpp, webrequest_json and the test file. We can probably simplify that somehow later.

Flags: needinfo?(mixedpuppy)
Attachment #9078898 - Attachment is obsolete: true
Pushed by scaraveo@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/2bfb73091f21
add classification data to webRequest API r=zombie,kmag,Fallen

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&revision=2bfb73091f2137235ed130f2227cd95bf0ecb9e2&selectedJob=261498003

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=261498003&repo=autoland&lineNumber=2793

Backout link: https://hg.mozilla.org/integration/autoland/rev/34b4421d3e0d66c77f629d1b0ac1df0224ad3a07

[task 2019-08-14T02:36:12.136Z] 02:36:12 INFO - TEST-SKIP | xpcshell.ini:toolkit/components/extensions/test/xpcshell/test_ext_captivePortal.js | took 1ms
[task 2019-08-14T02:36:12.136Z] 02:36:12 INFO - Retrying tests that failed when run in parallel.
[task 2019-08-14T02:36:12.144Z] 02:36:12 INFO - TEST-START | xpcshell.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_urlclassification.js
[task 2019-08-14T02:36:13.049Z] 02:36:13 WARNING - TEST-UNEXPECTED-FAIL | xpcshell.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_urlclassification.js | xpcshell return code: 0
[task 2019-08-14T02:36:13.049Z] 02:36:13 INFO - TEST-INFO took 904ms
[task 2019-08-14T02:36:13.050Z] 02:36:13 INFO - >>>>>>>
[task 2019-08-14T02:36:13.051Z] 02:36:13 INFO - PID 28704 | [28704, Main Thread] WARNING: Couldn't get the user appdata directory. Crash events may not be produced.: file /builds/worker/workspace/build/src/toolkit/crashreporter/nsExceptionHandler.cpp, line 2632
[task 2019-08-14T02:36:13.051Z] 02:36:13 INFO - PID 28704 | [28704, Main Thread] WARNING: NS_ENSURE_SUCCESS(rv, kKnownEsrVersion) failed with result 0x80004002: file /builds/worker/workspace/build/src/toolkit/components/resistfingerprinting/nsRFPService.cpp, line 662
[task 2019-08-14T02:36:13.052Z] 02:36:13 INFO - PID 28704 | [28704, Main Thread] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80520012: file /builds/worker/workspace/build/src/extensions/permissions/nsPermissionManager.cpp, line 2901
[task 2019-08-14T02:36:13.053Z] 02:36:13 INFO - (xpcshell/head.js) | test MAIN run_test pending (1)
[task 2019-08-14T02:36:13.053Z] 02:36:13 INFO - (xpcshell/head.js) | test run_next_test 0 pending (2)
[task 2019-08-14T02:36:13.054Z] 02:36:13 INFO - (xpcshell/head.js) | test MAIN run_test finished (2)
[task 2019-08-14T02:36:13.054Z] 02:36:13 INFO - running event loop
[task 2019-08-14T02:36:13.055Z] 02:36:13 INFO - xpcshell.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_urlclassification.js | Starting check_remote
[task 2019-08-14T02:36:13.055Z] 02:36:13 INFO - (xpcshell/head.js) | test check_remote pending (2)
[task 2019-08-14T02:36:13.056Z] 02:36:13 INFO - TEST-PASS | xpcshell.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_urlclassification.js | check_remote

Flags: needinfo?(mixedpuppy)

rebased and new flags added.

Flags: needinfo?(mixedpuppy)
Pushed by scaraveo@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/59db91645f2e
add classification data to webRequest API r=zombie,kmag,Fallen

https://hg.mozilla.org/mozilla-central/rev/59db91645f2e

Status: NEW → RESOLVED
Closed: 5 years ago
status-firefox70: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla70

Hello,

Will this fix require manual validation? If yes, please provide some steps to reproduce in order to correctly test it and also, please set the "qe-verify+" flag. Otherwise, could the "qe-verify-" flag be added? Thanks!

Flags: needinfo?(mixedpuppy)

This has tests

Flags: needinfo?(mixedpuppy) → qe-verify-

Looks like we never documented this. Who could help describe our current url classification flags?

Flags: needinfo?(mixedpuppy)
Flags: needinfo?(amarchesini)
Keywords: dev-doc-needed

Since it's a privileged api we shouldn't document on mdn, but we should get that documentation pulled together, perhaps on this bug at first.

With this bug, webRequest listeners receive an extra "urlClassification" attribute in the details object. This attribute is an object composed of "firstParty" and "thirdParty" properties, each one of them, arrays of classification flags. The possible flags are here:

https://searchfox.org/mozilla-central/rev/6866d3a650c826f1cabd123663e84b95ee743701/toolkit/components/extensions/schemas/web_request.json#354-356

The following is a description of those flags:

  • fingerprinting or fingerprinting_content: the request has been marked as a fingerprinting resource. "fingerprinting_content" is used when the resource is loaded from an origin that has been found to fingerprint but is not considered to participate in tracking, such as a payment provider.
  • cryptomining or cryptomining_content: similar to the previous category but for cryptomining resources.
  • tracking, tracking_ad, tracking_analytics, tracking_social, tracking_content: for tracking resources we have these categories. "tracking" is the generic one. "ad", "analytics", "social", "content" are used to identify the type of tracker
  • socialtracking, socialtracking_facebook, socialtracking_linkedin, socialtracking_twitter: similar to the previous categories but more related to social tracking. The major social networks have their own flag.
  • any_basic_tracking: this is a "meta" flag that combines any tracking and fingerprinting flag, excluding tracking_content and fingerprinting_content.
  • any_strict_tracking: this is another "meta" flag that combines any tracking and fingerprinting flag including tracking_content and fingerprinting_content
  • any_social_tracking: yet another "meta" flag that combines any socialtracking flag.

The "meta" flags are used for tracking-protection and ETP.

Flags: needinfo?(amarchesini)
Blocks: 1589494
Flags: needinfo?(mixedpuppy)

documented in bug 1589494

Keywords: dev-doc-needed
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: