1773695 - Implement email tracking URL classifier feature

Status: RESOLVED FIXED

(Core :: Privacy: Anti-Tracking, task, P2)

Comment 1

[Tim Huang[:timhuang]]

Attachment: Bug 1773695 - Part 1: Add prefs for email tracking protection. r?dimi

Depends on D151047

Comment 2

[Tim Huang[:timhuang]]

Attachment: Bug 1773695 - Part 2: Add flags for the email tracking protections. r?dimi

Depends on D151048

Comment 3

[Tim Huang[:timhuang]]

Attachment: Bug 1773695 - Part 3: Implement email tracking url classifier feature. r?dimi

Depends on D151049

Comment 4

[Tim Huang[:timhuang]]

Attachment: Bug 1773695 - Part 4: Add email tracking flags to webrequest. r?mixedpuppy!

Depends on D151050

Comment 5

[Tim Huang[:timhuang]]

Attachment: Bug 1773695 - Part 5: Add a test for the email tracking protection. r?dimi

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/fb625ae08f73
Part 1: Add prefs for email tracking protection. r=dimi
https://hg.mozilla.org/integration/autoland/rev/f4770b2fe53d
Part 2: Add flags for the email tracking protections. r=dimi
https://hg.mozilla.org/integration/autoland/rev/8ef4d3597d92
Part 3: Implement email tracking url classifier feature. r=dimi
https://hg.mozilla.org/integration/autoland/rev/001c38822c53
Part 4: Add email tracking flags to webrequest. r=mixedpuppy
https://hg.mozilla.org/integration/autoland/rev/f402f1bd4caa
Part 5: Add a test for the email tracking protection. r=dimi

Comment 7

[Cristina Cozmuta (:CrissCozmuta)]

Backed out for causing Bp-hybrid bustages on UrlClassifierFeatureEmailTrackingProtection
Backout link
Push with failures
Link to failure log
Failure line :
/builds/worker/checkouts/gecko/netwerk/url-classifier/UrlClassifierFeatureEmailTrackingProtection.cpp:42:6: error: use of undeclared identifier 'nsIClassifiedChannel'

New failure line :
TEST-UNEXPECTED-FAIL | xpcshell.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_urlclassification.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | xpcshell.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_urlclassification.js | test_webrequest_url_classification_enum - [test_webrequest_url_classification_enum : 28] schema emailtracking is in IDL - false == true
Failure log

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6c7bd0531bc6
Part 1: Add prefs for email tracking protection. r=dimi
https://hg.mozilla.org/integration/autoland/rev/cd29b0766f86
Part 2: Add flags for the email tracking protections. r=dimi
https://hg.mozilla.org/integration/autoland/rev/1c2556d56a53
Part 3: Implement email tracking url classifier feature. r=dimi
https://hg.mozilla.org/integration/autoland/rev/db926abd402c
Part 4: Add email tracking flags to webrequest. r=mixedpuppy
https://hg.mozilla.org/integration/autoland/rev/f1d59396c406
Part 5: Add a test for the email tracking protection. r=dimi

Comment 9

[Iulian Moraru]

Backed out for causing condprof xpcshell failures on test_ext_webRequest_urlclassification.js.

Push with failures

Failure log

Backout link

[task 2022-07-15T15:51:30.437Z] 15:51:30     INFO -  TEST-START | xpcshell.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_urlclassification.js
[task 2022-07-15T15:51:30.744Z] 15:51:30  WARNING -  TEST-UNEXPECTED-FAIL | xpcshell.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_urlclassification.js | xpcshell return code: 0
[task 2022-07-15T15:51:30.744Z] 15:51:30     INFO -  TEST-INFO took 305ms
[task 2022-07-15T15:51:30.744Z] 15:51:30     INFO -  >>>>>>>
[task 2022-07-15T15:51:30.752Z] 15:51:30     INFO -  (xpcshell/head.js) | test MAIN run_test pending (1)
[task 2022-07-15T15:51:30.752Z] 15:51:30     INFO -  (xpcshell/head.js) | test run_next_test 0 pending (2)
[task 2022-07-15T15:51:30.752Z] 15:51:30     INFO -  (xpcshell/head.js) | test MAIN run_test finished (2)
[task 2022-07-15T15:51:30.752Z] 15:51:30     INFO -  running event loop
[task 2022-07-15T15:51:30.752Z] 15:51:30     INFO -  xpcshell.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_urlclassification.js | Starting setup check_remote
[task 2022-07-15T15:51:30.752Z] 15:51:30     INFO -  (xpcshell/head.js) | test check_remote pending (2)
[task 2022-07-15T15:51:30.752Z] 15:51:30     INFO -  TEST-PASS | xpcshell.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_urlclassification.js | check_remote - [check_remote : 1] useRemoteWebExtensions matches - false == false
[task 2022-07-15T15:51:30.752Z] 15:51:30     INFO -  TEST-PASS | xpcshell.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_urlclassification.js | check_remote - [check_remote : 1] testing from extension process - true == true
[task 2022-07-15T15:51:30.752Z] 15:51:30     INFO -  (xpcshell/head.js) | test run_next_test 0 finished (2)
[task 2022-07-15T15:51:30.752Z] 15:51:30     INFO -  (xpcshell/head.js) | test run_next_test 1 pending (2)
[task 2022-07-15T15:51:30.752Z] 15:51:30     INFO -  (xpcshell/head.js) | test check_remote finished (2)
[task 2022-07-15T15:51:30.752Z] 15:51:30     INFO -  xpcshell.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_urlclassification.js | Starting test_webrequest_url_classification_enum
[task 2022-07-15T15:51:30.752Z] 15:51:30     INFO -  (xpcshell/head.js) | test test_webrequest_url_classification_enum pending (2)
[task 2022-07-15T15:51:30.752Z] 15:51:30     INFO -  (xpcshell/head.js) | test run_next_test 1 finished (2)
[task 2022-07-15T15:51:30.752Z] 15:51:30     INFO -  TEST-PASS | xpcshell.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_urlclassification.js | test_webrequest_url_classification_enum - [test_webrequest_url_classification_enum : 16] UrlClassificationFlags: ["fingerprinting","fingerprinting_content","cryptomining","cryptomining_content","tracking","tracking_ad","tracking_analytics","tracking_social","tracking_content","any_basic_tracking","any_strict_tracking","any_social_tracking"] - true == true
[task 2022-07-15T15:51:30.752Z] 15:51:30     INFO -  TEST-PASS | xpcshell.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_urlclassification.js | test_webrequest_url_classification_enum - [test_webrequest_url_classification_enum : 28] schema fingerprinting is in IDL - true == true
[task 2022-07-15T15:51:30.752Z] 15:51:30     INFO -  TEST-PASS | xpcshell.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_urlclassification.js | test_webrequest_url_classification_enum - [test_webrequest_url_classification_enum : 28] schema fingerprinting_content is in IDL - true == true
[task 2022-07-15T15:51:30.752Z] 15:51:30     INFO -  TEST-PASS | xpcshell.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_urlclassification.js | test_webrequest_url_classification_enum - [test_webrequest_url_classification_enum : 28] schema cryptomining is in IDL - true == true
[task 2022-07-15T15:51:30.752Z] 15:51:30     INFO -  TEST-PASS | xpcshell.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_urlclassification.js | test_webrequest_url_classification_enum - [test_webrequest_url_classification_enum : 28] schema cryptomining_content is in IDL - true == true
[task 2022-07-15T15:51:30.752Z] 15:51:30     INFO -  TEST-PASS | xpcshell.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_urlclassification.js | test_webrequest_url_classification_enum - [test_webrequest_url_classification_enum : 28] schema tracking is in IDL - true == true
[task 2022-07-15T15:51:30.753Z] 15:51:30     INFO -  TEST-PASS | xpcshell.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_urlclassification.js | test_webrequest_url_classification_enum - [test_webrequest_url_classification_enum : 28] schema tracking_ad is in IDL - true == true
[task 2022-07-15T15:51:30.753Z] 15:51:30     INFO -  TEST-PASS | xpcshell.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_urlclassification.js | test_webrequest_url_classification_enum - [test_webrequest_url_classification_enum : 28] schema tracking_analytics is in IDL - true == true
[task 2022-07-15T15:51:30.753Z] 15:51:30     INFO -  TEST-PASS | xpcshell.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_urlclassification.js | test_webrequest_url_classification_enum - [test_webrequest_url_classification_enum : 28] schema tracking_social is in IDL - true == true
[task 2022-07-15T15:51:30.753Z] 15:51:30     INFO -  TEST-PASS | xpcshell.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_urlclassification.js | test_webrequest_url_classification_enum - [test_webrequest_url_classification_enum : 28] schema tracking_content is in IDL - true == true
[task 2022-07-15T15:51:30.753Z] 15:51:30  WARNING -  TEST-UNEXPECTED-FAIL | xpcshell.ini:toolkit/components/extensions/test/xpcshell/test_ext_webRequest_urlclassification.js | test_webrequest_url_classification_enum - [test_webrequest_url_classification_enum : 28] schema emailtracking is in IDL - false == true
[task 2022-07-15T15:51:30.753Z] 15:51:30     INFO -  /builds/worker/workspace/build/tests/xpcshell/tests/toolkit/components/extensions/test/xpcshell/test_ext_webRequest_urlclassification.js:test_webrequest_url_classification_enum:28
[task 2022-07-15T15:51:30.753Z] 15:51:30     INFO -  /builds/worker/workspace/build/tests/xpcshell/head.js:_do_main:238
[task 2022-07-15T15:51:30.753Z] 15:51:30     INFO -  /builds/worker/workspace/build/tests/xpcshell/head.js:_execute_test:595
[task 2022-07-15T15:51:30.753Z] 15:51:30     INFO -  -e:null:1
[task 2022-07-15T15:51:30.753Z] 15:51:30     INFO -  exiting test
[task 2022-07-15T15:51:30.753Z] 15:51:30     INFO -  Unexpected exception NS_ERROR_ABORT:
[task 2022-07-15T15:51:30.753Z] 15:51:30     INFO -  _abort_failed_test@/builds/worker/workspace/build/tests/xpcshell/head.js:866:20
[task 2022-07-15T15:51:30.753Z] 15:51:30     INFO -  do_report_result@/builds/worker/workspace/build/tests/xpcshell/head.js:967:5
[task 2022-07-15T15:51:30.753Z] 15:51:30     INFO -  Assert<@/builds/worker/workspace/build/tests/xpcshell/head.js:73:21
[task 2022-07-15T15:51:30.753Z] 15:51:30     INFO -  proto.report@resource://testing-common/Assert.jsm:228:10
[task 2022-07-15T15:51:30.753Z] 15:51:30     INFO -  proto.ok@resource://testing-common/Assert.jsm:254:10
[task 2022-07-15T15:51:30.753Z] 15:51:30     INFO -  test_webrequest_url_classification_enum@/builds/worker/workspace/build/tests/xpcshell/tests/toolkit/components/extensions/test/xpcshell/test_ext_webRequest_urlclassification.js:28:9
[task 2022-07-15T15:51:30.753Z] 15:51:30     INFO -  _do_main@/builds/worker/workspace/build/tests/xpcshell/head.js:238:6
[task 2022-07-15T15:51:30.753Z] 15:51:30     INFO -  _execute_test@/builds/worker/workspace/build/tests/xpcshell/head.js:595:5
[task 2022-07-15T15:51:30.753Z] 15:51:30     INFO -  @-e:1:1
[task 2022-07-15T15:51:30.753Z] 15:51:30     INFO -  exiting test
[task 2022-07-15T15:51:30.753Z] 15:51:30     INFO -  <<<<<<<
[task 2022-07-15T15:51:32.084Z] 15:51:32     INFO -  INFO | Result summary:
[task 2022-07-15T15:51:32.084Z] 15:51:32     INFO -  INFO | Passed: 35
[task 2022-07-15T15:51:32.086Z] 15:51:32  WARNING -  INFO | Failed: 1
[task 2022-07-15T15:51:32.086Z] 15:51:32  WARNING -  One or more unittests failed.
[task 2022-07-15T15:51:32.086Z] 15:51:32     INFO -  INFO | Todo: 0
[task 2022-07-15T15:51:32.086Z] 15:51:32     INFO -  INFO | Retried: 1
[task 2022-07-15T15:51:32.086Z] 15:51:32     INFO -  SUITE-END | took 415s
[task 2022-07-15T15:51:32.086Z] 15:51:32     INFO -  Node moz-http2 server shutting down ...
[task 2022-07-15T15:51:32.089Z] 15:51:32     INFO -  http3Server server shutting down ...
[task 2022-07-15T15:51:32.270Z] 15:51:32    ERROR - Return code: 1
[task 2022-07-15T15:51:32.271Z] 15:51:32     INFO - TinderboxPrint: xpcshell-xpcshell<br/>35/<em class="testfail">1</em>/0
[task 2022-07-15T15:51:32.272Z] 15:51:32  WARNING - # TBPL FAILURE #
[task 2022-07-15T15:51:32.272Z] 15:51:32  WARNING - setting return code to 2
[task 2022-07-15T15:51:32.272Z] 15:51:32  WARNING - The xpcshell suite: xpcshell ran with return status: FAILURE

Comment 10

[Tim Huang[:timhuang]]

From the log, it looks to me that the web_request schema doesn't contain the emailtracking and emailtracking_content when running the conditioned profile test. I cannot reproduce the same failure in my local build with the regular xpcshell test. The schema in my local test has both enums.

It's weird because the difference between the regular test and the conditioned profile test is the profile used for the test. Does this mean that the extension schema is located in the profile instead of the firefox object, or it's some sort of caching mechanism in the profile?

Shane, do you know what can make this test fail in the conditioned profile test? Thanks.

Comment 11

[Tim Huang[:timhuang]]

I can confirm that it's the StartupCache that makes the test fail. I can fix the issue by removing the webext.sc.lz4(StartupCache file for extensions). I also found out that one test has done the same thing to avoid test failure in conditioned profile tests.

I will add another patch to fix the test.

Comment 12

[Tim Huang[:timhuang]]

Attachment: Bug 1773695 - Part 6: Removing the startup cache file in the beginning of the test test_ext_webRequest_urlclassification.js. r?mixedpuppy!

The patch adds a code to remove the startup cache file "webext.sc.lz4"
before running the tests in test_ext_webRequest_urlclassification.js.
This avoids the test to use an old web_request schema that is cached by
the StartupCache.

Depends on D151051

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2022fb62ccf5
Part 1: Add prefs for email tracking protection. r=dimi
https://hg.mozilla.org/integration/autoland/rev/1324bc495159
Part 2: Add flags for the email tracking protections. r=dimi
https://hg.mozilla.org/integration/autoland/rev/bd1c7581397b
Part 3: Implement email tracking url classifier feature. r=dimi
https://hg.mozilla.org/integration/autoland/rev/cb8f4b8ff31d
Part 4: Add email tracking flags to webrequest. r=mixedpuppy
https://hg.mozilla.org/integration/autoland/rev/68f89e27d331
Part 5: Add a test for the email tracking protection. r=dimi
https://hg.mozilla.org/integration/autoland/rev/d41d9a65cf6f
Part 6: Removing the startup cache file in the beginning of the test test_ext_webRequest_urlclassification.js. r=mixedpuppy

Comment 14

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/2022fb62ccf5
https://hg.mozilla.org/mozilla-central/rev/1324bc495159
https://hg.mozilla.org/mozilla-central/rev/bd1c7581397b
https://hg.mozilla.org/mozilla-central/rev/cb8f4b8ff31d
https://hg.mozilla.org/mozilla-central/rev/68f89e27d331
https://hg.mozilla.org/mozilla-central/rev/d41d9a65cf6f

Comment 15

[Richard Bloor]

Given their structure, I assume that emailtracking and emailtracking_content are considered separate from the tracking group? i.e. they should document on MDN in a separate bullet, similar to:

• emailtracking and emailtracking_content: indicates the request is involved in tracking emails. emailtracking is any generic email tracking request. The content suffixes identify the type of tracker.

Are emailtracking and emailtracking_content included in any_basic_tracking or any_strict_tracking?

Comment 16

[Tim Huang[:timhuang]]

No they are not in these two groups.

Comment 17

[Richard Bloor]

MDN content update in Email tracker classification in webRequest events #36401, BCD contem in webrequest additional tracking types #24994.