1813128 - Add CMP handling specific telemetry and update existing telemetry

Status: RESOLVED FIXED

(Core :: Privacy: Anti-Tracking, task, P2)

Description

[Emma Zühlcke [:emz]]

We want to answer the following questions requiring additional telemetry:

• How often do we handle a banner with a site specific rule, how often with a global rule?
• Distribution of CMP rules used. This does not require collecting domains so it should be privacy friendly
• How often do we fail to handle a banner with a global rule because there is no opt-out button present? E.g. OneTrust has variations of banners, only some include an opt-out button in the first view.
• Extend handle time telemetry to include whether its a global or site specific rule

Comment 1

[[e-mail]]

Additional Telemetry Considerations in Pre-Release channels:

• What domains are CMP rules executing on?
• What domains are CMP rules not executing on as expected? (Nice to have)
• Sample auto-click failure: Banner presence detected, opt-out selector detected, Action take, banner still present
• Sample cookie injection failure: Banner presence detected, site reloads with cookie injected in storage, banner still present

For the cookie injector example it's important not to unintentionally create a loop where the banner can never be dismissed so the browser just keeps trying to reload with the same cookie.

Comment 2

[Tim Huang[:timhuang]]

Attachment: data-request-Bug1813128.md

Comment 3

[[e-mail]]

Comment on attachment 9363467 [details]
data-request-Bug1813128.md

>!! Reminder: it is your responsibility to complete and check the correctness of
>!! this automatically-generated request skeleton before requesting Data
>!! Collection Review. See https://wiki.mozilla.org/Data_Collection for details.
>
>DATA REVIEW REQUEST
>1. What questions will you answer with this data?
>
>- How often a cookie banner is handled by site specific rule or a CMP rule?
>- The distribution of CMP rules used for handling cookie banners.
>- How often do we succeed, how often do we fail handling the banner with CMP rules.
>- Why did our mechanism fail to handle a banner with CMP rules?
>- How long does it take for handling a cookie banner with CMP rules.
>- The origin where CMP rules execute on for pre-release channels.
>- The origin where CMP rules fail to handle the banner for pre-release channels.
>
>2. Why does Mozilla need to answer these questions? Are there benefits for users?
>   Do we need this information to address product or business requirements?
>
>- This is vital telemetry to measure effectiveness and stability of our cookie banner blocker German rollout.
>
>3. What alternative methods did you consider to answer these questions?
>   Why were they not sufficient?
>
>- Internal foxfooding with manual reports gave us some limited initial data, telemetry at scale will give us additional insight.
>
>4. Can current instrumentation answer these questions?
>
>- No
>
>5. List all proposed measurements and indicate the category of data collection for each
>   measurement, using the Firefox data collection categories found on the Mozilla wiki.
>
>Measurement Name | Measurement Description | Data Collection Category | Tracking Bug
>---------------- | ----------------------- | ------------------------ | ------------
>`cookie_banners.cookie_injection_fail` | Counts how often the cookie banner is still shown even if we have injected cookies.  | interaction | https://bugzilla.mozilla.org/show_bug.cgi?id=1813128
>`cookie_banners.cookie_injection_fail_origin` | Records the origin of websites where injected cookies fail to handle the banner. We only collect this in pre-release channels.  | stored_content | https://bugzilla.mozilla.org/show_bug.cgi?id=1813128
>`cookie_banners_cmp.detected_cmp` | Counts how often a specific cmp has been detected by our cookie banner handling.  | interaction | https://bugzilla.mozilla.org/show_bug.cgi?id=1813128
>`cookie_banners_cmp.executed_origin` | Records the origin of websites where CMP rules execute on. We only collect this in pre-release channels.  | stored_content | https://bugzilla.mozilla.org/show_bug.cgi?id=1813128
>`cookie_banners_cmp.failed_origin` | Records the origin of websites where CMP rules fail to handle the banner. We only collect this in pre-release channels.  | stored_content | https://bugzilla.mozilla.org/show_bug.cgi?id=1813128
>`cookie_banners_cmp.handle_duration` | Counts how long it takes to handle cookie banners successfully using CMP rules from DOMContentLoaded until click.  | interaction | https://bugzilla.mozilla.org/show_bug.cgi?id=1813128
>`cookie_banners_cmp.handled_count` | The counter of successfully handling a cookie banner. | interaction | https://bugzilla.mozilla.org/show_bug.cgi?id=1813128
>`cookie_banners_cmp.ratio_handled_by_cmp_rule` | The proportion of cookie banners handled by a cmp rule. | interaction | https://bugzilla.mozilla.org/show_bug.cgi?id=1813128
>`cookie_banners_cmp.ratio_handled_by_site_rule` | The proportion of cookie banners handled by a per-site rule. | interaction | https://bugzilla.mozilla.org/show_bug.cgi?id=1813128
>`cookie_banners_cmp.result` | Given a matching CMP rule, how often do we handle or fail to handle cookie banners, labelled by reason. The 'success' and 'fail' counters count the total numbers independently of the reason counters. Counters are incremented after the content window has been destroyed.  | interaction | https://bugzilla.mozilla.org/show_bug.cgi?id=1813128
>
>6. Please provide a link to the documentation for this data collection which
>   describes the ultimate data set in a public, complete, and accurate way.
>
>This collection is Glean so is documented [in the Glean Dictionary](https://dictionary.telemetry.mozilla.org).
>
>7. How long will this data be collected?
>
>This collection has expiry '128'.
>
>8. What populations will you measure?
>
>All channels, countries, and locales. No filters.
>
>9. If this data collection is default on, what is the opt-out mechanism for users?
>
>These collections are Glean. The opt-out can be found in the product's preferences.
>
>10. Please provide a general description of how you will analyze this data.
>
>GLAM, STMO
>
>11. Where do you intend to share the results of your analysis?
>
>Internal documents, potentially a blog post
>
>12. Is there a third-party tool (i.e. not Glean or Telemetry) that you
>    are proposing to use for this data collection?
>
>No.

Comment 4

[[e-mail]]

Comment on attachment 9363467 [details]
data-request-Bug1813128.md

DATA COLLECTION REVIEW RESPONSE:
​
Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?
​
Yes.
​
Is there a control mechanism that allows the user to turn the data collection on and off?
​
Yes. This collection can be controlled through the product's preferences.
​
If the request is for permanent data collection, is there someone who will monitor the data over time?
​
No, data collection expires after 128 days.
​
Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?
​
Category 2, Interactions
Category 3, Stored Content
​
Is the data collection request for default-on or default-off?
​
Default on for all channels.
​
Does the instrumentation include the addition of any new identifiers?
​
No.
​
Is the data collection covered by the existing Firefox privacy notice?
​
Yes.
​
Does the data collection use a third-party collection tool?
​
No.
​

Result: datareview-

Collection of Cat 3 data requires Sensitive Data Review Process

Comment 5

[Tim Huang[:timhuang]]

Attachment: data-request-Bug1813128-1.md

Comment 6

[Tim Huang[:timhuang]]

Request another data review without collecting sensitive information. I will open another bug for collecting sensitive information.

Comment 7

[Tim Huang[:timhuang]]

Attachment: Bug 1813128 - Part 1: Populate the rule id to the clicking rule. r?pbz!

To collect the telemetry regarding CMPs, we need to know which CMP rule
is used for handling the cookie banner. So, we populate the rule id to
the click rule, the rule id contains the name of the CMP for global
rules. We will use the id to report telemetry.

Comment 8

[Tim Huang[:timhuang]]

Attachment: Bug 1813128 - Part 2: Implement CMP specific telemetry. r?pbz!

Depends on D193705

Comment 9

[Tim Huang[:timhuang]]

Attachment: Bug 1813128 - Part 3: Add tests to verify telemetry. r?pbz!

Depends on D193706

Comment 10

[[e-mail]]

Comment on attachment 9363778 [details]
data-request-Bug1813128-1.md

Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?
​
Yes.
​
Is there a control mechanism that allows the user to turn the data collection on and off?
​
Yes. This collection can be controlled through the product's preferences.
​
If the request is for permanent data collection, is there someone who will monitor the data over time?
​
No. This collection will expire in 128.
​
Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?
​
Category 2, Interaction.
​
Is the data collection request for default-on or default-off?
​
Default on for all channels.
​
Does the instrumentation include the addition of any new identifiers?
​
No.
​
Is the data collection covered by the existing Firefox privacy notice?
​
Yes.
​
Does the data collection use a third-party collection tool?
​
No.
​

Result: datareview+

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8cd9e92e3864
Part 1: Populate the rule id to the clicking rule. r=pbz
https://hg.mozilla.org/integration/autoland/rev/b512bfb2ab8f
Part 2: Implement CMP specific telemetry. r=pbz
https://hg.mozilla.org/integration/autoland/rev/87c8c2e8c54d
Part 3: Add tests to verify telemetry. r=pbz

Comment 12

[Natalia Csoregi [:nataliaCs]]

Backed out for causing cookie related leaks.

• backout: https://hg.mozilla.org/integration/autoland/rev/090e56d854375c3ce9a3fe22dec2ce6825d3549f
• push: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&selectedTaskRun=Cb1XO8IQQ6a7hLBNqj959w.0&revision=87c8c2e8c54d488d4c39c72b88eb09bba6f7faa2
• failure log: https://treeherder.mozilla.org/logviewer?job_id=437270353&repo=autoland&lineNumber=5927

[task 2023-11-22T13:50:00.371Z] 13:50:00     INFO - == BloatView: ALL (cumulative) LEAK AND BLOAT STATISTICS, default process 2673
[task 2023-11-22T13:50:00.371Z] 13:50:00     INFO - 
[task 2023-11-22T13:50:00.372Z] 13:50:00     INFO -      |<----------------Class--------------->|<-----Bytes------>|<----Objects---->|
[task 2023-11-22T13:50:00.372Z] 13:50:00     INFO -      |                                      | Per-Inst   Leaked|   Total      Rem|
[task 2023-11-22T13:50:00.373Z] 13:50:00     INFO -    0 |TOTAL                                 |       37   145472|21604192     4128|
[task 2023-11-22T13:50:00.373Z] 13:50:00     INFO -  228 |Cookie                                |      224    38528|     224      172|
[task 2023-11-22T13:50:00.376Z] 13:50:00     INFO - 1658 |nsClickRule                           |      104    43576|     419      419|
[task 2023-11-22T13:50:00.377Z] 13:50:00     INFO - 1680 |nsCookieBannerRule                    |       72    30168|     455      419|
[task 2023-11-22T13:50:00.377Z] 13:50:00     INFO - 1683 |nsCookieRule                          |       56     9632|     224      172|
[task 2023-11-22T13:50:00.378Z] 13:50:00     INFO - 1976 |nsStringBuffer                        |        8    23568|  900474     2946|
[task 2023-11-22T13:50:00.378Z] 13:50:00     INFO - 
[task 2023-11-22T13:50:00.379Z] 13:50:00     INFO - nsTraceRefcnt::DumpStatistics: 2121 entries
[task 2023-11-22T13:50:00.380Z] 13:50:00     INFO - TEST-INFO | leakcheck | default leaked 172 Cookie
[task 2023-11-22T13:50:00.380Z] 13:50:00     INFO - TEST-INFO | leakcheck | default leaked 419 nsClickRule
[task 2023-11-22T13:50:00.381Z] 13:50:00     INFO - TEST-INFO | leakcheck | default leaked 419 nsCookieBannerRule
[task 2023-11-22T13:50:00.382Z] 13:50:00     INFO - TEST-INFO | leakcheck | default leaked 172 nsCookieRule
[task 2023-11-22T13:50:00.382Z] 13:50:00     INFO - TEST-INFO | leakcheck | default leaked 2946 nsStringBuffer
[task 2023-11-22T13:50:00.383Z] 13:50:00     INFO - TEST-UNEXPECTED-FAIL | leakcheck | default 145472 bytes leaked (Cookie, nsClickRule, nsCookieBannerRule, nsCookieRule, nsStringBuffer)
[task 2023-11-22T13:50:00.383Z] 13:50:00     INFO - 

Also failing: TEST-UNEXPECTED-FAIL | toolkit/components/cookiebanners/test/browser/browser_cookieinjector_telemetry.js | Uncaught exception in test bound testCookieInjectionFailTelemetry - Waiting for metric to be ready. - timed out after 50 tries.

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6f63c50d10b5
Part 1: Populate the rule id to the clicking rule. r=pbz
https://hg.mozilla.org/integration/autoland/rev/ef551507170d
Part 2: Implement CMP specific telemetry. r=pbz
https://hg.mozilla.org/integration/autoland/rev/a1f726ef39fc
Part 3: Add tests to verify telemetry. r=pbz

Comment 14

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/6f63c50d10b5
https://hg.mozilla.org/mozilla-central/rev/ef551507170d
https://hg.mozilla.org/mozilla-central/rev/a1f726ef39fc

Comment 15

[Tim Huang[:timhuang]]

Attachment: Bug 1813128 - Part 1: Populate the rule id to the clicking rule.

To collect the telemetry regarding CMPs, we need to know which CMP rule
is used for handling the cookie banner. So, we populate the rule id to
the click rule, the rule id contains the name of the CMP for global
rules. We will use the id to report telemetry.

Original Revision: https://phabricator.services.mozilla.com/D193705

Comment 16

[Tim Huang[:timhuang]]

Attachment: Bug 1813128 - Part 2: Implement CMP specific telemetry.

Original Revision: https://phabricator.services.mozilla.com/D193706

Depends on D194511

Comment 17

[Tim Huang[:timhuang]]

Attachment: Bug 1813128 - Part 3: Add tests to verify telemetry.

Original Revision: https://phabricator.services.mozilla.com/D193707

Depends on D194512

Comment 18

[Phabricator Automation]

Uplift Approval Request

• Needs manual QE test: no
• Explanation of risk level: The patches don't change the behavior. It only adds Telemetry.
• Fix verified in Nightly: yes
• User impact if declined: No impace to users. But we won't be able to monitor the cookei banner blocker Germany rollout because of lacking Telemetry
• Code covered by automated testing: yes
• String changes made/needed: none
• Risk associated with taking this patch: low risk
• Is Android affected?: yes
• Steps to reproduce for manual QE testing: None

Comment 19

[Donal Meehan [:dmeehan]]

Comment on attachment 9365187 [details]
Bug 1813128 - Part 1: Populate the rule id to the clicking rule.

Approved for 120.0b3

Comment 20

[Donal Meehan [:dmeehan]]

Comment on attachment 9365188 [details]
Bug 1813128 - Part 2: Implement CMP specific telemetry.

Approved for 120.0b3

Comment 21

[Donal Meehan [:dmeehan]]

Comment on attachment 9365189 [details]
Bug 1813128 - Part 3: Add tests to verify telemetry.

Approved for 121.0b3

Comment 22

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/f73cca20062a
https://hg.mozilla.org/releases/mozilla-beta/rev/abd0ebea4ebe
https://hg.mozilla.org/releases/mozilla-beta/rev/63e4626951c0