1647825 - Allow users to opt-in to submitting URLs that are seeing XFO/CSP errors via telemetry

Status: RESOLVED FIXED

(Firefox :: Security, task, P2)

Description

[Johann Hofmann [:johannh]]

We are still trying to figure out whether the large volume of XFO errors users are seeing is expected or (at least partially) comes from a bug. To help with that, we should give users the opportunity to opt into sending us telemetry events containing the URL of the failed page and optionally some other details like security preferences.

This should probably look and behave very similar to the checkbox we already show on certificate errors when the security.ssl.errorReporting.enabled pref is set to true (it's currently false).

The one exception is that for SSL error reporting we use a custom back-end AFAIK while for this we should probably use Telemetry Events.

Comment 1

[Tim Huang[:timhuang]]

Attachment: Bug 1647825 - Part 1: Add the UI for enable reporting XFO and CSP:frame-ancestors error. r?nhnt11!,ckerschb

This patch adds the UI for allowing users to enable reporting XFO error.
The reporting UI will be displayed in the error page if the error is a
XFO error.

Comment 2

[Tim Huang[:timhuang]]

Attachment: Bug 1647825 - Part 2: Report the XFO and CSP: frame-ancestors error through the telemetry event. r?ckerschb,chutten!,nhnt11!

After user ticks the checkbox of allowing error reporting, we will
report the error through the telemetry event. The event includes the
error type, XFO policy, CSP policy, the frame uri and the top-level uri.

Depends on D82331

Comment 3

[Tim Huang[:timhuang]]

Attachment: Bug 1647825 - Part 3: Add a test for the blocking reporting. r?ckerschb

Depends on D82332

Comment 4

[Tim Huang[:timhuang]]

Attachment: bug-1647825-data-review.txt

Comment 5

[Chris H-C :chutten]

Comment on attachment 9161557 [details]
bug-1647825-data-review.txt

DATA COLLECTION REVIEW RESPONSE:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes.
    Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, Tim Huang is responsible.

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 4, Highly Sensitive Data.

    Is the data collection request for default-on or default-off?

Default on for all channels.

    Does the instrumentation include the addition of any new identifiers?

No.

    Is the data collection covered by the existing Firefox privacy notice?

No. This is Category 4 data collected default-on.

    Does there need to be a check-in in the future to determine whether to renew the data?

No. This collection is permanent.

---
Result: datareview-, for default-on collection of Category 4 Data

ni?agray from Trust about how this need might be met, and under what circumstances we might permit this sort of collection.

ni?timhuang, I thought originally we discussed a default-off collection that users could opt-in to.

Comment 6

[Alicia Gray]

Keeping my NI open, but would like to understand from timhuang why this would be default collection rather than an opt-in model. If it is an opt-in model, would also like to know if the initial opt-in would then allow default collection post-initial opt-in on or if it would be an opt-in each time.

Comment 7

[Tim Huang[:timhuang]]

Sorry, I didn't explain it clearly in the data review request.

No, we are not making this opt-out, this will be opt-in. We provide a UI in the error page to allow users to opt-in the collection. I think I put a wrong release_channel_collection attribute in the Events.yaml file, it supposes to be 'opt-in'. Because I thought it requires an extra UI to make people opt-in the collection if I put 'opt-in' for this field rather than just use the UI we provide in the error page.

Chutten, does this make sense, and do I need to submit another data review request?

Comment 8

[Chris H-C :chutten]

No, that's the correct value for release_channel_collection if you wish to collect on the release channel (for historical reasons. Sorry for any confusion).

The part I think confused me was the data review talking about how users can opt out of it. I think Alicia makes a good point about whether opting in is a persistent choice or whether users opt in on every submission. I think your form suggests it's the first of those two options?

Perhaps there's a user flow explanation documented somewhere? If not, could you sketch out the flows that users can follow to submit (or choose not to submit) this collection?

Comment 9

[Tim Huang[:timhuang]]

Sure,

The reporting is disabled by default. When users hit an XFO or a CSP error page, there is a checkbox UI to allow user opt-in the reporting. Once users check the checkbox, we will start to report. So, we will report every time users hit an XFO or CSP error page. And users can opt-out the reporting by unchecking the checkbox. After that, we will no longer report the error.

Comment 10

[Chris H-C :chutten]

Pass to Alicia -- It's opt-in as described in comment 9.

Comment 11

[Alicia Gray]

Tim,

Can you share the dialog box/checkbox UI so I can see the language?
How does the user get to the opt-out control after they've opted in? Is the UI checkbox surfaced each time there is an error so that they can uncheck again?

Comment 12

[Tim Huang[:timhuang]]

Attachment: Screenshot 2020-07-08 at 10.15.59.png

Hi Alicia,

Here is the message will be shown to users. And yes, you are right about the opt-out control. This checkbox will always be shown on the XFO and CSP error pages, user can uncheck the checkbox to opt-out the reporting.

Comment 13

[Alicia Gray]

Thanks, Tim.

:chutten - opt-in error reporting is in line with policy so approved from Trust. Also noting that crash report errors are covered under the Firefox Privacy Notice under the Crash Reports disclosure section. This proposed model comports with our Notice. Please let me know if you need anything else.

Comment 14

[Chris H-C :chutten]

Comment on attachment 9161557 [details]
bug-1647825-data-review.txt

That's all from me on the data review front. Opt-in error reporting of Cat4 data (a la crash reporting) is all good from DAta Stewardship's POV.

Now let me switch to my Firefox Telemetry hat to see about the mechanism by which you're reporting these URLs. (I'll comment on the code review)

Comment 15

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla-services/mozilla-pipeline-schemas/pull/570

Comment 16

[Mark Reid [:mreid]]

This is now unblocked on the data pipeline side by bug 1654558.

Comment 17

[Mark Reid [:mreid]]

To clarify: bug 1654558 unblocks the client implementation herein, and with the existing data-review+ my understanding is that it is now safe to land.

The PR to mozilla-pipeline-schemas will continue to be blocked by bug 1654558 but does not block landing of the client code. Until that PR is merged and deployed, however, the collected data will not be available for data analysis.

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7481bdcd760d
Part 1: Add the UI for enable reporting XFO and CSP:frame-ancestors error. r=ckerschb,nhnt11
https://hg.mozilla.org/integration/autoland/rev/402c5e37eb04
Part 2: Report the XFO and CSP: frame-ancestors error through the telemetry event. r=ckerschb,chutten,nhnt11
https://hg.mozilla.org/integration/autoland/rev/994d05162929
Part 3: Add a test for the blocking reporting. r=ckerschb,nhnt11

Comment 19

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/7481bdcd760d
https://hg.mozilla.org/mozilla-central/rev/402c5e37eb04
https://hg.mozilla.org/mozilla-central/rev/994d05162929