Closed Bug 1272788 Opened 3 years ago Closed 3 years ago

Make Application Reputation/Safe Browsing Malware Filtered Probes opt-out


(Toolkit :: Telemetry, defect, P1)




Tracking Status
firefox49 --- fixed


(Reporter: javaun, Assigned: gfritzsche)


(Whiteboard: [measurement:client])


(1 file)

These are only on in opt-in Telemetry AIUI. This is how much malware we filter. In Fx 48 we are greatly improving our ability to filter Malware downloads. Currently we support 1 of 3 Google lists, Malware. We're adding filtering and warnings for the other two lists: Uncommon downloads (most turn out to be malware), and Potentially Unwanted, a category Google defines by their policy that mostly includes spyware/adware and other software that unwittingly changes user settings:

Moving from 1 list to 3 and should increase filtered malware 5-10x. We know this because we have probes in opt-in telemetry (i.e. pre-release) but we have no way to measure the improvement in release. Here is what our pre-release Telemetry shows.!cumulative=0&end_date=2016-04-10&keys=__none__!__none__!__none__&max_channel_version=nightly%252F48&measure=APPLICATION_REPUTATION_SERVER_VERDICT&min_channel_version=null&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2016-03-07&table=1&trim=1&use_submission_date=0

My proposal is to 
1. make these probes part of opt-out Telemetry for release users, **prior to 47 GA release**, so that we can have a baseline before/after the improved filtering to understand how much more malware we catch. 
2. At this time I don't require any more reporting than the out-of-the-box Telemetry histogram above. 

Our hypothesis is that there will be long-term trickle down effects and greater overall satisfaction with Firefox. They will be very difficult to attribute to increased filtering, since the gains may only happen when folks get new machines, reinstall, and don't become re-infected. 

Going forward, I'd like to watch the filtered malware trend to understand how much our users are encountering as a proxy for how dirty the web environment is, and could cause us to ratchet up warnings or present more UI.
Assignee: nobody → benjamin
Priority: -- → P1
APPLICATION_REPUTATION_SERVER_VERDICT... does this measure only the first time the full hash is downloaded? Is there also a probe for when the cached full hash is hit? 

If not, this probe may undercount. For instance, on a false positive (shavar match, full hash miss), we record a miss on the verdict probe. But a subsequent shavar hit that is a full hash hit will not be counted.

This effect would be worsened by how a shavar may correspond to multiple full hashes.
Flags: needinfo?(jmoradi)
(In reply to Chris H-C :chutten from comment #1)
> APPLICATION_REPUTATION_SERVER_VERDICT... does this measure only the first
> time the full hash is downloaded? Is there also a probe for when the cached
> full hash is hit? 

I think you're confusing the browsing protection and download protection/application reputation parts of Safe Browsing.

The server verdict that is measured here corresponds to Step 5 in
Flags: needinfo?(jmoradi)
Javaun asked me to weigh in on this from a policy perspective. Provided Benjamin signs off, I'm good with adding this to the opt-out bucket. This is measurement for security purposes, which has a compelling user benefit. And having a baseline prior to 47 make sense. If Benjamin has concern, I'm happy to discuss further. 

Will the probes be time limited?  It seems like you might not need these permanently.
Unassigning since this isn't my bug. I don't think we can do this *before* 47 because we've already shipped the last 46 and can't change the opt-in stats from a system addon.

This is not a problem for 47, although I do the final data review on the change itself. Unassigning the bug since it's not my engineering action.

Be aware that does not include opt-out datasets. You will need to query this data directly, either using a spark cluster or against longitudinal, to get useful numbers from the release population.
Assignee: benjamin → nobody
Merwin, does having these probes be temporary/permanent change our review? BDS alluded that it might change the level of effort. I don't currently foresee a need to keep the permanent but would like to see them for quite some time (6-12 months) to see if there is variability. We currently have poor insight here.

Benjamin: to be clear, I meant to do the work prior to 47 launch so it could launch in 47 -- not that we'd try to put it in release prior to 46. Sorry for the ambiguity.
Javaun, no, it does not change the review.  If we anticipate not needing the data in 12 months, then we should set the time limit. If this changes the level of effort significantly, then we can move forward without the time limit.
If the implementation is no different, I don't want to set a time limit on the data. I could see us adding more measures here, as the Firefox experience involves both malware blocked as well as how users interact with the warnings. The usable security protection is the ultimate goal: how do we protect users, display firm warnings when the danger is clear, and allow users to make informed choice when we're less certain

Bumping this one up. BDS has unassigned himself. To whom should this one be assigned?
Flags: needinfo?(thuelbert)
pulling this into the current sprint - assigning to Georg with a p2.
Assignee: nobody → gfritzsche
Priority: P1 → P2
Priority: P2 → P1
Whiteboard: [measurement:client]
Flags: needinfo?(thuelbert)
This is easy enough to do and can be tested using:

Moving forward i think building out or improving those measurements is best discussed with the owning team, which AFAIK is the Firefox Desktop Privacy and Security team of Panagiotis Astithas.
Comment on attachment 8756278 [details] [diff] [review]

Please make this expires_in_version: 52 per discussion with Javaun. data-r=me with that change.
Attachment #8756278 - Flags: review?(benjamin) → review+
Summary: Request to add Application Reputation/Safe Browsing Malware Filtered Probes to Opt-out Telemetry → Make Application Reputation/Safe Browsing Malware Filtered Probes opt-out
This is a trivial change client-side, so asking for review + data-review.
Attachment #8756278 - Flags: review?(benjamin)
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla49
You need to log in before you can comment on or make changes to this bug.