Telemetry probe for mixed content blocker prefs

RESOLVED FIXED in Firefox 43

Status

()

defect
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: tanvi, Assigned: kmckinley)

Tracking

(Blocks 1 bug)

unspecified
mozilla43
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox43 fixed)

Details

Attachments

(2 attachments, 2 obsolete attachments)

Reporter

Description

4 years ago
There are two about:config prefs for mixed content blocker:

security.mixed_content.block_active_content -> set to true by default
security.mixed_content.block_display_content -> set to false by default

I'd like to know what percentage of users have changed the default pref.

I propose we write a telemetry probe for each one of them and report on the values (ex: 0 if the value is false and 1 if the value is true).

Telemetry probes I've looked at in the past usually report on a per page basis.  I'd like to report this on a per user basis, so I'm not sure exactly how this will work.  Is there a place in the code where you add probes that should be reported once a day?  I'm sure there are other about:config prefs that have probes and we could look at those as examples.
I pinged gfritzsche and asked him for some information relating to creating a probe that can report back on a daily basis and he mentioned that we don't really have that capability in the new telemetry. We can push the values either on startup or whenever they change.

The draw back with the startup approach is that we won't get values if a user changes the pref at runtime but We'll be able to see the new value whenever they restart firefox either way.
Flags: needinfo?(tanvi)
Reporter

Comment 2

4 years ago
(In reply to Kamil Jozwiak [:kjozwiak] from comment #1)
> I pinged gfritzsche and asked him for some information relating to creating
> a probe that can report back on a daily basis and he mentioned that we don't
> really have that capability in the new telemetry. We can push the values
> either on startup or whenever they change.

This is an either/or or an and?  We can push the values at both startup and when they change.  Or we have to choose one or the other?

> 
> The draw back with the startup approach is that we won't get values if a
> user changes the pref at runtime but We'll be able to see the new value
> whenever they restart firefox either way.

What I'd like to know is what percentage of users have a pref set to true and what percentage have a pref set to false.  If we just record at startup time, that will give us a percentage.  We will lose data about users who frequently toggle the pref.  But that is probably a very minor minor percentage.  Most users who would even go to about:config and change the pref, will change it once and leave it at that.  So we should just return the values at startup.
Flags: needinfo?(tanvi)
Assignee: nobody → kmckinley
Attachment #8640777 - Flags: review?(tanvi)
Status: NEW → ASSIGNED
Reporter

Comment 4

4 years ago
Comment on attachment 8640777 [details] [diff] [review]
Telemetry probe for mixed content blocker prefs


+  "MIXED_CONTENT_BLOCK_ACTIVE_CONTENT_PREF": {
+    "expires_in_version": "never",
+    "kind": "boolean",
+    "description": "Whether active content is blocked on mixed-content pages. (default: true)"
+  },
Change description to:
Whether mixed active content is blocked on https pages. (default: true)

+  "MIXED_CONTENT_BLOCK_DISPLAY_CONTENT_PREF": {
+    "expires_in_version": "never",
+    "kind": "boolean",
+    "description": "Whether to block showing any content on mixed-content pages. (default:false)"
+  },

Change description:
Whether mixed active content is blocked on https pages. (default: false)


What does the below part do exactly:
+    Telemetry::Accumulate(mozilla::Telemetry::MIXED_CONTENT_BLOCK_ACTIVE_CONTENT_PREF,
+            Preferences::GetBool("security.mixed_content.block_active_content", true));
If the pref is set to true, Preferences:GetBool returns true and we report true.  Otherwise we report false.  Correct?

+    Telemetry::Accumulate(mozilla::Telemetry::MIXED_CONTENT_BLOCK_DISPLAY_CONTENT_PREF,
+            Preferences::GetBool("security.mixed_content.block_display_content", false));
If the pref is set to false, Preferences:GetBool returns true and we report true.  Otherwise we report false.  It would be nice if we instead just reported the value the pref is set to.  That way, if/when we block mixed display by default, the metrics will be easier to compare.

I'm not sure if nsAppRunner is the right place to put this code; but I'm assume someone directed you to it after you asked about adding a probe for an about:config pref.  I'm not familiar with it.

Thanks for this patch Kate!
Attachment #8640777 - Flags: review?(tanvi)
This picks up the telemetry along with the rest of the environmental telemetry. It does not currently make it into the telemetry dashboards, so we will need to develop our own analysis.
Attachment #8640777 - Attachment is obsolete: true
Attachment #8658274 - Flags: feedback?(tanvi)
Reporter

Comment 6

4 years ago
Comment on attachment 8658274 [details] [diff] [review]
Add mixed content blocker prefs to environmental telemetry

(In reply to Kate McKinley [:kmckinley] from comment #5)
> Created attachment 8658274 [details] [diff] [review]
> Add mixed content blocker prefs to environmental telemetry
> 
> This picks up the telemetry along with the rest of the environmental
> telemetry. It does not currently make it into the telemetry dashboards, so
> we will need to develop our own analysis.

Looks good!  How do we develop our own analysis?  Does each consumer of the prefs listed here collects the data from the telemetry server and host it somewhere themselves?  If so, perhaps we can piggyback on one of the existing consumers.
Attachment #8658274 - Flags: feedback?(tanvi) → feedback+
IPython notebook for analyzing the returned telemetry on a weekly basis.
Attachment #8660172 - Flags: review?(rvitillo)
Flags: needinfo?(benjamin)
Attachment #8660172 - Flags: review?(rvitillo) → review+
(In reply to Kate McKinley [:kmckinley] from comment #5)
> Created attachment 8658274 [details] [diff] [review]
> Add mixed content blocker prefs to environmental telemetry
> 
> This picks up the telemetry along with the rest of the environmental
> telemetry. It does not currently make it into the telemetry dashboards, so
> we will need to develop our own analysis.

You could use the notebook to produce a daily/weekly dataset and upload it to S3. A custom dashboard could then read that data and display it.
Alternatively, you could use the notebook itself as a simple dashboard. In order to do that, you can set the notebook up to run on a scheduled basis [1] , which you would have to do anyway to implement my first suggestion.

[1] http://robertovitillo.com/2015/03/13/simple-dashboards-with-scheduled-spark-jobs-and-plotly/

Comment 9

4 years ago
Comment on attachment 8658274 [details] [diff] [review]
Add mixed content blocker prefs to environmental telemetry

Is the NEEDINFO for me here for data collection review? If so you can just use the feedback flag on the patch.
Flags: needinfo?(benjamin)
Attachment #8658274 - Flags: feedback+
Update commit message
Attachment #8658274 - Attachment is obsolete: true
Attachment #8661484 - Flags: review+
Keywords: checkin-needed
https://hg.mozilla.org/mozilla-central/rev/b3f291fbd9f8
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla43
You need to log in before you can comment on or make changes to this bug.