1584308 - ETP On Analysis

Status: RESOLVED FIXED

(Data Science :: General, task)

Description

[Tanvi Vyas[:tanvi]]

Brief description of the request:
With Firefox 69, we turned on enhanced tracking protection (network.cookie.cookieBehavior=4) for all users. But the data is showing that we are not at 100% (or close to 100% of users).

Link to any assets:
https://sql.telemetry.mozilla.org/queries/64085#163704 - shows 73%
https://sql.telemetry.mozilla.org/queries/65049/source#165711 - shows 97% of Firefox 69 users, and then a decline to 86%.

This drop from 97 to 86% doesn't really make sense. 10% of users turning it off Globally within 3 weeks. We know breakage is limited. And we have seen sites that ask the user to turn off the protection on a per page basis, but not globally. 10% turning it off globally is quiet high given this.

There was some facebook game breakage which we fixed https://bugzilla.mozilla.org/show_bug.cgi?id=1580416

Is there a specific data scientist you would like or someone who has helped to triage this request:
Felix or Ben.

Comment 1

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

The numbers are actually much better! The histogram is only reported once per browser session, at startup (it's scheduled here). We can receive multiple pings per browser session; browser sessions can span multiple calendar days. On all subsequent pings in a browser session, histogram_parent_cookie_behavior will be NULL.

The query at https://sql.telemetry.mozilla.org/queries/65049/source#165711 treats NULL as "off" and aggregates by UTC calendar days. If the browser is used for multiple days, the user will be recorded as having ETP turned off for all days except the first.

A new query at https://sql.telemetry.mozilla.org/queries/65059/source#165731 additionally aggregates by session_id, so that it only counts users on the day they sent their first ping from a browser session (i.e. the ping that contains the histogram value). It shows values much closer to 99% over the reporting period.

Comment 2

[Jess Mokrzecki [:jmok]]

Hi Tanvi,

After looking at Tim's input above could you let us know if this bug is still needed?

Comment 3

[Tanvi Vyas[:tanvi]]

Thanks Tim! This is great! Glad you were able to explain this and update the query. Would you be to run this query for older Firefox versions? Specifically, versions between 66-68? So we can see how many users have not upgraded to 69+, but have ETP enabled (either on their own or through a study). If this isn't the best way to achieve retrieving that data, that is fine also. We can work out another way to get it and file a separate data science bug for it.

Thanks!

Comment 4

[Jess Mokrzecki [:jmok]]

Please note your request has been received and reviewed. At this time we are not assigning new work unrelated to skyline until after it has been released and the team has more capacity for requests.

Comment 5

[Tanvi Vyas[:tanvi]]

(In reply to Jess Mokrzecki [:jmok] from comment #4)

Please note your request has been received and reviewed. At this time we are not assigning new work unrelated to skyline until after it has been released and the team has more capacity for requests.

Okay thanks! Let's pick this up after skyline. The information gathered here will help inform what to do with a few studies that only apply to Firefox 66-68 users. We can punt that decision to later, and let the studies continue running for now.

Comment 6

[Tanvi Vyas[:tanvi]]

Now that Skyline is over, does data science have capacity to pick this up?

Comment 7

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Jess is a better person to ask; redirecting.

It would be helpful to understand what decisions those queries would help you make.

Comment 8

[Tanvi Vyas[:tanvi]]

(In reply to Tim Smith 👨‍🔬 [:tdsmith] from comment #7)

It would be helpful to understand what decisions those queries would help you make.

(In reply to Tanvi Vyas[:tanvi] from comment #5)

The information gathered here will help inform what to do with a few studies that only apply to Firefox 66-68 users. We can punt that decision to later, and let the studies continue running for now.

If the number of users is low, we can turn those studies off now. If the numbers are still high, we should leave those studies on so the users continue to get ETP.

Comment 9

[Felix Lawrence]

I had a go at extending tim's query to answer your question. I made a time series of the fraction of client sessions with ETP on per day, broken down by app version. Looks like v70 has 98-99%; v69 has 93-95%, then there's a big drop: v68 has around 30%, v67 has around 26%. v66 has only 1%.

To put this in context, I also plotted the number of client sessions per day with each version of firefox (well I accidentally divided it by 100 - but the proportions should be about right). About 4-5% of client sessions are on v66-68.