Closed Bug 1269466 Opened 8 years ago Closed 7 years ago

DAU/MAU Engagment ratio for Push Users

Categories

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P2)

Product:
Cloud Services Graveyard
Component:
Metrics: Pipeline
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: clarkbw, Assigned: bugzilla)

References

Details

Attachments

(3 files)

ER for push profiles
5.44 KB, application/pdf
Details
ER for webnotif profiles
5.44 KB, application/pdf
Details
[telemetry-batch-view] mozilla:bug-1269466 > mozilla:master
56 bytes, text/x-github-pull-request
Details | Review 
The Push product team is trying to better understand the nature of Push and its effect on Firefox users.

Our initial hypothesis is: "We believe a push user is a more engaged Firefox user than the average Firefox user" where we compare engagement ratios of a push user to the current definition of an average firefox user.

That definition immediately leads to the question of "What is a push user?"

And here are some additional hypotheses on how we could define a push user.

A Push User is:

A Person who has accepted the Push Permission Dialog
POPUP_NOTIFICATION_STATS (opt-out)

A Person who received a Push Notification
PUSH_API_NOTIFICATION_RECEIVED (opt-in)

A person who has seen a Web Notification
WEB_NOTIFICATION_SHOWN (opt-out)
:rweiss - Bryan mentioned you were planning on running some queries to get some early numbers.  Please let us know how that's going.
Flags: needinfo?(rweiss)
This is pushed until after Test Pilot launch.  Needinfo me again on 5/23/16.
Flags: needinfo?(rweiss)
over to Ryan in case he can find time/people to work on this in the near term.
Assignee: nobody → ryanvm
Roberto, can you or Mauro please take a look at this?
Flags: needinfo?(rvitillo)
Depends on: 1270482
Flags: needinfo?(rvitillo)
Assignee: ryanvm → rvitillo
moving to pipeline for easier tracking
Component: Metrics: Product Metrics → Metrics: Pipeline
Points: --- → 2
Priority: -- → P2
:rweiss you requested to be need info again. Push team is blocked on this, needing support to understand how we derive our success metric.
Flags: needinfo?(rweiss)
:rweiss is working on a high priority task for me right now and unfortunately doesn't have bandwidth to handle this. Can we find another way of getting a measurement that doesn't involve her?
Flags: needinfo?(rweiss)
:joy can you help us review the 'user story' on defining what is a 'push user' then we'll need DAU/MAU numbers.
Flags: needinfo?(sguha)
I can spend some time on this. I will need to see if we collect the data in our data sets (and which one).
I also recommend keeping your definition of push user to restricted to opt out measures.

When was this probe inserted into Firefox? it is part of telemetry or telemtry-ui?


A Person who has accepted the Push Permission Dialog
POPUP_NOTIFICATION_STATS (opt-out)

A person who has seen a Web Notification
WEB_NOTIFICATION_SHOWN (opt-out)
Flags: needinfo?(sguha)
both were set to opt-out in Feb targeting Fx 47 see bug 1228671
Got time on this but a bit confused, a push user as defined in Comment 9, is someone who sees

a) popup_notification
and/or
b) web_notification

A popup_notification could come from a profile that uses FX password manager
A web notification could come from a website (e.g.irccloud)

All of these come under the definition of 'push user'
Is this correct?

For example one particular user had


 Row(offered=1, action_1=0, action_2=0, action_3=0, action_last=0, dismissal_click_elsewhere=1, dismissal_leave_page=0, dismissa
l_close_button=0, dismissal_not_now=0, open_submenu=0, learn_more=0, reopen_offered=0, reopen_action_1=0, reopen_action_2=0, reopen_action
_3=0, reopen_action_last=0, reopen_dismissal_click_elsewhere=0, reopen_dismissal_leave_page=0, reopen_dismissal_close_button=0, reopen_dis
missal_not_now=0, reopen_open_submenu=0, reopen_learn_more=0),
 u'password': Row(offered=1, action_1=0, action_2=0, action_3=0, action_last=0, dismissal_click_elsewhere=1, dismissal_leave_page=0, dismi
ssal_close_button=0, dismissal_not_now=0, open_submenu=0, learn_more=0, reopen_offered=0, reopen_action_1=0, reopen_action_2=0, reopen_act
ion_3=0, reopen_action_last=0, reopen_dismissal_click_elsewhere=0, reopen_dismissal_leave_page=0, reopen_dismissal_close_button=0, reopen_
dismissal_not_now=0, reopen_open_submenu=0, reopen_learn_more=0)}

and yet web_notification_shown is None.

So they had a password popup but no web notifications. Hence user is considered a 'push user'


Did i get that correct?
Regards
Saptarshi
Hello
I took a 1% sample of clients from main_summary/v3 of Firefox profiles on versions 47 or
above. This included their usage behavior and the variables
'web_notification_shown' and 'popup_notification_stats' (in order to define push).

## Push User

A user is classified as a 'push user' if they had at least one non empty/non
zero value for 'web_notification_shown' and/or 'popup_notification_stats' during the
period 2016-07-01 ... 2016-07-31.

A user is a webnotification user if they had at least one non empty value for
"web_notification_shown" during the same period.


## Results
19% of clients have no information in July. There were dropped.
Among the remaining 20%,

- 68.7% of clients are push profile
- 1.8% are web notification users

## Figures

Figure x1.pdf graphs the engagement ratio for the period  Aug, 2016 for Push
users (TRUE) vs non push (FALSE) users (and overall). Push users are associated
with higher engagement ratios.

Figure x2.pdf graphs the ER for same period for webnotification users(TRUE) vs
non webnotifcation users (FALSE). They too are associated with higher ERs.

## Caveat

These users might have always been engaged with our browser even before the introduction of Push notifications. In other words, Push notifications hasn't necc. changed behavior, but is associated with higher behavior.
Assignee: rvitillo → ssuh
In the process of getting my changes above reviewed and chatting about this with Bryan, it sounds like we should tweak the definition of "push user" -- specifically, instead of just checking for a non-null POPUP_NOTIFICATION_STATS histogram, we should be looking at a specific key in there called "web-notifications", and the "action_1" value within that key (which indicates agreement to the push notification permission popup.)

With the new beefier Presto cluster I was able to get a query running on a 1% sample of main_summary that I think gives us a pretty good measure (pending review, so please don't rely on these numbers quite yet):
https://sql.telemetry.mozilla.org/queries/1356/source#2394

A couple notes about the query:
- The definition of push user in this query is someone who accepted a push permission dialog (popup_notification_stats['web-notifications'].action_1 > 0) and then received a web notification (web_notification_shown > 0) within 30 days of the push acceptance. This means, however, that a client can be counted as a push user in dau *before* they receive the web notification (which seemed acceptable to me). This *also* means that there will be a falloff in MAU and DAU counts on recent dates since there's less of a chance for a user to qualify as a "push user".
- There's an ugly hack in the query where I create an array of client IDs to get push user mau since presto can't do a real RANGE window or COUNT DISTINCT within a window. Given that the number of clients that qualify as push users is relatively low (~1500/day accepting the push permission dialog within the 1% sample), this works (but will probably become a problem if the number of push users grows significantly.)
- I'm using s3_submission_date for the date -- I'm not sure this is the correct date to use for calculating ER
- The numbers are scaled back up by multiplying by 100 at the end of the query to reflect numbers that are closer to the whole user population

:rvitillo would you mind reviewing this for correctness & style?
:bwclark could you review this to make sure the overall data maps to your intuition and the metrics used make sense?

After this is reviewed and approved I'll stick these graphs into a dashboard and schedule periodic runs.
Status: NEW → ASSIGNED
I left some comments here: https://gist.github.com/vitillo/73851c25ef33987e24bbc175808fc3d4. We really need a better way to review those queries...
Thanks Roberto! I'm going to address this comment here so this info is in the bug (and so that Bryan can correct me if I got any of this wrong):
> RV: why do we need to check explicitly for push acceptance? I.e. can’t we infer push acceptance from a push notification and treat push notifications and acceptance more generally as push events?

The histogram that tells us whether a user has received a push notification or not is PUSH_API_NOTIFICATION_RECEIVED, but that's an opt-in metric so it's not particularly useful for this. The WEB_NOTIFICATION_SHOWN histogram tells us about usage of the firefox notification API (e.g. the kind that manifests as a desktop banner notification on OS X) but this API is used by both push notifications as well as HTML5 web notifications. The "web-notifications" key in the POPUP_NOTIFICATION_STATS histogram can tell us that someone's *accepted* a website's request to send push notifications (the push permission popup). A surprising number of users don't have any web notifications within 30 days after accepting the push permission (around 50%). We'll certainly get false-positives (users who never actually received a push notification, only web notifications after accepting the push permission popup) and false-negatives (users who eventually received a push notification after the 30 day window) by defining "push users" in this way, but this seemed like a reasonable compromise given the constraints.
(In reply to Sunah Suh from comment #18)
> Thanks Roberto! I'm going to address this comment here so this info is in
> the bug (and so that Bryan can correct me if I got any of this wrong):
> > RV: why do we need to check explicitly for push acceptance? I.e. can’t we infer push acceptance from a push notification and treat push notifications and acceptance more generally as push events?
> 
> The histogram that tells us whether a user has received a push notification
> or not is PUSH_API_NOTIFICATION_RECEIVED, but that's an opt-in metric so
> it's not particularly useful for this. The WEB_NOTIFICATION_SHOWN histogram
> tells us about usage of the firefox notification API (e.g. the kind that
> manifests as a desktop banner notification on OS X) but this API is used by
> both push notifications as well as HTML5 web notifications. The
> "web-notifications" key in the POPUP_NOTIFICATION_STATS histogram can tell
> us that someone's *accepted* a website's request to send push notifications
> (the push permission popup). A surprising number of users don't have any web
> notifications within 30 days after accepting the push permission (around
> 50%). We'll certainly get false-positives (users who never actually received
> a push notification, only web notifications after accepting the push
> permission popup) and false-negatives (users who eventually received a push
> notification after the 30 day window) by defining "push users" in this way,
> but this seemed like a reasonable compromise given the constraints.

I see, that makes sense. I can see how this definition is useful to measure push users right now. What about adding a new measurement (scalar or histogram) that counts push notifications so that we avoid false positives and false negatives in the future?
That's PUSH_API_NOTIFICATION_RECEIVED, which is unfortunately opt-in. I'm not sure how the opt-in/opt-out decision is made, but if there's a way to get the ball rolling on getting it to be opt-out that definitely feels like it'd be the best solution.
Flags: needinfo?(gfritzsche)
Engineering-wise, it's adding "releaseChannelCollection":"opt-out" to the histogram entry:
https://dxr.mozilla.org/mozilla-central/rev/26f2e65163e9f3cfab071a9823293217b1e1de8d/toolkit/components/telemetry/Histograms.json#9716

The decision process is started with a request for data review from a data collection peer, as outlined here:
https://wiki.mozilla.org/Firefox/Data_Collection
Flags: needinfo?(gfritzsche)
I was just reviewing this from the start and found something very interesting.  In bug 1276693 comment 3 Kit mentioned that PUSH_API_NOTIFICATION_RECEIVED would be less accurate than PUSH_API_NOTIFY and so actually made PUSH_API_NOTIFY opt-out in that bug.

So I believe we might actually have the metric we need, I'm not sure how it got lost in the shuffle.
Okay, Firefox 50, the version with PUSH_API_NOTIFY went out to release on 11/8. We added the column to main_summary starting on that day. I've written up a query that uses the column but it won't start being very useful until we have a few more days of data. Link is here: https://sql.telemetry.mozilla.org/queries/1667/source#2960
Depends on: 1311174
Correction: it's not going out to release until 11/15 -- didn't realize the date had slipped. We'll have to wait a bit longer for useful results.
Sunah, why is the smoothed DAU lower?  The current DAU looks fairly smooth already.  Is it just too early?
Yeah, basically -- it's the 7-day running average so given DAU has been rising steadily until 11/25 (from upgrades rolling through) we probably won't see a stable smoothed DAU (and, consequently, ER) until ~12/2
The numbers seem to have stabilized a bit now, so I've added them to the push dashboard: https://sql.telemetry.mozilla.org/dashboard/push

If you have any other comments/changes you'd like to see, please comment in the next few days -- otherwise, I'm going to close this ticket next week
Everything looks good to me, feel free to close this.  Thanks for all your work!
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Product: Cloud Services → Cloud Services Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: