1019395 - Strategy (eg, telemetry) to determine success of sync migration progress.

Status: RESOLVED FIXED

(Firefox :: Sync, defect)

Description

[Mark Hammond [:markh] [:mhammond]]

We should ensure we can get some insights into this migration - eg, number of users who actually receive the EOL notification, how many of them click on the migration offer, how many of these successfully complete, etc.  I'm guessing this might be a combination of client-side telemetry and services-side reports (cc rfkelly for this part).

This will allow us to tweak the UX if we find conversion rates are lower than we hope/expect.

Comment 1

[Edwin Wong [:edwong]]

Katie, might you be the right person to identify the KPI for this. Maybe Ryan can help execute the metrics that we need.

Comment 2

[Ryan Kelly [:rfkelly]]

IIRC :bobm already does bit of server-side analysis to see how many sync1.1 accounts have moved over to sync1.5, i.e. calculating the intersection of "email addresses in sync1.1" and "email addresses in FxA".  If we want more specific server-side measures we can certainly add them.

Comment 3

[Mark Hammond [:markh] [:mhammond]]

We might need multiple bugs here, but I think there is also interesting data we can capture purely in the client about how effective our "nagging" is - off the top of my head, things like "how many times did we show the notification before the user acted?", "how often was the notification dismissed vs how often was it ignored", etc.  We could use this data to tweak our messaging to try and achieve higher upgrade rates (eg, a high rate of "user ignored notification without dismissing it" might imply they simply didn't notice it.)

Comment 4

[Katie Parlante]

Use cases/principles:

1. As an engineering/ops manager/developer, I want to know how many people are still using sync1.1 so that I can make decisions about taking down the server(s).

2. As a UX/product manager/developer, I want to know how many people are succeeding/failing at the migration process (and ideally how they are failing), so that we can take corrective action (including tweaking the UX) if things are going awry.

For the first one, we have existing measures:
- How many sync1.1 accounts have moved over to sync1.5 (intersection of email addresses)
- Daily active sync1.1 users
- Daily active sync1.5 users (+ new accounts per day)
The first two are run manually (we don't have a friendly dashboard for sharing with a wide audience). :edwong, is it good enough to run these queries manually at a few points in time? Should we define/schedule that?

FWIW, my understanding is that we also know that ~6% of sync1.1 users have a browser that is not sync1.5 compatible.

For the second set of measures, if we want to track the number of people that go on to "join the party" (login/attach a device), resend email for verification, or create an account via the migration party, we should pass in a marker to the content-server client metrics. We could use "entrypoint=migration", or some custom queryparam.

:markh, you probably have a better idea how to use telemetry effectively for the client activity (good suggestions above). John and Ryan framed questions this way:
- How many people get each notification/messaging push? What is the success rate for each push (i.e. dismissed vs ignored)?
- Once we signal a user, how long does it take them to take action? How long to finish?

Comment 5

[Katie Parlante]

(In reply to Katie Parlante from comment #4)
> For the second set of measures, if we want to track the number of people
> that go on to "join the party" (login/attach a device), resend email for
> verification, or create an account via the migration party, we should pass
> in a marker to the content-server client metrics. We could use
> "entrypoint=migration", or some custom queryparam.

I should say, if we want to track/investigate whether or not the users successfully complete those tasks.

Comment 6

[Mark Hammond [:markh] [:mhammond]]

Katie, I just opened bug 1097406 and bug 1097410 which should capture the stuff we need in the client.  Does that sound OK to you (or can you delegate the needinfo? as necessary)?

Comment 7

[Katie Parlante]

:markh, looks good -- I also added the "entrypoint" bug.

Comment 8

[Edwin Wong [:edwong]]

I think we're good here - removing NI on me

Comment 9

[Mark Hammond [:markh] [:mhammond]]

I think we have all the telemetry we need via those other bugs.