In order to measure how effective our Sync setup in firstrun is, we want to correlate firstrun experiment actions with whether people end up with a sync account. What telemetry data do we have for "user has a sync account", and is that accessible to the data analysis tools (redash?) that adavis and barbara use to analyze firstrun success? Alex, Barbara, what sources/tools do you use to collect data  for deciding onboarding experiment success? Do they give you enough data about sync? Are there probes you still need in the tools that you use to make correlations? mfinkle: do you know how to add more data to redash?   https://gecko.readthedocs.io/en/latest/toolkit/components/telemetry/telemetry/index.html  https://wiki.mozilla.org/Mobile/Metrics/Redash#Data_Tables_.2F_Schemas
2 years ago
Barbara and I have been using redash to access test data. To my knowledge, we've only been able to tell if a user clicked on the button in onboarding to start the process to create an account but we never know if they actually finished making it. I am unaware of how I could find out if they successfully made an account. (or are logged into to one) When reviewing our wiki, I am unsure if or where I could find the account information: https://wiki.mozilla.org/Mobile/Metrics/Redash
To clarify, we would like to know with certainty that someone completed making an account during onboarding (or during first session). I don't know if can tell that now in redash.
I'm sure there's somewhere where we track whether a user is signed into a sync account, but perhaps it's just not accessible from Redash. Alex, what data have our previous experiment results been concluded from, the button click data, or a general trend in Sync accounts, or something else? I suspect that the sync account completion is tracked on the sync servers, which would have no idea where the sync setup was initiated from. Would it be sufficient to correlate whether people in experiments are signed into a sync account, or not? mfinkle, do you know what determines whether a probe or telemetry is part of Redash? Who would be a good person to talk who knows the places where that sync account state is collected, and whether we can get that data into Redash to use in analysis? If none of that is possible, we can add some kind of hacky probe for when we receive the "Your account has been confirmed - Sync preferences" page, and that would presumably be an event that triggers within the onboarding session.
adding nalexander and katie, as I believe we had a similar conversation with them previoulsy
(In reply to Chenxia Liu [:liuche] from comment #3) > Alex, what data have our previous experiment results been concluded from, > the button click data, or a general trend in Sync accounts, or something > else? I suspect that the sync account completion is tracked on the sync > servers, which would have no idea where the sync setup was initiated from. > Would it be sufficient to correlate whether people in experiments are signed > into a sync account, or not? -We've been using the clicks on the button that's in the firstrun funnel. -I'd be fine to know if people signed into an account (and when).
We added several event probes for interactions with a Fx Account: fxaccount-changepassword: user changes a password fxaccount-changeprofile: user changes their profile avatar (i think) fxaccount-create : user creates a fx account fxaccount-delete : user deletes a fx account fxaccount-login: user logs into their fx account fxaccount-syncprefs: user edits their sync preferences
It looks like we should be able to select these fxaccount action extras tagged with the firstrun session from a funnel of each onboarding experiment, so I'll close this bug. I'll play around with this later too, and see if I can make a query that does this.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.