Closed Bug 1380854 Opened 2 years ago Closed 8 months ago

Send a Leanplum event when the user logs in to their Firefox Account

Categories

(Firefox for Android :: Metrics, enhancement, P1)

Other
iOS
enhancement

Tracking

()

RESOLVED FIXED
Firefox 67
Tracking Status
firefox63 --- wontfix
firefox64 --- wontfix
firefox65 --- wontfix
firefox66 --- wontfix
firefox67 --- fixed

People

(Reporter: jcollings, Assigned: vlad.baicu)

Details

(Whiteboard: [Leanplum] [61] [priority:high][fenixtransition][fenix])

Attachments

(4 files)

Looking to create an event when a user signs in / signs up for FxA. This will help us figure which message creates higher conversions.

Not a blocker for 56.
The event could happen if our app is in the background. Or when user is browsing other pages. I wonder if that should be real time thing (once the sign up is success.. we fire that event....)

Hi Grisha.
Do you know when could we get the exact moment when a user successfully signed  for FxA?
Component: General → Metrics
Flags: needinfo?(gkruglov)
What is meant by an "event" here? Akin to using Telemetry.sendUIEvent?

With both sign-in and sign-up there are two stages - "verified", and "unverified". When users sign-up, they will always need to verify their email address. When users sign-in, they sometimes will need to verify their sign-in via some out-of-band mechanism (link an email...).

So we have a combination of various states a user can be in, and even more events we could be sending out.

There is a point at which we create an Android account, that might be high-level enough for your purposes and you'll be able to use our UI telemetry there. See https://dxr.mozilla.org/mozilla-central/source/mobile/android/base/java/org/mozilla/gecko/AccountsHelper.java#90

With some work you should be able to differentiate between a "sign-up" and a "sign-in" at that level.

For completeness:
We also receive a GCM push message once we're verified, and we'll know if we're verified or not whenever we try to sync, but those things all happen in the background, without a reliable access to our regular gecko UI telemetry. We do have our java-side telemetry as well, and plumbing in place to flow various telemetry events out of background services (see https://dxr.mozilla.org/mozilla-central/source/mobile/android/base/java/org/mozilla/gecko/telemetry/TelemetryBackgroundReceiver.java, but it likely won't be useful to you).
Flags: needinfo?(gkruglov)
Comment on attachment 8891287 [details]
Bug 1380854 - Drop an event when User signs in.

https://reviewboard.mozilla.org/r/162492/#review168984

::: mobile/android/base/java/org/mozilla/gecko/AccountsHelper.java:160
(Diff revision 1)
>  
>          } else if ("Accounts:UpdateFirefoxAccountFromJSON".equals(event)) {
>              // We might be significantly changing state of the account; let's ensure our in-memory
>              // accounts cache is empty so that there are no undesired side-effects.
>              AndroidFxAccount.invalidateCaches();
> +            MmaDelegate.track(MmaDelegate.USER_SIGNED_IN_TO_FXA);

This will also fire on password changes. See https://dxr.mozilla.org/mozilla-central/source/mobile/android/modules/FxAccountsWebChannel.jsm#305

It seems that if you don't want to count password changes as logins, you'll need to propagate a type of "update" all the way from FxAccountsWebChannel to java code here.
Attachment #8891287 - Flags: review?(gkruglov) → review-
Comment on attachment 8891288 [details]
Bug 1380854 - Add the document for mma event 'Sign In Firefox Account'.

https://reviewboard.mozilla.org/r/162494/#review170002

Flag me for review again, need a little clarification.

::: mobile/android/docs/mma.rst:143
(Diff revision 1)
>  }
>  * Open a new tab
>  {
>    "event" : "E_Opened_New_Tab"
>  }
> +* Sign In Firefox Account

A little more clarification here - can you list the situations when this is fired? e.g. when they click "Login" to Firefox, and the login completes (or even if it doesn't?
Attachment #8891288 - Flags: review?(liuche)
Whiteboard: [FNC][SPT#57.1][MVP]
Whiteboard: [FNC][SPT#57.1][MVP] → [FNC][SPT#57.1][BL]
Whiteboard: [FNC][SPT#57.1][BL] → [FNC][SPT57.1][BL]
Whiteboard: [FNC][SPT57.1][BL]
Priority: -- → P2
[triage] Bulk edit: product thinks leanplum is high priority: P1 rank 1.
Assignee: cnevinchen → nobody
Rank: 1
Priority: P2 → P1
Whiteboard: [Leanplum] [61]
Assignee: nobody → vlad.baicu
I see that in the previous patch Nevin tracked signed in events, I'm wondering if there should be different events reported to LP for both sign up and sign in?
Flags: needinfo?(sdaswani)
Vlad, I think we need some more product / marketing direction here. Let me NI Kat and add it to the list of items we need direction on.
Flags: needinfo?(sdaswani) → needinfo?(kplam)
Flags: needinfo?(kplam) → needinfo?(cpark)
Cherry, we already track a Sign In event, do you want a separate event ('Sign Up') when folks register for FxA?
Waiting for the team's thoughts, will update with the NI when I have the confirmed answer.
Hello Team,

Yes, we want to track when folks sign up, sign in and sync.

Thank you,

Jess Osorio
Flags: needinfo?(cpark)
Thanks Jess! Vlad feel free to get this on someone's radar as soon as possible.
Flags: needinfo?(vlad.baicu)
Whiteboard: [Leanplum] [61] → [Leanplum] [61] [priority:high]
During my tests I noticed that even with an existing account, when attempting to sign in, 'Accounts.getFirefoxAccount()' does not return an account and it always fired the 'Accounts:CreateFirefoxAccountFromJSON' event. I have added a new event and promise to differentiate between account updates and logins, but for me it never triggered, nor would it with Nevin's solution as well. Grisha is this what you meant by differentiating between the 2 events or maybe I'm missing something here ?
Flags: needinfo?(vlad.baicu) → needinfo?(gkruglov)
What do you mean by "existing account"? It exists in FxA, or it exists on the device?

'getFirefoxAccount' and friends in FirefoxAccounts.java will query the AccountManager API, and see if it has a "Firefox" account in its internal database. Before device is signed-in, it's expected that 'getFirefoxAccount' will return `null` (internal db doesn't have our account). As part of the sign-in flow, 'Accounts:CreateFirefoxAccountFromJSON' is fired which will end up creating a Firefox account in the Android's AccountManager. Once that happens, 'getFirefoxAccount' should return a relevant Account object.

Looking at the *jsm code, I think it's (innocently) wrong actually: https://dxr.mozilla.org/mozilla-central/source/mobile/android/modules/FxAccountsWebChannel.jsm#275 -> we'll attempt to 'update' an account if one isn't returned from 'getFirefoxAccount', but that's meaningless - we can't update what we don't have. And java side will bail out if that path is taken: https://dxr.mozilla.org/mozilla-central/source/mobile/android/base/java/org/mozilla/gecko/AccountsHelper.java?q=AccountsHelper.java&redirect_type=direct#153

(feel free to clean this up in a Pre patch, if you want).

For the purposes of telemetry, I suggest a less risky solution (the current account system is thorny around the edges). Instead of altering the current flow of events or doing any refactorings, add a single "Accounts:MMATelemetry" event with data to indicate what happened (sign-in, sign-up), and fire that event at the source of your "events" - in FxAccountsWebChannel.jsm. This will have a benefit of firing these events after the promise returns (so you know these actions really did succeed), and will make your patch very simple.
Flags: needinfo?(gkruglov)
Attachment #8976222 - Flags: review?(sdaswani) → review?(gkruglov)
While adding a new event to track the mma teletry in jsm is no big deal, my issue is with the fact that in WebChannel we are only notified about the login command with no distinction if the account was created or logged in. For example regardless if the user registered or not in FxA, in the FxAccountsWebChannel.jsm I'm always receiving the command_login event and the data json is of not much help either. It is not exactly clear to me how I can differentiate here between the registers and logins at this level,  my guess is that I need to dig a bit deeper starting with where this listener is attached, any pointers would be of great help
Flags: needinfo?(gkruglov)
Upon receiving COMMAND_LOGIN, jsm code differentiates between "creation" and "login": https://dxr.mozilla.org/mozilla-central/source/mobile/android/modules/FxAccountsWebChannel.jsm#271 - based on the presence of the account in the AccountManager.

In fact, there's already telemetry in place for 'create' and 'login' events:
- https://dxr.mozilla.org/mozilla-central/source/mobile/android/modules/FxAccountsWebChannel.jsm#271
- https://dxr.mozilla.org/mozilla-central/source/mobile/android/modules/FxAccountsWebChannel.jsm#279

Wouldn't that work well enough for your case?
Flags: needinfo?(gkruglov)
I tought so too, however after testing, this line for me at least is never reached https://dxr.mozilla.org/mozilla-central/source/mobile/android/modules/FxAccountsWebChannel.jsm#279

I created an account, confirmed it by mail, logged out and then logged in again. Tried multiple other scenarios like clearing private data, signed in on other devices as well, etc. Put logs everywhere and it never reached that point for me.
Since we are receiving telemetry events from that line I'm guessing that we are either missing something or this is a new problem
Flags: needinfo?(gkruglov)
Status: NEW → ASSIGNED
Re-triaging per https://bugzilla.mozilla.org/show_bug.cgi?id=1473195

Needinfo :susheel if you think this bug should be re-triaged.
Priority: P1 → P5
Emailing grisha to follow up.  Is this still a problem that affects Nightly 65? 
Stefan, do you have any thoughts on who can advise us on the possible sync issue?
Flags: needinfo?(sarentz)
Priority: P5 → P1
Hi Liz,

This bug isn't really a "problem" that affects users but rather a request to track some additional metrics around usage of sync/fxa. I'll follow up with Vlad on implementation details.
Flags: needinfo?(sarentz)
Janet, Grisha has moved off Sync, can you find someone to help us with this?
Flags: needinfo?(gkruglov) → needinfo?(jdragojevic)
This ticket is more than 1 year old, and there's a few things that are not clear to me...

1. What is the platform? (Field shows iOS, seeing comments related to Android)
2. Is this functionality still needed by the original reporter @Jean Collins?
3. What exactly is being requested and what is the priority of the request?
Flags: needinfo?(jdragojevic) → needinfo?(jcollings)
Hi Janet:
1) Should be for Android - not sure why Platform says "Other iOS". It's in the Firefox for Android Product 
2) I am not working on this project anymore and will cc Miray and team for confirmation - but I believe this is still needed 
3) The original request was the ability to track when a user actually signs in/ signs up for FxA based on our marketing messages. We currently cannot track this event through Leanplum.
Flags: needinfo?(malanlar)
Flags: needinfo?(josorio)
Flags: needinfo?(jcollings)
Thanks Jean for the note!

Hi Janet,
Nice to e-meet you here!
I'm running Leanplum (our mobile app engagement SaaS) for product in-app communications with NCPG team now.
I've had a chance to discuss this with our PMM (laustin@mozilla.com ) and my team lead Jess (josorio@mozilla.com ) without knowing this was requested before. As mentioned, we can't currently track sign-ups and sign-ins for FxA at Fennec. However, We can track those for Firefox- IOS which gives us better visibility on the effectiveness of the campaigns.

I'll follow up with the priority of the request but in the meantime, feel free to reach out if you have any questions.

Best,
Miray
Flags: needinfo?(malanlar)
Flags: needinfo?(adavis)
I'm not sure that a new event is needed here. If the appropriate utm parameters are passed to FxA forms, we can easily track logins and registration for a given flow or touchpoint in FxA telemetry (accessible via Re:Dash and Amplitude).
Flags: needinfo?(adavis)
Hello Alex, I don't think there is a need for new event, last time I tested, the events do not trigger as they should in FxAccountsWebChannel.jsm and we cannot distinguish properly between logins and registrations. Should we submit a new bug for this issue ?
Flags: needinfo?(adavis)
> As mentioned, we can't currently track sign-ups and sign-ins for FxA at Fennec.

Do you absolutely need to track the performance of LeanPlum campaigns to drive FxA registrations and logins in LeanPlum or would you be OK with tracking them in Firefox Accounts telemetry? (available in Amplitude: https://analytics.amplitude.com/mozilla-corp/team-spaces )

If you are OK with measuring success of LeanPlum campaigns in Amplitude, then we explain what utm paramaters you can pass us to measure the success of your campaigns here: https://mozilla.github.io/application-services/docs/accounts/metrics.html
Flags: needinfo?(adavis)
I think someone from marketing is best suited to reply to this question
Flags: needinfo?(malanlar)
Hi Alex,

It would be ideal for us to track sign-ups and sign-ins for FxA at Fennec through Leanplum and yes we would need it.

E.g: We've been testing aggressive messaging to our FxAndroid users during their first 7 days to improve retention and some of push messages are " Get an Fx Account" and "Set us as Default". We are able to track adoption level and percentage of lift on users setting FxAndroid as their default browser however it's hard to see if they sign up for an account as a result of a certain lifecycle campaign.

I'm not sure how easily and effectively we can attribute LP campaigns' success in Amplitude or outside of LP but that'd be helpful if we can fully leverage LP analytics as they constantly improve the tool and provide a better understanding on performance. 

Lmk if you need to jump on a call and discuss this further.

Best,

Miray
Flags: needinfo?(malanlar)

loines: can you look into what would be needed to pass FxA utm params on mobile so that they can track the success of Leanplum campaigns in Amplitude? (campaigns that promote accounts)

Flags: needinfo?(loines)

I'll do my best to get to this by EOW (sort of swamped ATM)

After discussing with Alex IRL, we're going to continue with a request for a Leanplum event, but that work will need to be coordinated by the Fennec team.

Passing mobile attribution data through UTMs still needs to be explored, but is not a requirement for this specific project to proceed.

Vlad,

Yes, we'e still pursuing the "signed up for FxA" event, as per Miray's comment #32. What is the ETA for getting this complete?

Leif,
We spoke with Alex, earlier this week and he sent us a rough example of what a deeplink with tracking parameters might look like. It might work as is. We're going to test the tracking deeplink below. If it works, we'll proceed with this template for future tracking deeplinks. If it doesn't work we'll need help from Android Engineering and you, Leif, to develop tracking deeplinks that work.

firefox://sign_up?uid={{User ID}}&utm_source=leanplum&utm_campaign=f7t&utm_content=d0_syn_now

Thank you,

Jess Osorio

Flags: needinfo?(vlad.baicu)
Flags: needinfo?(josorio)
Flags: needinfo?(abovens)

Leif,

If we need to create new "tracking deeplinks" we will open a new bug that includes engineering and you and start it as a new request. Thank you!

  • Jess Osorio

The patch on the Android side is very small, however I need some help on JS in order to proceed further with this issue.

Flags: needinfo?(vlad.baicu)
Flags: needinfo?(abovens)

Vlad, do we have confidence we can we discard the original FxA based solution and go with the deeplinks?

If so, can you start the Java work and explain here what kind of help you need with the JavaScript side? I will find someone who can assist with that part.

Flags: needinfo?(vlad.baicu)
Whiteboard: [Leanplum] [61] [priority:high] → [Leanplum] [61] [priority:high][fenix]
Whiteboard: [Leanplum] [61] [priority:high][fenix] → [Leanplum] [61] [priority:high][fenixtransition]
Whiteboard: [Leanplum] [61] [priority:high][fenixtransition] → [Leanplum] [61] [priority:high][fenixtransition][fenix]

Apologies, I got this all wrong. Ignore my previous comment. There seem to be two specific requests in this bug:

  1. Leanplum events for FxA SignIn and Register
  2. Adding UTM tracking to the firefox://sign_up deeplink (not sure if that is a new or existing deeplink)

Vlad, can you explain where were we are with this Leanplum bug and what I can do to unblock you?

Jessica, can you file a new bug for the deeplinking part and ping me on Slack with the bug number? I'll keep track of that one too.

Flags: needinfo?(josorio)

Hi Stefan,

Leanplum Events we're seeking for Fennec:

  • "User registers for a Fx Account"
  • "User has synced another device"

Questions, per https://bugzilla.mozilla.org/show_bug.cgi?id=1380854#c2, I'm wondering if it is possible to know this much information about FxAccount we drive through leanplum? Maybe this is too complex for Fennec cross-sell purposes. But we should definitely get this cleared up before we start creating FxA attributes and events for Fenix.

Thank you,

Jess Osorio

Flags: needinfo?(josorio) → needinfo?(sarentz)

(In reply to Vlad Baicu from comment #29)

Hello Alex, I don't think there is a need for new event, last time I tested,
the events do not trigger as they should in FxAccountsWebChannel.jsm and we
cannot distinguish properly between logins and registrations. Should we
submit a new bug for this issue ?

Hello, last time I tried to implement this, there was a bug on the JS side of FxA which could not allow us to properly distinguish between the events and track the metrics we want. In order to unblock this issue I would need a few hours with an engineer that has expertise on FxA.

Flags: needinfo?(vlad.baicu)

Hi Stefan,

Also confirming here that Loren filed that deeplink bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1522704

Thank you,

Jess Osorio

Vlad, we have simplified this request to the following:

“Send a Leanplum event when the user logs in to their Firefox Account”

This leanplum event will fire when the account status changes from ‘not logged in’ to ‘logged in and verified’. We will not be able to see the difference between ‘sign in’ and ‘account creation’.

This event will not be real-time but will fire within a reasonable amount of time after the user has logged in.

Flags: needinfo?(sarentz) → needinfo?(vlad.baicu)
Summary: Drop an event when User signs in / signs up for FxA (from Leanplum contextual hints) → Send a Leanplum event when the user logs in to their Firefox Account

I've simplified the patch and tested it on my development environment, the events appear in LP almost immediately. Also added an event to track when the sync is finished.

Flags: needinfo?(vlad.baicu)
Keywords: checkin-needed

Pushed by nbeleuzu@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/1d82c648684d
Track Sign in and sync finished events in LP.r=petru

Keywords: checkin-needed
Status: ASSIGNED → RESOLVED
Closed: 8 months ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 67

Since Leanplum isn't working on Nightly, I have tested this issue with a build from Vlad and I can confirm that the events are triggered photo.

Comment on attachment 9049513 [details]
Bug 1380854 - Track Sign in and sync finished events in LP.r=petru

Beta/Release Uplift Approval Request

  • Feature/Bug causing the regression: None
  • User impact if declined:
  • Is this code covered by automated tests?: No
  • Has the fix been verified in Nightly?: No
  • Needs manual test from QE?: Yes
  • If yes, steps to reproduce: Sign in to Firefox Accounts, trigger a sync, watch the events appear in Leanplum.
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): We are simply adding 2 more events to be tracked in LP.
  • String changes made/needed:
Attachment #9049513 - Flags: approval-mozilla-beta?

Is this a release blocking issue for 66?

Flags: needinfo?(sarentz)

I think this is too late to make the RC build for 66 unless it is really a release blocking issue.

After discussion with Stefan we are going to hold off on this for it to have time in beta on 67.

Attachment #9049513 - Flags: approval-mozilla-beta? → approval-mozilla-beta-

Stefan, I took a look at the patch and it'll overcount sign-in events. It's emitting a sign-in event while processing a 'UpdateFirefoxAccountFromJSON' message, which will be emitted during password changes: https://dxr.mozilla.org/mozilla-central/source/mobile/android/modules/FxAccountsWebChannel.jsm#304 (also, see Comment 5).

It's a little unfortunate, but changing this behaviour will require a more involved patch (not a hard one: just need to pipe up an extra boolean flag from the JS side).

... but, I imagine that as long as we know what the password change volume is, roughly, the marketing team can adjust their data accordingly?

Grisha - this should be workable for Marketing's purposes in the short term. Can you confirm the password change request would only occur if a user has an account?

Here's my logic:
Taken as a given that a password change would only occur if a user has an account.
We would be targeting users who we have never seen sign in to account, and we therefore assume do not have an account.

If we message them and they change their password, that would be confirmation they have an account.
If we message them and they sign in, that would be confirmation they have an account.

Therefore, the two events are functionally equivalent: both give us confirmation that the user has created an account.

Flags: needinfo?(josorio)

@Grisha - are we moving ahead with this solution?

Flags: needinfo?(gkruglov)

We should only see password change events for users with an account, yes.
So overall this sounds reasonable to me. I misunderstood the purpose for shipping this! If it's to differentiate FxA users from non-FxA users for in-product communication purposes, then the shipped solution should be fine.

Flags: needinfo?(sarentz)
Flags: needinfo?(loines)
Flags: needinfo?(josorio)
Flags: needinfo?(gkruglov)
You need to log in before you can comment on or make changes to this bug.