Closed Bug 1249029 Opened 8 years ago Closed 8 years ago

Add quality metrics for FxA Verification via Push

Categories

(Firefox :: Firefox Accounts, defect)

defect
Not set
normal

Tracking

()

RESOLVED FIXED
Firefox 48
Tracking Status
firefox47 --- wontfix
firefox48 --- fixed

People

(Reporter: vladikoff, Assigned: vfilippov)

References

Details

(Whiteboard: [fxa-waffle])

Attachments

(1 file)

We need to be able to measure if FxA Verification via Push works properly. 

This issue is related to: https://bugzilla.mozilla.org/show_bug.cgi?id=1235607
It does not block the issue above.
Depends on: 1235607
Component: Sync → FxAccounts
Product: Firefox → Core
Version: 47 Branch → unspecified
Whiteboard: [fxa-waffle]
Comment from markh in Bug 1254404

"Bug 1247786 landed basic support for a push notification to detect when an account is verified. Polling is still enabled as a backup, so we should try and measure how effective push is, with an eye to either fixing it if it doesn't get most users, or removing (or severely limiting) the polling if it does."
According to the feature card at [1] our current "success criteria" are:

"""
This feature will be successful if we successfully eliminate polling for account verification from the browser code on Desktop. Mobile browsers are not in scope for the initial version of this feature.

As a concrete metric we can measure the number of calls to /recovery_email/status and check that it decreases as this feature rolls out.
"""

This doesn't seem to capture the current state of things.  At the very least, we should de-scope to remove the word "eliminate" ;-)

If I understand correctly, the patch for this at [2] reduces the frequency with which we poll /recovery_email/status.  I definitely expect this to decrease the number of calls to that endpoint, but it will do so even if push is not working.  So what's the new strategy for measuring success of this feature?

Vlad, in Bug 1254404 Comment 1 you mentioned a server-side strategy using DataDog, when you get a chance can you please update the feature card to include some details of it?

[1] https://github.com/mozilla/fxa/tree/master/features/FxA-54-push-on-status-change
[2] https://hg.mozilla.org/integration/mozilla-inbound/rev/4221fa1f45c3
(In reply to Ryan Kelly [:rfkelly] from comment #3)
> If I understand correctly, the patch for this at [2] reduces the frequency
> with which we poll /recovery_email/status.  I definitely expect this to
> decrease the number of calls to that endpoint, but it will do so even if
> push is not working.  So what's the new strategy for measuring success of
> this feature?

Note also that in the patch as landed, we still check /status after receiving the push notification (ie, we don't treat the message itself as "user is verified" but instead as "now would be a great time to check if they are verified".) So without the client sending a specific signal to the server (eg, query param) I'm struggling to see how the server can help work out whether the client hit that endpoint due to normal polling, or due to the push message. (I guess if the server knows the time the message was delivered and the time the /status call was made it could infer this, but that sounds awkward)
Comment on attachment 8731000 [details]
MozReview Request: Bug 1249029 - Add quality metrics for FxA Verification via Push r=markh

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/40301/diff/1-2/
https://reviewboard.mozilla.org/r/40301/#review36811

@markh, do you agree with this approach? I shall add tests if this is on the right track
Comment on attachment 8731000 [details]
MozReview Request: Bug 1249029 - Add quality metrics for FxA Verification via Push r=markh

We discussed a slightly different strategy on IRC - clearing the review flag to make things clear.
Attachment #8731000 - Flags: review?(markh)
Comment on attachment 8731000 [details]
MozReview Request: Bug 1249029 - Add quality metrics for FxA Verification via Push r=markh

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/40301/diff/2-3/
Attachment #8731000 - Flags: review?(markh)
Comment on attachment 8731000 [details]
MozReview Request: Bug 1249029 - Add quality metrics for FxA Verification via Push r=markh

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/40301/diff/3-4/
Comment on attachment 8731000 [details]
MozReview Request: Bug 1249029 - Add quality metrics for FxA Verification via Push r=markh

https://reviewboard.mozilla.org/r/40301/#review38027
Attachment #8731000 - Flags: review?(markh) → review+
Thank you Mark!!
https://hg.mozilla.org/mozilla-central/rev/d5d05c652d6b
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla48
Product: Core → Firefox
Target Milestone: mozilla48 → Firefox 48
You need to log in before you can comment on or make changes to this bug.