1082189 - signup flow client metrics json, for accounts dashboard

Status: RESOLVED WONTFIX

(Cloud Services :: Operations: Metrics/Monitoring, task)

Description

[Katie Parlante]

For the signup flow, we want to generate 3 measures from the client metrics data, capturing :

1. "Intent to signup"
- the idea is to count all people who show up at the signup page, excluding people who are actually trying to sign in
- kibana query: "(events:screen.signup AND -events:screen.signin) OR (events:screen.signup AND events:screen.confirm)"
- fxa_content_signup_intent.json

2. "Engage with signup"
- count people who succeed, and people who have errors (whether or not they succeeded). The intent is to count all people who engage in the signup flow.
- kibana query: "(events:screen.signup AND events:screen.confirm) OR (events:screen.signup AND events:error* AND -events:screen.signin)"
- fxa_content_signup_engage.json

3. "Signup success"
- count all users who are successful at this part of the flow (create an account)
- kibana query: "events:screen.signup AND events:screen.confirm"
- fxa_content_signup_success.json

Comment 1

[Katie Parlante]

I have this up and running on my local dashboard (whoohoo). The "engage" numbers look too high. I'm expecting:
https://kibana.fxa.us-west-2.prod.mozaws.net/#dashboard/temp/1bBJ1u4qRnKouni-0ZT2Sg

Comment 2

[Katie Parlante]

fwiw, here's a graph with all 3: https://kibana.fxa.us-west-2.prod.mozaws.net/#dashboard/temp/C9GPmoVeTlm1kxuFd4jUHA

Comment 3

[Wesley Dawson [:whd]]

Code has been updated with the fix. When I generate backfill I'll make sure to fix all the days up until tomorrow (when the number should be accurate).

Comment 4

[Wesley Dawson [:whd]]

Backfill has been added. Katie, how does it look now?

Comment 5

[Katie Parlante]

Initial set of graphs up here: https://metrics.services.mozilla.com/accounts-dashboard/flow/

So, the jump in "failure rate" at 33 can be explained by the introduction of a timeout error (1017). Scenarios that counted as "bounces" before 33 show up as "engage & fail" because of this error. We should change the "engage" query to not count this.

In other words, right now the clause: (events:screen.signup AND events:error* AND -events:screen.signin) currently counts error.1017 timeout errors. We don't want to count any scenarios that have *only* the 1017 error. If they have some other error and then timeout, we do want to count them.