1183234 - Telemetry probe for usage of about:logins UI interactions

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: Logins, Passwords and Form Fill, defect)

Comment 1

[:Margaret Leibovic]

We already have UI telemetry for settings and menu items, so I don't actually know that we need a new probe for this... this probe would only capture the case where users directly type "about:logins" but I think that would be an edge case.

Comment 2

[Barbara Bermes [:barbara] - NI please!]

I think this came up after we've talked to capture how long users might stay on this page, i.e. to verify the performance issue? 

How can we track the event click "Menu->Tools->Logins"? I thought that would bring them to about:logins?

Either way, I feel we need to get more info around how users understand this tool, so can I ask for the sky :)?

- How many times users click on any of the login menu items, how many times do people "edit" a password, or "show the password" etc.
- Of course, also I'd like to know the amount of passwords per user (or average). If see see this increase over time, it will be an indicator that they feel comfortable with the feature and know how to use it. If we only see people going to the feature without doing anything or lets say only save 1-2 passwords per user, it probably means they don't see a value in the tool, or have issues using it.

Comment 3

[:Margaret Leibovic]

(In reply to Barbara Bermes [:bbermes] from comment #2)
> I think this came up after we've talked to capture how long users might stay
> on this page, i.e. to verify the performance issue? 

Ah, I was just confused because there wasn't any context in this bug. I thought that it was just about a probe for "how often do users use this page?".

> How can we track the event click "Menu->Tools->Logins"? I thought that would
> bring them to about:logins?

Yes. This should be covered by our menu UI telemetry. You can see this in mfinkle's UI telemetry dashboard if you search for "logins":
https://www.dropbox.com/s/3izxcs8j2412chi/Screenshot%202015-07-16%2015.14.23.PNG?dl=0

> Either way, I feel we need to get more info around how users understand this
> tool, so can I ask for the sky :)?
>
> - How many times users click on any of the login menu items, how many times
> do people "edit" a password, or "show the password" etc.

Yes, we can add a new UI telemetry probe for this.

> - Of course, also I'd like to know the amount of passwords per user (or
> average). If see see this increase over time, it will be an indicator that
> they feel comfortable with the feature and know how to use it. If we only
> see people going to the feature without doing anything or lets say only save
> 1-2 passwords per user, it probably means they don't see a value in the
> tool, or have issues using it.

We do already have histogram data for this:
http://telemetry.mozilla.org/dist.html#measure=PWMGR_NUM_SAVED_PASSWORDS&max_channel_version=nightly%2F41&min_channel_version=nightly%2F38&product=Fennec&cumulative=0&use_submission_date=0&start_date=2015-05-11&end_date=2015-06-29&sanitize=1

Comment 4

[Barbara Bermes [:barbara] - NI please!]

(In reply to :Margaret Leibovic from comment #3)

> > - How many times users click on any of the login menu items, how many times
> > do people "edit" a password, or "show the password" etc.
> 
> Yes, we can add a new UI telemetry probe for this.
Great, thanks -- should I create a new bug for this?

> We do already have histogram data for this:
> http://telemetry.mozilla.org/dist.
> html#measure=PWMGR_NUM_SAVED_PASSWORDS&max_channel_version=nightly%2F41&min_c
> hannel_version=nightly%2F38&product=Fennec&cumulative=0&use_submission_date=0
> &start_date=2015-05-11&end_date=2015-06-29&sanitize=1

wonderful, this will help us find out if we can move the needle, i.e. define adaptability.

Comment 5

[:Margaret Leibovic]

(In reply to Barbara Bermes [:bbermes] from comment #4)
> (In reply to :Margaret Leibovic from comment #3)
> 
> > > - How many times users click on any of the login menu items, how many times
> > > do people "edit" a password, or "show the password" etc.
> > 
> > Yes, we can add a new UI telemetry probe for this.
> Great, thanks -- should I create a new bug for this?

Since we already seem to have probes for the other items mentioned in this bug, let's just make this bug about probes for the UI of about:logins.

At the very least, let's get a UI telemetry probe for the actions that appear when you hit a login. We can follow existing patterns for context menu probes for this.

We could also get a probe to see how often users actually change their password when they go to the edit login page. It could also be interesting to know how often they hit the "show/hide" button.

Ally, feel free to chime in with other ideas, since you're the one who originally filed this bug.

Comment 6

[Barbara Bermes [:barbara] - NI please!]

sounds good and I like your suggestions as well

Comment 7

[Allison Naaktgeboren :ally]

Since UI telemetry's dashboard has been down for a quite a while and will not be up for awhile and UITelemetry lacks the control of the expiration in version it is thought that a stand alone probe will do better for our usage here.  

Margaret, settings -> privacy -> Manage Logins drops you into aboutLogins.js/xhtml

Expiration in Firefox 55 should be enough data & then some since this is rolling out in 42.

Comment 8

[Allison Naaktgeboren :ally]

Attachment: isAboutLoginsUsedAtAll

Vlad, the goal of this probe is to ascertain if about:logins and its edit login page are actually used in the in wild, and perhaps what parts they might use most.

I don't know how you feel about instrumenting the context menu until we know that the page(s) itself are used and worth investment.

this probe tracks how the list view is shown, how often the edit view is shown, how often the logins are edited, and how often people use the show/hide button on edit logins page. I'm undecided if the last one is really worth measuring at this time.   

Thoughts ?

Comment 9

[Mark Finkle (:mfinkle) (use needinfo?)]

Comment on attachment 8636851 [details] [diff] [review]
isAboutLoginsUsedAtAll

We don't usually see UI elements being probed in such a way (using histograms). For UI events and sessions like this, we usually use some UI Telemetry.

Comment 10

[Vladan Djeric (:vladan)]

Comment on attachment 8636851 [details] [diff] [review]
isAboutLoginsUsedAtAll

Review of attachment 8636851 [details] [diff] [review]:
-----------------------------------------------------------------

::: mobile/android/chrome/content/aboutLogins.js
@@ +198,5 @@
>        headerText.textContent = gStringBundle.GetStringFromName("editLogin.fallbackTitle");
>      }
>    },
>    _onSaveEditLogin: function() {
> +    Services.telemetry.getHistogramById("PWMGR_ABOUT_LOGINS_USAGE").add(LOGIN_EDITED);

do you want to add a sample every time a login is edited in the same session or limit to one-sample-per-session? it changes how the resulting histogram is interpreted

::: toolkit/components/telemetry/Histograms.json
@@ +8173,5 @@
>    },
> +  "PWMGR_ABOUT_LOGINS_USAGE": {
> +    "expires_in_version": "55",
> +    "kind": "enumerated",
> +    "n_values": 4,

you might want to give n_values room to grow. otherwise, you'll have to declare a new histogram if you need to add enum values in the future

Comment 11

[Vladan Djeric (:vladan)]

The resulting histogram will show you the relative popularity of each of the about:logins elements, and it will also provide absolute counts for how many times each was used (by the opt-in Telemetry population).

Is this what you wanted?

Comment 12

[:Margaret Leibovic]

(In reply to Allison Naaktgeboren please NEEDINFO? :ally from comment #7)
> Since UI telemetry's dashboard has been down for a quite a while and will
> not be up for awhile and UITelemetry lacks the control of the expiration in
> version it is thought that a stand alone probe will do better for our usage
> here.  

For mobile UI telemetry, I usually just use this: https://people.mozilla.org/~mfinkle/uitelemetry/

Comment 13

[Barbara Bermes [:barbara] - NI please!]

(In reply to Vladan Djeric (:vladan) -- please needinfo! from comment #11)
> The resulting histogram will show you the relative popularity of each of the
> about:logins elements, and it will also provide absolute counts for how many
> times each was used (by the opt-in Telemetry population).
> 
> Is this what you wanted?

Great, yes. Thanks

Comment 14

[:Margaret Leibovic]

What's left to do here? Can we just address the feedback from comment 10 and land this?

Comment 15

[Allison Naaktgeboren :ally]

(In reply to Vladan Djeric (:vladan) -- please needinfo! from comment #10)
> Comment on attachment 8636851 [details] [diff] [review]
> isAboutLoginsUsedAtAll
> 
> Review of attachment 8636851 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: mobile/android/chrome/content/aboutLogins.js
> @@ +198,5 @@
> >        headerText.textContent = gStringBundle.GetStringFromName("editLogin.fallbackTitle");
> >      }
> >    },
> >    _onSaveEditLogin: function() {
> > +    Services.telemetry.getHistogramById("PWMGR_ABOUT_LOGINS_USAGE").add(LOGIN_EDITED);
> 
> do you want to add a sample every time a login is edited in the same session
> or limit to one-sample-per-session? it changes how the resulting histogram
> is interpreted
> 
I believe the idea is to know how often users edit any passwords in each session. So I believe that would be the former.

Yes we can do that. Its currently tabled behind some more kidfox work since I can do that in teh window between aug 5 and 10th.

Comment 16

[Pulsebot]

https://hg.mozilla.org/integration/fx-team/rev/3e37f8f83299

Comment 17

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/3e37f8f83299