1656355 - Add a histogram indicating how many times each stored card has been filled, ever.

Status: VERIFIED FIXED

(Toolkit :: Form Autofill, enhancement, P2)

Description

[Zibi Braniecki [:zbraniecki][:gandalf]]

Product would like to have a histogram indicating how many times each stored card has been filled, ever.

Example:

if I have 3 cards saved that have been used 10 times, and 1 card that has been used 2 times, then the bin including 10 should have a value of 3, and the bin including 2 should have a value of 1. the total sum of the histogram's values should therefore be equal to the number of cards saved, and when multiplied by the bin values should tell us how many times any card has ever been used.

Each value should be a number of card numbers with that usage amount

Comment 1

[Zibi Braniecki [:zbraniecki][:gandalf]]

I'm trying to wrap my head around this.

If I'm not mistaken, the way to approach it is to reset the histogram on each use of the credit card and repopulate it with new values, per card, no?
if I have 3 cards with uses 3/5/10: [card1: 3, card2: 5, card3: 10] then I want my histogram to show 1 for 3, 1 for 5 and 1 for 10.

if then the user uses the first card twice, I want to end up with 2 for 5 and 1 for 10.

so I need to repopulate the given histogram key on each use accumulating for number of cards that have the given number of uses, am I correct?

Comment 2

[Zibi Braniecki [:zbraniecki][:gandalf]]

Histograms don't really cater to our needs here :(

I'm trying to wrap my head around how to store such data.

Option 1

Install a number of scalar probes named formautofill.creditCards.useCount_card[1-x] where x is some arbitrary number.
Store use count per card's position (so, if someone "shuffles" cards and puts a different one on top, the counter for card1 continues to increase, but now is for a different card).

The upside of that is that assuming users very rarely shuffle cards (there's no UI for that!) we protect users privacy, while learning if users repeatedly use one card, or multiple cards equally.
The downside is that since order of cards is a bit arbitrary, one user may use card1 mostly while the other will use card2 mostly and any accumulative statistics will be impossible.

Option 2

We sort cards by use popularity and then use either the scalars like above or keyed historiogram to send information which states something like:
The most popular card was used 24 times, The second most popular was used 10 times

The benefit here is that we are removing the co-factor of randomness of card order and rely on card use frequency, which should be the same for most questions around this probe that I can imagine.

======

Generally speaking, I think we need to decide what questions we need to answer before we can decide on what (imperfect) way we can use to gather the data.

Leif - can you help us here?

Comment 3

[Leif Oines [:loines]]

Ideally, we want the measurement to allow us to answer three questions:

1. How many cards are stored? In the original spec this would be conveyed by the sum of the histogram's values (height of the bins).
2. How many times the cards that are stored have been used (this was conveyed by the value of the bin labels).
3. How many total uses there has been across all cards (sum-product of 1 and 2).

We don't care about having a stable ID of each card over time, we just want to know how many cards there are, and among all cards, how often they were used. For example, if usage bin 2 had 4 cards (value of 4) and bin 3 had 0 cards ( histogram looks like { 2:4, 3:0 } ) and one card is used once, then we now have { 2:3, 3:1 }. We don't care about which card moved over to bin 3.

In this scheme, we know how many cards are stored (sum of values: 4), how many times the cards were used (3 cards were used twice, once card was used three times), and how many total usages there were (sum of the products, (2*3) + (3*1) = 9).

Here is an example of a histogram that works in a similar way. IIRC the bin labels here are number of days and the values are the number of logins that were used that many days ago.

We could break up the measurements. For example, have a scalar for number of cards stored, another scalar for total uses across all cards. That would cover 1 and 3. But it won't describe the distribution of card usage (2). We would like to have that, since we may want to know if users are storing a lot of cards and using only a handful frequently, or spreading their usage across all of them.

I think you are right that you would have to re-write the histogram values after each card usage (you can't just increment after each usage). LMK if that causes problems. I am also OK if we don't handle card deletion. We want the full usage history, so I'm ok if cards continue to count to the histogram even after they are removed (they will just be stuck in one bin forever). If they need to drop out of the histogram, I guess that's OK too.

I'm trying to get my head around if option 1 would cover 1-3 above. I think it might, but I could use your opinion there. At first glace it seems a bit more cumbersome to have many scalars rather than a single histogram, but if there are technical reasons to do so let me know. I think option 2 might not cover (1) in the case of a tie in usage.

Hopefully that made things more clear. I could be missing something - let me know.

Comment 4

[Zibi Braniecki [:zbraniecki][:gandalf]]

Ah! Thank you!

This is what I needed:
https://searchfox.org/mozilla-central/source/toolkit/components/passwordmgr/LoginManager.jsm#165

I think we should be able to handle it with histogram now! Let me try!

Comment 5

[Zibi Braniecki [:zbraniecki][:gandalf]]

Attachment: Bug 1656355 - Add a histogram indicating how many times each stored card has been filled. r?abr

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3a42daad56e2
Add a histogram indicating how many times each stored card has been filled. r=abr

Comment 7

[Zibi Braniecki [:zbraniecki][:gandalf]]

Comment on attachment 9167763 [details]
Bug 1656355 - Add a histogram indicating how many times each stored card has been filled. r?abr

Beta/Release Uplift Approval Request

User impact if declined: None
Is this code covered by automated tests?: Yes
Has the fix been verified in Nightly?: Yes
Needs manual test from QE?: No
If yes, steps to reproduce:
List of other uplifts needed:
Risk to taking this patch: Low
Why is the change risky/not risky? (and alternatives if risky): It only adds telemetry
String changes made/needed: None

Comment 8

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/3a42daad56e2

Comment 9

[Adam Roach [:abr]]

Attachment: Bug 1656355: Move histogram counting to CreditCards r=zbraniecki!

Comment 10

[Julien Cristau [:jcristau]]

Comment on attachment 9167763 [details]
Bug 1656355 - Add a histogram indicating how many times each stored card has been filled. r?abr

approved for 80.0b5

Comment 11

[Julien Cristau [:jcristau]]

https://hg.mozilla.org/releases/mozilla-beta/rev/4d47e2c820d8

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9c793e797070
Move histogram counting to CreditCards r=zbraniecki

Comment 13

[Adam Roach [:abr]]

Comment on attachment 9168350 [details]
Bug 1656355: Move histogram counting to CreditCards r=zbraniecki!

Beta/Release Uplift Approval Request

• User impact if declined: None; however, our telemetry histogram will be incorrect, as it will count street addresses as well as credit cards in its buckets.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This is the same code as the other patch on this bug, but moved to the CreditCard specific class instead of the parent class from which CreditCard and Address are jointly derived.
• String changes made/needed:

Comment 14

[Julien Cristau [:jcristau]]

Comment on attachment 9168350 [details]
Bug 1656355: Move histogram counting to CreditCards r=zbraniecki!

approved for 80.0b5

Comment 15

[Julien Cristau [:jcristau]]

https://hg.mozilla.org/releases/mozilla-beta/rev/c6cf97d251e5

Comment 16

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/9c793e797070

Comment 17

[Maria Heres, inactive, contact Ecosystem QA]

We have verified the CREDITCARD_NUM_USES histogram that records the number of times the saved credit cards have been used on Firefox 80.0b5 (Build ID: 20200806203447) and the latest Nightly 81.0a1 build (Build ID: 20200806215439) on Windows 10 x64, Mac 10.14.6 and Ubuntu 20.04 x64 and Linux Mint 19.2.
In order to verify this we have used the Credit Card Autofill Metrics Spec document.
The histogram records the number of uses on the x-axis and the number of cards used on the y-axis. Saved cards that have not yet been used are recorded in the 0 column.
The histogram also displays 2 additional columns both having y values of 0: the smallest number of uses - 1, the largest number of uses +1.