1822119 - Store new UUID in shared prefs and send in metrics ping

Status: VERIFIED FIXED

(Fenix :: Experimentation and Telemetry, enhancement)

Description

[Raphael Aurich [:raphael] UTC+02:00]

Background

Recent data analysis suggests that some Firefox on Android clients regenerate their Glean client ID. We would like to better understand the technical differences between storage mechanisms in Android, specifically the Glean DB and Shared Preferences.

Plan

1. Define a new UUID metric in Glean with lifetime = ping and expires = never
   1. Call this metric shared_prefs_UUID
   2. Send this new metric in the metrics ping
2. On first run, app start or when the app becomes active
   1. Attempt to load the value for key shared_prefs_UUID from shared preferences
   2. If it does not exist, generate it in the Fenix application layer and save it to the shared prefs
3. Every time the app becomes active
   1. Set the value for the shared_prefs_UUID Glean metric to the value save to the shared prefs

Analysis

1. We will have <client_id, shared_prefs_uuid> pairs in the metrics ping
2. Questions
   1. Do we see a single shared_pref_uuid with multiple client_ids? If so, this is indicative of client ID regeneration that shared pref storage is immune to
   2. Do we see a single client_id with multiple shared_prefs_uuids? If so, shared prefs are a more unsafe to store this data than the Glean DB (filesystem)
   3. If they match exactly, then the Glean DB and shared prefs are getting reset at the same times (this may be expected behavior)

Comment 1

[Raphael Aurich [:raphael] UTC+02:00]

Update on where we are with this change:

• We were hoping to set the value for the shared_prefs_UUID Glean metric to the value saved to the shared prefs every time the app becomes active. If we only send this metric via the metrics ping and use a metric lifetime of ping, then new UUID values generated during the measurement window of the metrics ping will overwrite the previous one. Which means we will only have access to the most recent value and won't know how often it changed during the measurement window.
• I looked into sending the shared_prefs_UUID in the baseline ping since that is closer in level of granularity to what we're trying to measure (update on app becoming active). However the baseline ping is intended for activity measurement so the Glean SDK team discourages putting additional metrics there (glean-parser throws an error ERROR: BASELINE_PING: metrics.shared_prefs_uuid: The baseline ping is Glean-internal. Remove 'baseline' from the send_in_pings array. The Glean SDK team is open to discussing the option of adding it there, but this requires more thought and discussion, so doesn't help in the near-term.)
• Frank and I figured we can add a new counter metric for the number of shared_prefs_UUID values seen during the measurement window (increment if the Shared Prefs value does not match the Glean metric value). However due to the write-only philosophy of the Glean SDK reading metric values is not supported (only in test environments using .testGetValue()).
• Travis and Chutten suggested sending a custom ping on the schedule we need. This is what we do with the client-deduplication ping, but it runs behind a Nimbus feature flag and is tied to an experiment. So this would need to be a new custom ping.
• For the purpose of the Client ID Regeneration investigation and monitoring the issue in the future, it would be valuable to have the ability to inspect Glean errors for regenerated client IDs. Specifically when rkv detects that the DB file does not exist or is corrupt (see bug 1815253). Glean sends errors in the metrics ping. We might be able to this with a custom ping by joining the custom ping table with the metrics ping table.

Comment 2

[Raphael Aurich [:raphael] UTC+02:00]

Anything to add to my comment above, team?

Comment 3

[Raphael Aurich [:raphael] UTC+02:00]

Attachment: Request for data collection review form

Comment 4

[Raphael Aurich [:raphael] UTC+02:00]

Attachment: GitHub Pull Request

Comment 5

[Raphael Aurich [:raphael] UTC+02:00]

I've updated my GitHub branch and marked the pull request as Ready for Review.

The sharedPrefsUUID value in Shared Preferences is now reset on telemetry data collection opt-out.

Comment 6

[Chris H-C :chutten]

The comment is congruent with my understanding as well, and I have nothing to add.

Comment 7

[Travis Long [:travis_]]

Comment on attachment 9329330 [details]
Request for data collection review form

Data Review

1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?

Yes, through the metrics.yaml file and the Glean Dictionary.

2. Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, through the "Send Usage Data" preference in the application settings. Per the comment in this bug the shared_prefs_uuid will also be reset along with the client_id.

3. If the request is for permanent data collection, is there someone who will monitor the data over time?

N/A, collection to be removed or renewed by version 116

4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 1, Technical Data.

5. Is the data collection request for default-on or default-off?

Default-on

6. Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

Yes. This identifier doesn't contain any user info, instead it identifies the state of the shared pref storage through a randomly generated UUID.

7. Is the data collection covered by the existing Firefox privacy notice?

Yes

8. Does the data collection use a third-party collection tool?

No

Result

data-review+

Comment 8

[Raphael Aurich [:raphael] UTC+02:00]

Thank you for the review, Travis!

Comment 9

[BMO Github Automation]

Authored by https://github.com/hackebrot
https://github.com/mozilla-mobile/firefox-android/commit/627e5646fb35b2377de13be3ec4ff9347b71fafe
[main] Bug 1822119 - Store new UUID in shared prefs and send in metrics ping (#1541)

Comment 10

[BMO Github Automation]

Attachment: [mozilla-mobile/firefox-android] Bug 1822119 - Store new UUID in shared prefs and send in metrics ping (backport #1541) (#1791)

Comment 11

[Raphael Aurich [:raphael] UTC+02:00]

Hi Krupa, please approve uplift for v113. Thank you!

Comment 12

[krupa raj [:krupa--use this to needinfo]]

I approve the uplift for v113. Thanks all.

Comment 13

[Andrei Mocirean]

Verified as fixed on the latest Nightly (114.0a1) from 2nd of May 2023.
The metrics.shared_prefs_uuid is present in the metrics reports from Glean.
Devices used for testing: Google Pixel 7 Pro (Android 13), Lenovo Tab P11 Plus (Android 12).
Thank you!

Comment 14

[Raphael Aurich [:raphael] UTC+02:00]

Comment on attachment 9330425 [details] [review]
[mozilla-mobile/firefox-android] Bug 1822119 - Store new UUID in shared prefs and send in metrics ping (backport #1541) (#1791)

Beta/Release Uplift Approval Request

• User impact if declined: Analysis of technical differences between Shared Preferences and Glean DB cannot be performed in v113. The sooner we can start collecting this technical data, the sooner we can monitor client ID regeneration on Android and assess whether Shared Preferences storage is immune to data loss.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This change is adding a new Glean metric, setting a new Shared Pref similar to how other prefs are set and is resetting the Shared Pref value in an existing event listener. It is not changing user-facing functionality.
• String changes made/needed:
• Is Android affected?: Yes

Comment 15

[Raphael Aurich [:raphael] UTC+02:00]

Comment on attachment 9330425 [details] [review]
[mozilla-mobile/firefox-android] Bug 1822119 - Store new UUID in shared prefs and send in metrics ping (backport #1541) (#1791)

Beta/Release Uplift Approval Request

• User impact if declined:
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: Yes
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky):
• String changes made/needed:
• Is Android affected?: Yes

Comment 16

[Raphael Aurich [:raphael] UTC+02:00]

Comment on attachment 9330425 [details] [review]
[mozilla-mobile/firefox-android] Bug 1822119 - Store new UUID in shared prefs and send in metrics ping (backport #1541) (#1791)

Beta/Release Uplift Approval Request

• User impact if declined: Analysis of technical differences between Shared Preferences and Glean DB cannot be performed in v113. The sooner we can start collecting this technical data, the sooner we can monitor client ID regeneration on Android and assess whether Shared Preferences storage is immune to data loss.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This change is adding a new Glean metric, setting a new Shared Pref similar to how other prefs are set and is resetting the Shared Pref value in an existing event listener. It is not changing user-facing functionality.
• String changes made/needed:
• Is Android affected?: Yes

Comment 17

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9330425 [details] [review]
[mozilla-mobile/firefox-android] Bug 1822119 - Store new UUID in shared prefs and send in metrics ping (backport #1541) (#1791)

Approved for 113.0rc2.

Comment 18

[BMO Github Automation]

Authored by https://github.com/hackebrot
https://github.com/mozilla-mobile/firefox-android/commit/1fc88b83a9f50dc702ec7432ef698bdbdedeca5f
[releases_v113] Bug 1822119 - Store new UUID in shared prefs and send in metrics ping (#1541)

Comment 19

[Adina Petridean]

Verified as fixed on the RC 113.0 build 2 from 5th of May 2023 with Samsung GalaxyZ Fold 4 (Android 13).
The metrics.shared_prefs_uuid is present in the metrics reports from Glean.