1583525 - Add telemetry for PWA installs

Status: RESOLVED INCOMPLETE

(Firefox for Android Graveyard :: Web Apps (PWAs), task, P1)

Description

[Petru-Mugurel Lingurar [:petru]]

We need to get a better insight into how many of our DAUs have a PWA shortcut on their home screen. And how many shortcuts.

We already know:

• when users add a PWA to homescreen - following bug 1409268 we already track the following user actions
  from which pwa_confirm_added would be sent for every user action of adding a pwa to the homescreen.
• when users open a PWA from the homescreen - following bug 1490670 we know track this with the pwa.1 telemetry event.

Knowing exactly the number of PWAs a user has on it's homescreen is possible on Android 25+ using ShortcutManager#getPinnedShortcuts and this data can be added in the core ping.
Unfortunately for lower Android versions querying this data is not possible.

Comment 1

[Petru-Mugurel Lingurar [:petru]]

Frank, regarding the core ping structure, is there a specific place where this new data should be placed?
Also, because this data is only available on Android 25+, how should we handle reports for lower versions of Android? Reporting 0 as default would be a false positive. Maybe -1 would allow for proper queries?

Comment 2

[Stefan Arentz | :st3fan | ⏰ EST | he/him]

Petru, lets do this and try to get this out in a dot update.

I think what we want to record is not the event happening, but simply counting the actual number at that point in time using ShortcutManager. it is fine that this only works on 25+. For older versions we can simply skip this.

Will ShortcutManager#getPinnedShortcuts allow us to understand the difference between PWA and regular shortcut? This bug is primarily about understanding PWA so if possible we should not count those shortcuts that are not PWA.

Comment 3

[Petru-Mugurel Lingurar [:petru]]

We could have easily counted the number of times the user has added a PWA to homescreen (supported on all api levels) but then we wouldn't have any way of knowing when the user removed them. So we will always just keep incrementing the value.

ShortcutManager#getPinnedShortcuts allows us to query the shortcuts user currently has on homescreen.
Then we can use the fact that we pin them with a different action - ACTION_WEBAPP for PWAs vs ACTION_HOMESCREEN_SHORTCUT for pinned websites to only log the PWAS the user currently has on his homescreen.

Comment 4

[Petru-Mugurel Lingurar [:petru]]

Attachment: Bug 1583525 - Send the current pwas number in the core ping; r?AndreiLazar

The only way to know exactly how many Fennec installed PWAs the user has on
homescreen is ShortcutManager#getPinnedShortcuts which is only available on
Android >=25.
On Android >=25 we'll send the value from ShortcutManager#getPinnedShortcuts,
On Android <25 we'll send a default value of -1.

Comment 5

[Petru-Mugurel Lingurar [:petru]]

Attachment: Bug 1583525 - Document the new core ping "fennec" fields; r?AndreiLazar

As this tickets adds to the the previous "fennec" fields which lacked
documentation this patch

• adds full documentation for all the new core ping fields
• increments the ping format version

Depends on D55259

Comment 6

[Petru-Mugurel Lingurar [:petru]]

Attachment: Bug 1583525 - Remove unused imports in TelemetryCorePingBuilder; r?AndreiLazar

Depends on D55260

Comment 7

[Petru-Mugurel Lingurar [:petru]]

Attachment: fennec_core_updates_data_review.txt

The purpose of this ticket was to add yet another field in the new "fennec" object of the core ping.
To have this new field documented I also added the full documentation for the "fennec" object
And also incremented the ping format version.

The "fennec" core ping fields already got a tentative data-review + in https://bugzilla.mozilla.org/show_bug.cgi?id=1579196#c1 but pinging you again now that all implementation is done.

Comment 8

[Petru-Mugurel Lingurar [:petru]]

Attachment: fennec_core_updates_data_review.txt

The purpose of this ticket was to add yet another field in the new "fennec" object of the core ping.
To have this new field documented I also added the full documentation for the "fennec" object
And also incremented the ping format version.

The "fennec" core ping fields already got a tentative data-review + in https://bugzilla.mozilla.org/show_bug.cgi?id=1579196#c1 but pinging you again now that all implementation is done.

Comment 9

[Chenxia Liu [:liuche]]

Comment on attachment 9113695 [details]
fennec_core_updates_data_review.txt

data-review+

# Data Review Form (to be filled by Data Stewards)

Instructions: Data Stewards will review a request for data collection and endorse responses to each question. **If the request does not provide answers to questions, reviewers give an r- and point to the questions that can’t be answered.**

1) Is there or will there be **documentation** that describes the schema for the ultimate data set in a public, complete, and accurate way? 

Yes, patch included adds fields to https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/data/core-ping.html

2) Is there a control mechanism that allows the user to turn the data collection on and off? 

Yes, can be turned off in fennec telemetry settings

3) If the request is for permanent data collection, is there someone who will monitor the data over time?
Yes, Ashley and Vesta

4) Using the **[category system of data types](https://wiki.mozilla.org/Firefox/Data_Collection)** on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Type 2, and Type 3 for addons, but this has been approved for short-term collection by Alicia Gray, until Fennec EOL, in parity with Desktop addon telemetry.

5) Is the data collection request for default-on or default-off?

Default on

6) Does the instrumentation include the addition of **any *new* identifiers** (whether anonymous or otherwise; e.g., username, random IDs, etc.  See the appendix for more details)?
No

7) Is the data collection covered by the existing Firefox privacy notice? 

Yes

8) Does there need to be a check-in in the future to determine whether to renew the data? (Yes/No) (If yes, set a todo reminder or file a bug if appropriate)**

Permanent collection, but Fennec will EOL by June 2020, so collection will stop by then at the latest.

9) Does the data collection use a third-party collection tool? **If yes, escalate to legal.**

No

Comment 10

[Petru-Mugurel Lingurar [:petru]]

Comment on attachment 9112508 [details]
Bug 1583525 - Send the current pwas number in the core ping; r?AndreiLazar

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: New telemetry that would improve our understanding on the most used features on Fennec.
  Useful for prioritizing Fenix work.
• User impact if declined: No useful insights regarding pwas usages that would help build a better experience on Fenix.
• Fix Landed on Version:
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Small code changes.
• String or UUID changes made by this patch: --

Comment 11

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9112508 [details]
Bug 1583525 - Send the current pwas number in the core ping; r?AndreiLazar

More Fennec usage Telemetry needed for informing Fenix migration priorities. Approved for 68.4b2.

Comment 12

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr68/rev/3bebea25e867
https://hg.mozilla.org/releases/mozilla-esr68/rev/219f650b2dfc
https://hg.mozilla.org/releases/mozilla-esr68/rev/eb49d907115b

Comment 13

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Thanks for updating the docs! Can you also land the changes to core-ping.rst to mozilla-central so that the docs are visible at https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/data/core-ping.html?

Comment 14

[Petru-Mugurel Lingurar [:petru]]

(In reply to Tim Smith 👨‍🔬 [:tdsmith] from comment #13)

Thanks for updating the docs! Can you also land the changes to core-ping.rst to mozilla-central so that the docs are visible at https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/data/core-ping.html?

Ryan, can you help merge just the second patch - "Bug 1583525 - Document the new core ping "fennec" fields; r?AndreiLazar" to m-c, or should I file a new ticket?

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6c3a79522b8d
Document the new core ping "fennec" fields. r=AndreiLazar

Comment 16

[Diana Rus]

Hi, checked core ping "current_pwas_count" based on the provided information from developer related for the ping from: https://phabricator.services.mozilla.com/D55260 - line 137 with One Plus 6T/A6013 (Android 9), Samsung Galaxy S9 (Android 8), Nexus 9 (Android 7.1.1) and Sony Xperia Z5 (Android 7) on Firefox Beta 68.4b2

• the ping count increments with each new pwa app added for devices with android api 28, android api 26
• for android api 24(Sony Xperia Z5-Android 7.0) the value is set to be -1
• for devices with android api 25 (Nexus 9 (Android 7.1.1)) there is no incrementation the value is always 0. From documentation - "As Android APIs for querying this are only available on Android >=25" now it's only Android >25 it seems.

Notes:

• The testing was done through redash
• The ping "current_pwas_count" is present in column "additional properties" and not on "fennec.page_options.current_pwas_count"
• The testing was done with android api 28, android api 26, android api 25 and android api 24.

@Frank, can you please take a look at the below questions.

• Shouldn't we apply this ping based on the schema that can be identified on line 440 (fennec.page_options.current_pwas_count)?
• This ping does not need to be reset when a new ping is sent?
• In case of update of pwa's eliminating them should a decrementation action be added?

Thank you

Comment 17

[Petru-Mugurel Lingurar [:petru]]

Thanks Diana for testing! Great findings!

Regarding your last 3 questions:

1. Indeed, I think the Redash schema needs to be updated to account for the new field. It's great that in the meantime the values do get to Redash though.
2. The goal of this data is to reflect the current number of pwas the user has on the screen at the moment the core ping is sent.
   For this we query the system. This value cannot be "resetted". It should always exactly match what the user has on screen.
3. Doesn't a decrementation happen?
   Again, we are querying the system for the current value. This should automatically be kept in sync for us by the system.

Also, great finding on that api 25 device - "for devices with android api 25 (Nexus 9 (Android 7.1.1)) there is no incrementation the value is always 0"
Will look into that.

Comment 18

[Frank Bertsch [:frank]]

Hey Petru, apologies for the late response.

We do need to update the schema, I'll take care of that. We'll add the new field and it should be available tomorrow.

Comment 19

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla-services/mozilla-pipeline-schemas/pull/464

Comment 20

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/6c3a79522b8d

Comment 21

[Diana Rus]

Hi,
From comment17:

• For point 1 we will check with Frank on this matter, I did rechecked after: fennec.page_options.current_pwas_count, based on comment18, but for the new ping the schema is still not applied in redash.
  I will put a NI on Frank. Any news?
• For point 2 - understood, no reset will be done
• For point 3 - I checked the update of the value based on the existence/non-existence of the pwa on screen and it works correctly.

Testing Environment:
Device: Samsung Galaxy S10 Plus (Android 9) - android api 28
Build: Firefox Beta 68.4b2

Comment 22

[Diana Rus]

Hi, field "fennec.page_options.current_pwas_count" has been added to the schema, checked with Google Pixel 3 XL (Android 9) - android api 28 and Samsung Galaxy S8+ (Android 8.0.0) - android api 26 on Firefox Beta 68.4b3.
Scenarios:

• snake_case is applied
• the default value is displayed correctly
• the field updates correctly added pwa's for the mentioned devices
• the field updates correctly when removing the pwa's
  I will remove the NI.

Comment 23

[Julien Cristau [:jcristau]]

Does anything need to happen here or can this bug's status for esr68 be set back to fixed?

Comment 24

[Diana Rus]

Hi, yes based on Comment16 the ping for pwa's count is not updated correctly:

• for devices with android api 25 (Nexus 9 (Android 7.1.1)) there is no incrementation the value is always 0. From documentation - "As Android APIs for querying this are only available on Android >=25".
  Remained as for developer to look into it, as per Comment17.

Comment 25

[Petru-Mugurel Lingurar [:petru]]

I checked this also on a Sony Z5 with Android 7.1.1 and saw that indeed it reports 0
although as per the official documentation we should be able to query the pinned shortcuts starting with this Android version
https://developer.android.com/reference/android/content/pm/ShortcutManager.html#getPinnedShortcuts()

Seems to be a bug in the Android 7.1.1 framework. I also asked on SO about this.
If the system call fails to return the right number we might also skip it from reporting the number of installed pwas.
Since this would leave us with only 4 versions of Android that could report the right value (API 26-29) I'd like to investigate a bit more.

Comment 26

[Petru-Mugurel Lingurar [:petru]]

After looking into this more it seems that Android's ShortcutManager.html#getPinnedShortcuts() always returning an empty list on API level 25 devices is probably a bug in the framework.

To mitigate false-positives on this API level I've created bug 1606820.

Comment 27

[Petru-Mugurel Lingurar [:petru]]

Didn't intend for this to be closed atm.

Comment 28

[Diana Rus]

Hi, verified as fixed on Google Pixel 3a (Android 9) - api level 28, Samsung Galaxy S9 (Android 8.0.0) - android api 26 and Sony Xperia Z5 (Android 7.0.0) - api level 24, on Firefox RC 68.4.0 from playstore.

Comment 29

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9112508 [details]
Bug 1583525 - Send the current pwas number in the core ping; r?AndreiLazar

Clearing the approval flags to get this off the needs-uplift radar.

Comment 30

[Diana Rus]

Hi, based on Comment10Bug1606820, I will mark this as verified as the ping has been tested on RC and Beta for multiple api's.

Comment 31

[BMO Automation]

We have completed our launch of our new Firefox on Android. The development of the new versions use GitHub for issue tracking. If the bug report still reproduces in a current version of [Firefox on Android nightly](https://play.google.com/store/apps/details?id=org.mozilla.fenix) an issue can be reported at the [Fenix GitHub project](https://github.com/mozilla-mobile/fenix/). If you want to discuss your report please use [Mozilla's chat](https://wiki.mozilla.org/Matrix#Connect_to_Matrix) server https://chat.mozilla.org and join the [#fenix](https://chat.mozilla.org/#/room/#fenix:mozilla.org) channel.