Closed Bug 1583525 Opened 2 years ago Closed 6 months ago

Add telemetry for PWA installs

Categories

(Firefox for Android Graveyard :: Web Apps (PWAs), task, P1)

Unspecified
Android

Tracking

(firefox-esr68 verified)

RESOLVED INCOMPLETE
Tracking Status
firefox-esr68 --- verified

People

(Reporter: petru, Assigned: petru)

References

Details

Attachments

(5 files, 1 obsolete file)

We need to get a better insight into how many of our DAUs have a PWA shortcut on their home screen. And how many shortcuts.

We already know:

Knowing exactly the number of PWAs a user has on it's homescreen is possible on Android 25+ using ShortcutManager#getPinnedShortcuts and this data can be added in the core ping.
Unfortunately for lower Android versions querying this data is not possible.

Assignee: nobody → petru.lingurar
Group: mozilla-employee-confidential
CC list accessible: false
Not accessible to reporter

Frank, regarding the core ping structure, is there a specific place where this new data should be placed?
Also, because this data is only available on Android 25+, how should we handle reports for lower versions of Android? Reporting 0 as default would be a false positive. Maybe -1 would allow for proper queries?

Flags: needinfo?(fbertsch)

Petru, lets do this and try to get this out in a dot update.

I think what we want to record is not the event happening, but simply counting the actual number at that point in time using ShortcutManager. it is fine that this only works on 25+. For older versions we can simply skip this.

Will ShortcutManager#getPinnedShortcuts allow us to understand the difference between PWA and regular shortcut? This bug is primarily about understanding PWA so if possible we should not count those shortcuts that are not PWA.

Component: General → Web Apps (PWAs)
Product: Firefox → Firefox for Android
Version: other → unspecified

We could have easily counted the number of times the user has added a PWA to homescreen (supported on all api levels) but then we wouldn't have any way of knowing when the user removed them. So we will always just keep incrementing the value.

ShortcutManager#getPinnedShortcuts allows us to query the shortcuts user currently has on homescreen.
Then we can use the fact that we pin them with a different action - ACTION_WEBAPP for PWAs vs ACTION_HOMESCREEN_SHORTCUT for pinned websites to only log the PWAS the user currently has on his homescreen.

The only way to know exactly how many Fennec installed PWAs the user has on
homescreen is ShortcutManager#getPinnedShortcuts which is only available on
Android >=25.
On Android >=25 we'll send the value from ShortcutManager#getPinnedShortcuts,
On Android <25 we'll send a default value of -1.

As this tickets adds to the the previous "fennec" fields which lacked
documentation this patch

  • adds full documentation for all the new core ping fields
  • increments the ping format version

Depends on D55259

The purpose of this ticket was to add yet another field in the new "fennec" object of the core ping.
To have this new field documented I also added the full documentation for the "fennec" object
And also incremented the ping format version.

The "fennec" core ping fields already got a tentative data-review + in https://bugzilla.mozilla.org/show_bug.cgi?id=1579196#c1 but pinging you again now that all implementation is done.

Attachment #9112512 - Attachment is obsolete: true
Attachment #9113695 - Flags: data-review?(liuche)
See Also: → 1579196
Comment on attachment 9113695 [details]
fennec_core_updates_data_review.txt

data-review+

# Data Review Form (to be filled by Data Stewards)

Instructions: Data Stewards will review a request for data collection and endorse responses to each question. **If the request does not provide answers to questions, reviewers give an r- and point to the questions that can’t be answered.**

1) Is there or will there be **documentation** that describes the schema for the ultimate data set in a public, complete, and accurate way? 

Yes, patch included adds fields to https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/data/core-ping.html

2) Is there a control mechanism that allows the user to turn the data collection on and off? 

Yes, can be turned off in fennec telemetry settings

3) If the request is for permanent data collection, is there someone who will monitor the data over time?
Yes, Ashley and Vesta

4) Using the **[category system of data types](https://wiki.mozilla.org/Firefox/Data_Collection)** on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Type 2, and Type 3 for addons, but this has been approved for short-term collection by Alicia Gray, until Fennec EOL, in parity with Desktop addon telemetry.

5) Is the data collection request for default-on or default-off?

Default on

6) Does the instrumentation include the addition of **any *new* identifiers** (whether anonymous or otherwise; e.g., username, random IDs, etc.  See the appendix for more details)?
No

7) Is the data collection covered by the existing Firefox privacy notice? 

Yes

8) Does there need to be a check-in in the future to determine whether to renew the data? (Yes/No) (If yes, set a todo reminder or file a bug if appropriate)**

Permanent collection, but Fennec will EOL by June 2020, so collection will stop by then at the latest.

9) Does the data collection use a third-party collection tool? **If yes, escalate to legal.**

No
Flags: needinfo?(liuche)
Attachment #9113695 - Flags: data-review?(liuche) → data-review+

Comment on attachment 9112508 [details]
Bug 1583525 - Send the current pwas number in the core ping; r?AndreiLazar

ESR Uplift Approval Request

  • If this is not a sec:{high,crit} bug, please state case for ESR consideration: New telemetry that would improve our understanding on the most used features on Fennec.
    Useful for prioritizing Fenix work.
  • User impact if declined: No useful insights regarding pwas usages that would help build a better experience on Fenix.
  • Fix Landed on Version:
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): Small code changes.
  • String or UUID changes made by this patch: --
Attachment #9112508 - Flags: approval-mozilla-esr68?
Attachment #9112509 - Flags: approval-mozilla-esr68?
Attachment #9112510 - Flags: approval-mozilla-esr68?

Comment on attachment 9112508 [details]
Bug 1583525 - Send the current pwas number in the core ping; r?AndreiLazar

More Fennec usage Telemetry needed for informing Fenix migration priorities. Approved for 68.4b2.

Attachment #9112508 - Flags: approval-mozilla-esr68? → approval-mozilla-esr68+
Attachment #9112509 - Flags: approval-mozilla-esr68? → approval-mozilla-esr68+
Attachment #9112510 - Flags: approval-mozilla-esr68? → approval-mozilla-esr68+
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED

Thanks for updating the docs! Can you also land the changes to core-ping.rst to mozilla-central so that the docs are visible at https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/data/core-ping.html?

Flags: needinfo?(petru.lingurar)
Flags: needinfo?(petru.lingurar) → qe-verify+

(In reply to Tim Smith 👨‍🔬 [:tdsmith] from comment #13)

Thanks for updating the docs! Can you also land the changes to core-ping.rst to mozilla-central so that the docs are visible at https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/data/core-ping.html?

Ryan, can you help merge just the second patch - "Bug 1583525 - Document the new core ping "fennec" fields; r?AndreiLazar" to m-c, or should I file a new ticket?

Flags: needinfo?(ryanvm)
Flags: needinfo?(ryanvm)
Pushed by ryanvm@gmail.com:
https://hg.mozilla.org/integration/autoland/rev/6c3a79522b8d
Document the new core ping "fennec" fields. r=AndreiLazar

Hi, checked core ping "current_pwas_count" based on the provided information from developer related for the ping from: https://phabricator.services.mozilla.com/D55260 - line 137 with One Plus 6T/A6013 (Android 9), Samsung Galaxy S9 (Android 8), Nexus 9 (Android 7.1.1) and Sony Xperia Z5 (Android 7) on Firefox Beta 68.4b2

  • the ping count increments with each new pwa app added for devices with android api 28, android api 26
  • for android api 24(Sony Xperia Z5-Android 7.0) the value is set to be -1
  • for devices with android api 25 (Nexus 9 (Android 7.1.1)) there is no incrementation the value is always 0. From documentation - "As Android APIs for querying this are only available on Android >=25" now it's only Android >25 it seems.

Notes:

  • The testing was done through redash
  • The ping "current_pwas_count" is present in column "additional properties" and not on "fennec.page_options.current_pwas_count"
  • The testing was done with android api 28, android api 26, android api 25 and android api 24.

@Frank, can you please take a look at the below questions.

  • Shouldn't we apply this ping based on the schema that can be identified on line 440 (fennec.page_options.current_pwas_count)?
  • This ping does not need to be reset when a new ping is sent?
  • In case of update of pwa's eliminating them should a decrementation action be added?

Thank you

Thanks Diana for testing! Great findings!

Regarding your last 3 questions:

  1. Indeed, I think the Redash schema needs to be updated to account for the new field. It's great that in the meantime the values do get to Redash though.
  2. The goal of this data is to reflect the current number of pwas the user has on the screen at the moment the core ping is sent.
    For this we query the system. This value cannot be "resetted". It should always exactly match what the user has on screen.
  3. Doesn't a decrementation happen?
    Again, we are querying the system for the current value. This should automatically be kept in sync for us by the system.

Also, great finding on that api 25 device - "for devices with android api 25 (Nexus 9 (Android 7.1.1)) there is no incrementation the value is always 0"
Will look into that.

Hey Petru, apologies for the late response.

We do need to update the schema, I'll take care of that. We'll add the new field and it should be available tomorrow.

Flags: needinfo?(fbertsch)

Hi,
From comment17:

  • For point 1 we will check with Frank on this matter, I did rechecked after: fennec.page_options.current_pwas_count, based on comment18, but for the new ping the schema is still not applied in redash.
    I will put a NI on Frank. Any news?
  • For point 2 - understood, no reset will be done
  • For point 3 - I checked the update of the value based on the existence/non-existence of the pwa on screen and it works correctly.

Testing Environment:
Device: Samsung Galaxy S10 Plus (Android 9) - android api 28
Build: Firefox Beta 68.4b2

Flags: needinfo?(fbertsch)

Hi, field "fennec.page_options.current_pwas_count" has been added to the schema, checked with Google Pixel 3 XL (Android 9) - android api 28 and Samsung Galaxy S8+ (Android 8.0.0) - android api 26 on Firefox Beta 68.4b3.
Scenarios:

  • snake_case is applied
  • the default value is displayed correctly
  • the field updates correctly added pwa's for the mentioned devices
  • the field updates correctly when removing the pwa's
    I will remove the NI.
Flags: needinfo?(fbertsch)

Does anything need to happen here or can this bug's status for esr68 be set back to fixed?

Flags: needinfo?(diana.rus)

Hi, yes based on Comment16 the ping for pwa's count is not updated correctly:

  • for devices with android api 25 (Nexus 9 (Android 7.1.1)) there is no incrementation the value is always 0. From documentation - "As Android APIs for querying this are only available on Android >=25".
    Remained as for developer to look into it, as per Comment17.
Flags: needinfo?(diana.rus)
Status: RESOLVED → REOPENED
Flags: needinfo?(petru.lingurar)
Resolution: FIXED → ---

I checked this also on a Sony Z5 with Android 7.1.1 and saw that indeed it reports 0
although as per the official documentation we should be able to query the pinned shortcuts starting with this Android version
https://developer.android.com/reference/android/content/pm/ShortcutManager.html#getPinnedShortcuts()

Seems to be a bug in the Android 7.1.1 framework. I also asked on SO about this.
If the system call fails to return the right number we might also skip it from reporting the number of installed pwas.
Since this would leave us with only 4 versions of Android that could report the right value (API 26-29) I'd like to investigate a bit more.

After looking into this more it seems that Android's ShortcutManager.html#getPinnedShortcuts() always returning an empty list on API level 25 devices is probably a bug in the framework.

To mitigate false-positives on this API level I've created bug 1606820.

Status: REOPENED → RESOLVED
Closed: 2 years ago2 years ago
Depends on: 1606820
Flags: needinfo?(petru.lingurar)
Resolution: --- → FIXED

Didn't intend for this to be closed atm.

Status: RESOLVED → REOPENED
Resolution: FIXED → ---

Hi, verified as fixed on Google Pixel 3a (Android 9) - api level 28, Samsung Galaxy S9 (Android 8.0.0) - android api 26 and Sony Xperia Z5 (Android 7.0.0) - api level 24, on Firefox RC 68.4.0 from playstore.

Comment on attachment 9112508 [details]
Bug 1583525 - Send the current pwas number in the core ping; r?AndreiLazar

Clearing the approval flags to get this off the needs-uplift radar.

Attachment #9112508 - Flags: approval-mozilla-esr68+
Attachment #9112509 - Flags: approval-mozilla-esr68+
Attachment #9112510 - Flags: approval-mozilla-esr68+

Hi, based on Comment10Bug1606820, I will mark this as verified as the ping has been tested on RC and Beta for multiple api's.

Flags: qe-verify+
We have completed our launch of our new Firefox on Android. The development of the new versions use GitHub for issue tracking. If the bug report still reproduces in a current version of [Firefox on Android nightly](https://play.google.com/store/apps/details?id=org.mozilla.fenix) an issue can be reported at the [Fenix GitHub project](https://github.com/mozilla-mobile/fenix/). If you want to discuss your report please use [Mozilla's chat](https://wiki.mozilla.org/Matrix#Connect_to_Matrix) server https://chat.mozilla.org and join the [#fenix](https://chat.mozilla.org/#/room/#fenix:mozilla.org) channel.
Status: REOPENED → RESOLVED
Closed: 2 years ago6 months ago
Resolution: --- → INCOMPLETE
Product: Firefox for Android → Firefox for Android Graveyard
You need to log in before you can comment on or make changes to this bug.