Closed Bug 1198346 Opened 5 years ago Closed 4 years ago

[Metrics] AddOns Metric collection

Categories

(Firefox OS Graveyard :: General, defect)

ARM
Gonk (Firefox OS)
defect
Not set

Tracking

(feature-b2g:2.5+)

RESOLVED FIXED
feature-b2g 2.5+

People

(Reporter: rdandu, Assigned: rnicoletti)

References

Details

User Story

As Mozilla, I want to know the active Usage of AddOns on FxOS, so that I can determine the engagement and retention of Users.

As Mozilla, I want to know the installation and un-installation of AddOns on FxOS, so that I can track how many users are discovering and acquiring AddOns. 

As FxOS User, I want to be able to choose to share and unshare my Addon usage with Mozilla, so that I can contribute in improving FxOS.

Attachments

(2 files)

Add On Events need to be tracked on FxOS
-Install, Uninstall: Users installing and uninstalling AddOns should be tracked
-Enable,Disable: Enabling and disabling events on Add-ons

These need to be tracked on FirefoxOS, B2GDroid, and Android Ports.
Assignee: nobody → thills
AddOns from outside FxOS Marketplace should not be tracked. This was one of the privacy aspects decided for AppUsage, and should be similar for AddOns.
 Example scenario: A country discriminates against a certain group (eg: Russia for LGBT). That group could host their own marketplace with add-ons which we will not track.
Hi Ravi, based on my testing, we're already recording app usage metrics (e.g., installs and uninstalls) for add-ons. Given that we're treating add-ons the same as apps in terms of app usage metrics, I plan on adding 'enables' and 'disables' to the existing set of metrics. Also, I'm planning on adding an 'addon' field indicating whether or not the 'app' is an add-on. Does this approach meet your requirements?
Assignee: thills → rnicoletti
Status: NEW → ASSIGNED
Flags: needinfo?(rdandu)
QA Contact: rnicoletti
Hi Tamara, I've added 'enables' and 'disables' to the UsageData and a flag indicating whether the 'app' is an addon. I've tested it and it works. What do you think of this approach? Am I missing anything?
Attachment #8655764 - Flags: feedback?(thills)
Comment on attachment 8655764 [details] [diff] [review]
bug-1198346.patch

Hi Russ,

The approach looks good.  Can you check the .merge() function as I believe you will need to add your enables/disables in there when the metrics get merged from a failed attempt.

Also, I would ni to Dominik to see if he is ok with this from a server perspective.  Would be helpful if you can just send him a sample payload so it's clear where the new attributes are.  As a shortcut, just hack the AUM.REPORT_INTERVAL to shorter interval and then set TR.DEBUG= true in telemetry.js and you should see the printout of the payload in console.log

Thanks,

-tamara
Attachment #8655764 - Flags: feedback?(thills) → feedback+
Russ, Tamara, is there a method where QA can find the numbers to track for testing purposes?  I think QA for both Marketplace and B2G will need to have access to the numbers, and stats to test this.
Flags: needinfo?(thills)
Flags: needinfo?(rnicoletti)
Flags: needinfo?(krupa.mozbugs)
Hi Naoki,

If you're looking to test the implementation of this, it would be part of the testing that Shing-Lyu and Gerry Change will do when it lands.

If you're looking for the actual data (e.g. counts for addons that are enabled/disabled, installed/uninstalled), I believe this data will be aggregated with the work that Dominik Strohmeier does on the AppUsage Metrics.  Perhaps it could be part of a dashboard, but I'll let Dominik comment on that.

Thanks,
-tamara
Flags: needinfo?(thills) → needinfo?(dstrohmeier)
feature-b2g: --- → 2.5+
Hi Russ, in response to your comment #5 asking if tracking enables, disables is ok. Here are the comments:
1) 'enables', 'disables', and indicate 'addon' in the packet, so that difference between app and addon can be known on the server. 
2) As you mention, 'installs', 'uinstalls' should be available already with existing app usage framework
3) The AddOns metric should use the same opt-out mechanism as AppUsage (Default On)
4) AddOns from outside FxOS Marketplace should not be tracked (reason described in comment #1)
Flags: needinfo?(rdandu)
User Story: (updated)
> If you're looking for the actual data (e.g. counts for addons that are
> enabled/disabled, installed/uninstalled), I believe this data will be
> aggregated with the work that Dominik Strohmeier does on the AppUsage
> Metrics.  Perhaps it could be part of a dashboard, but I'll let Dominik
> comment on that.

The add-on metrics will be processed similar to apps as they report into the same AppUsage database. We are currently working on getting the dashboards up and running for basic reporting of apps and add-on metrics. More advanced questions will be processed upon request.

Dominik
Flags: needinfo?(dstrohmeier)
Dominik, below is a sample of the payload I'm planning to introduce to support add-on metrics. I've added 'enables', 'disables', and an 'addOn' flag ("true" or "false"). Does it look ok to you?

  "apps":{
    "app://verticalhome.gaiamobile.org/manifest.webapp":{
      "20150902":{
        "usageTime":11,
        "invocations":1,
        "installs":0,
        "uninstalls":0,
        "enables":0,
        "disables":0,
        "activities":{},
        "addOn":false
      }
    },
    "app://addon1.gaiamobile.org/manifest.webapp":{
      "20150902":{
        "usageTime":0,
        "invocations":0,
        "installs":1,
        "uninstalls":0,
        "enables":4,
        "disables":4,
        "activities":{},
        "addOn":true 
      }
    }
  }
Flags: needinfo?(rnicoletti) → needinfo?(dstrohmeier)
Comment on attachment 8656841 [details] [review]
[gaia] russnicoletti:bug-1198346 > mozilla-b2g:master

I've updated the merging code to include 'enables' and 'disables' and I've updated and added tests to the unit tests.

Is there a certain system owner/peer you recommend to review? Vivien?
Attachment #8656841 - Flags: feedback?(thills)
(In reply to Russ Nicoletti [:russn] from comment #9)
> Dominik, below is a sample of the payload I'm planning to introduce to
> support add-on metrics. I've added 'enables', 'disables', and an 'addOn'
> flag ("true" or "false"). Does it look ok to you?

Hi Russ,
looks good to me. This change will allow us to use the same analysis scripts. 

Thanks
Dominik
Flags: needinfo?(dstrohmeier)
Comment on attachment 8656841 [details] [review]
[gaia] russnicoletti:bug-1198346 > mozilla-b2g:master

Hi Marshall, can you have a look at this patch? Thanks.
Attachment #8656841 - Flags: feedback?(thills) → review?(marshall)
How is updating counted?  Is that uninstall/install?  or just install?
Also should we consider the version of the addon?
(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #14)
> How is updating counted?  Is that uninstall/install?  or just install?
> Also should we consider the version of the addon?

Fabrice, does the app updating process trigger the app mgmt `uninstall` and `install` events?
Flags: needinfo?(fabrice)
No, you don't get uninstall/install events when updating apps. The interesting events to track updates are on the app objects themselves, not on mozApps.mgmt. See http://mxr.mozilla.org/mozilla-central/source/dom/webidl/Apps.webidl#81 for instance.
Flags: needinfo?(fabrice)
Comment on attachment 8656841 [details] [review]
[gaia] russnicoletti:bug-1198346 > mozilla-b2g:master

This looks good to me, but I don't usually review code in applications.js, can you tag someone else for review of that piece?
Attachment #8656841 - Flags: review?(marshall) → review+
Comment on attachment 8656841 [details] [review]
[gaia] russnicoletti:bug-1198346 > mozilla-b2g:master

Hi Fabrice, Marshall has reviewed app_usage_metrics.js, can you review applications.js?
Attachment #8656841 - Flags: review?(fabrice)
Attachment #8656841 - Flags: review?(fabrice) → review+
Master: https://github.com/mozilla-b2g/gaia/commit/f93abe18fa4f60e46e7726947af0bbc587fd8271
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
QA Whiteboard: [COM=Telemetry]
Flags: needinfo?(krupa.mozbugs)
You need to log in before you can comment on or make changes to this bug.