1137063 - [Metrics] Usage time is not tracked for apps from the Marketplace

Status: VERIFIED FIXED

(Firefox OS Graveyard :: Gaia::System, defect)

Description

[Shing Lyu [:shinglyu]]

*** Description
The usage time is always 0 for apps installed from the Marketplace

*** Steps to Reproduce
1. Install an app from the Firefox Marketplace (e.g. La Republica)
2. Open the app after install (using the Marketplace "open" button)
3. Use it for 5 sec.
4. Press the home button to go back to homescreen
5. Uninstall the app
5. Wait for the app usage ping to reach the server

*** Expected Results
The usage data for the app should be:
   "usageTime": 5,
   "installs": 1,
   "uninstalls": 1,

*** Actual Results
The usage time is 0
   "usageTime": 0, //something wrong here
   "installs": 1,
   "uninstalls": 1,

*** Other Notes


*** Reproduction Frequency
100

*** Build
  Gaia-Rev        8e98fe665f3821d10d4d982cbb14cbe5b94d0be5
  Gecko-Rev       https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/62c80c92b39e
  Build-ID        20150215162503
  Version         37.0a2
  Device-Name     flame
  FW-Release      4.4.2
  FW-Incremental  65
  FW-Date         Mon Dec 15 18:51:29 CST 2014
  Bootloader      L1TC000118D0

Comment 1

[Marshall Culpepper [:marshall_law]]

Looking into this today

Comment 2

[Hema Koka [:hema]]

Marshall, any update?

Thanks
Hema

Comment 3

[Hema Koka [:hema]]

Per Marshall's update on 3/9:

He was unable to reproduce it last week. He says he will try reproducing again now that his laptop is back from repairs.

Shing: please retry again on 2.2 to see if you are able to reproduce this. Some of the bugs related to metrics landed in 2.2 last week. 

Thanks
Hema

Comment 4

[Shing Lyu [:shinglyu]]

I'll test again today

Comment 5

[Shing Lyu [:shinglyu]]

Attachment: adb.log

Comment 6

[Shing Lyu [:shinglyu]]

Attachment: ping.txt

Comment 7

[Shing Lyu [:shinglyu]]

I am able to reproduce this on today's 2.2 

Steps:
1. Open Marketplace, find ConnectA2 app
2. Install the app, tap "open" after the "install" button changed to "open"
3. Wait for a few seconds
4. Open the app again
5. Ping transmitted

You can see adb.log for the adb logcat output, and ping.txt for the ping payload. Notice that the usageTime=0.

Version
  Gaia-Rev        4aefc3f6f30a40ac67fdf841b7c90cd648b85369
  Gecko-Rev       https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/3db27b3c9298
  Build-ID        20150312162502
  Version         37.0
  Device-Name     flame
  FW-Release      4.4.2
  FW-Incremental  eng.cltbld.20150310.200728
  FW-Date         Tue Mar 10 20:07:39 EDT 2015
  Bootloader      L1TC000118D0

Comment 8

[Marshall Culpepper [:marshall_law]]

Thanks Shing, I'll try to repro again today

Comment 9

[Marshall Culpepper [:marshall_law]]

I think I was previously looking at only install counts (which are working), not app usage time. I've been able to repro this now with ConnectA2, and am investigating

Comment 10

[Marshall Culpepper [:marshall_law]]

It looks like |e.detail.installOrigin| is undefined for the appopened event, but set properly to https://marketplace.firefox.com for applicationinstall and applicationuninstall events.

What isn't clear to me ATM is why installOrigin would be left undefined for appopened events. It could be a bug, but maybe there is a privacy reason of some kind? Tagging Fabrice here to see if he has any ideas.

Assuming that the behavior is not a bug, I think I have a course of action, but want to air the idea for sanity:

Add an additional check for the app's manifestURL with a prefix of "https://marketplace.firefox.com/app/". Do any apps on Marketplace have non-Marketplace URLs? Here's ConnectA2's manifestURL for example:

https://marketplace.firefox.com/app/956189a1-c985-4824-85a9-de708d38d6f9/manifest.webapp

I'm also going to test to see if the re-querying mozApps.mgmt gives me a proper installOrigin. If it does, that would be another path forward (though it would introduce some async behavior, and be pretty risky)

Comment 11

[Shing Lyu [:shinglyu]]

(In reply to Marshall Culpepper [:marshall_law] from comment #10)

> Do any apps on Marketplace have
> non-Marketplace URLs? 

I do have seen marketplace apps with their own domain name.

Comment 12

[[:fabrice] Fabrice Desré]

I don't know what fires the `appopened` event, but it's not gecko. So it's likely that gaia is not building the event detail correctly.

Comment 13

[Shing Lyu [:shinglyu]]

FC is coming soon (4/6), we need to be quick.

Comment 14

[Marshall Culpepper [:marshall_law]]

I've followed the trail of events back to webapps-launch, and it looks Gecko is only passing along the timestamp, url, and manifestURL to Gaia. I have a hacked up solution working locally by reusing the actual app object from the 'applications' cache. It isn't perfect, but it's not as impactful as changing Gecko directly. I'll have a patch up soon.

Comment 15

[GitHub Autolander Bot]

Attachment: [gaia] marshall:bug1137063_trackMarketplaceApps > mozilla-b2g:master

Comment 16

[Marshall Culpepper [:marshall_law]]

Comment on attachment 8581763 [details] [review]
[gaia] marshall:bug1137063_trackMarketplaceApps > mozilla-b2g:master

This patch works around the current appopened event behavior by pulling from the Gaia system app 'applications' cache. An arguably more proper fix would be to pass along |app.installOrigin| from Gecko, then copy that into the appopened event in |browser_config_helper.js|, but the impact of those changes seem riskier to me.

Comment 17

[David Flanagan [:djf]]

Comment on attachment 8581763 [details] [review]
[gaia] marshall:bug1137063_trackMarketplaceApps > mozilla-b2g:master

This looks okay to me.

Longer term, it would be good to stop using the variable name 'app' for these objects that we now discover are not actually apps but just objects with manifestURL properties.  Better to just take the manifestURL from the event and use that. If we'd done that in the first place then this bug would never have occurred. 

But given that this is 2.2+, this fix seems good enough as it is.

Comment 18

[Tamara Hills [:thills]]

Comment on attachment 8581763 [details] [review]
[gaia] marshall:bug1137063_trackMarketplaceApps > mozilla-b2g:master

Hi Marshall,

Sorry for the delay.  It looks good to me.  It seems like the better choice for a fix.  I did test out and verify as well.

Thanks,

-tamara

Comment 19

[Marshall Culpepper [:marshall_law]]

master - https://github.com/mozilla-b2g/gaia/commit/5de3325ff83004acd0796dfc9447f377e4614ad2

Comment 20

[GitHub Autolander Bot]

Attachment: [gaia] marshall:bug1137063_trackMarketplaceApps_v2.2 > mozilla-b2g:v2.2

Comment 21

[Marshall Culpepper [:marshall_law]]

Comment on attachment 8585600 [details] [review]
[gaia] marshall:bug1137063_trackMarketplaceApps_v2.2 > mozilla-b2g:v2.2

Fix tracking of apps installed from the Marketplace

[Bug caused by] (feature/regressing bug #): Bug 1119009

[User impact] if declined: We will not have usage time for apps installed from marketplace

[Testing completed]: Tested w/ ConnectA2 app from the marketplace

[Risk to taking this patch] (and alternatives if risky): Adds a new step in the AUM bootup process, possibly slowing down startup of the service

[String changes made]: None

Comment 22

[Ryan VanderMeulen [:RyanVM]]

v2.2: https://github.com/mozilla-b2g/gaia/commit/c2cf7076555a00ba32f0d6797f7cf203b4e4ee60

Comment 23

[Shally Li（leave from Mozilla）]

Hi Shing,
   
    Could you help to provide the ways to get "ping.txt"(usage time)?

Thank you very much.

Comment 24

[Shing Lyu [:shinglyu]]

Shally, please follow the steps here: https://wiki.mozilla.org/B2G/QA/Ping_Testing#Metrics_Testing_for_FxOS_2.2

Let me know if any step is unclear.

Comment 25

[Shally Li（leave from Mozilla）]

Hi Shing,

    According to the steps of Comment 0 and Comment 24, I have verified on latest Flame v2.2.The data of "usageTime" & "installs" & "uninstalls" as shown below,could you help to confirm the below results? 

Thank you very much.

-------------------------------------------------------------------------------
Actual results:
Message 1 content: 
{"usageTime": 0,"invocations": 0,"installs": 1,"uninstalls": 0,"activities": {}}

Message 2 content: 
{"usageTime": 0,"invocations": 0,"installs": 0,"uninstalls": 1,"activities": {}
"usageTime": 68,"invocations": 1,"installs": 0,"uninstalls": 0,"activities": {}

Please refer attachment:ping_1.txt and ping_2.txt.

--------------------------------------------------------------------------------
Device: Flame 2.2 build
Build ID               20150407162504
Gaia Revision          ea735c21bfb0d78333213ff0376fce1eac89ead6
Gaia Date              2015-04-07 20:58:15
Gecko Revision         https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/3f86ddb7f719
Gecko Version          37.0
Device Name            flame
Firmware(Release)      4.4.2
Firmware(Incremental)  eng.cltbld.20150407.195227
Firmware Date          Tue Apr  7 19:52:39 EDT 2015
Bootloader             L1TC000118D0

Comment 26

[Shally Li（leave from Mozilla）]

Attachment: ping_1.txt

Comment 27

[Shally Li（leave from Mozilla）]

Attachment: ping_2.txt

Comment 28

[Shing Lyu [:shinglyu]]

Shally,
This is wierd. I can see the usageTime > 0 after this patch landed. An example would be like this:
"
    "https://marketplace.firefox.com/app/bb680dfa-2ac1-45e1-b71b-3db4520c8ca8/manifest.webapp": {
        "20150408": {
            "activities": {},
            "installs": 1,
            "invocations": 1,
            "uninstalls": 0,
            "usageTime": 7
        }
    }
"
Have you opened the app after installation?

It's verified on 
  Gaia-Rev        ea735c21bfb0d78333213ff0376fce1eac89ead6
  Gecko-Rev       https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/245f37f44017
  Build-ID        20150407002501
  Version         37.0
  Device-Name     hammerhead
  FW-Release      5.0
  FW-Incremental  eng.cltbld.20150407.040951
  FW-Date         Tue Apr  7 04:10:07 EDT 2015
  Bootloader      HHZ12d

Comment 29

[Shally Li（leave from Mozilla）]

Hi Shing,
 
    I receive several usage ping messages on the server from Flame v2.2,but the data of "installs" and "usageTime" is not in the same message and is different with Comment 28. 
    After 5 times attempt,it still has some problems about Usage time. Could you help to verify this bug?

Thank you.
-----------------------------------------------------------------------------
Please see attachments: verify_v2.2.MP4,ping_v2.2_1.txt and ping_v2.2_2.txt.

STR:
1.Install "La Republica" app from the Marketplace.
2.Open the app after install (using the Marketplace "Open" button)
3.Use it for >5 sec.
4.Press the home button to go back to homescreen.
5.Uninstall the app.
6.Wait for the app usage ping messages to reach the server.
**Actual Results：
-->Message 1:
"usageTime": 0,
"invocations": 0,
"installs": 1,
"uninstalls": 0,
"activities".

-->Message 2:
{"start": 
1428651821943,
"apps": 
{"app://verticalhome.gaiamobile.org/manifest.webapp": 
{"20150410": 
{"usageTime": 80,
"invocations": 3,
"installs": 0,
"uninstalls": 0,
"activities": {
}}},"https://marketplace.firefox.com/app/54f24b41-0bfd-4942-8651-6f397fb5614d/manifest.webapp": {"20150410": 
{"usageTime": 0,
"invocations": 0,
"installs": 0,
"uninstalls": 1,
"activities": {}}

-----------------------------------------------------------------------------------
Device: Flame 2.2 build
Build ID               20150409162502
Gaia Revision          df0e04acad7c8c993f6ffe07b0ccb0ec20ee50bb
Gaia Date              2015-04-09 22:35:05
Gecko Revision         https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/091b1cc1240b
Gecko Version          37.0
Device Name            flame
Firmware(Release)      4.4.2
Firmware(Incremental)  eng.cltbld.20150409.200526
Firmware Date          Thu Apr  9 20:05:37 EDT 2015
Bootloader             L1TC000118D0

Comment 30

[Shally Li（leave from Mozilla）]

Attachment: ping_v2.2_1.txt

Comment 31

[Shally Li（leave from Mozilla）]

Attachment: ping_v2.2_2.txt

Comment 32

[Shally Li（leave from Mozilla）]

Attachment: verify_v2.2.mp4

Comment 33

[Shing Lyu [:shinglyu]]

Shally, 

I encountered that sometimes when the ping was transmitted right after the installation. Please check the pings between the first one ("installs:1") and the last one ("uninstalls:1), if any one of that has usageTime > 0 for the "https://marketplace.firefox.com/app/54f24b41-0bfd-4942-8651-6f397fb5614d/manifest.webapp" app, then the test passes.

Comment 34

[Shally Li（leave from Mozilla）]

I will try to verify this bug tomorrow.

Comment 35

[Shally Li（leave from Mozilla）]

This bug has been successfully verified on latest Nightly Flame v2.2.
See attachment: verified_v2.2.mp4
Reproduce rate: 0/5

Actual results:
"https://marketplace.firefox.com/app/54f24b41-0bfd-4942-8651-6f397fb5614d/manifest.webapp": {"20150414": {
"usageTime": 0,
"invocations": 0,
"installs": 1,
"uninstalls": 0,
"activities": {}
------------------------------------------------------------------
"https://marketplace.firefox.com/app/54f24b41-0bfd-4942-8651-6f397fb5614d/manifest.webapp": {"20150414": {
"usageTime": 62,
"invocations": 1,
"installs": 0,
"uninstalls": 1,
"activities": {}

------------------------------------------------------------------------
Device: Flame 2.2 user build (Pass)
Build ID               20150413002502
Gaia Revision          73645b097720f3ca594a14d288b87d3885d7fc9d
Gaia Date              2015-04-13 19:30:36
Gecko Revision         https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/5005522fd68e
Gecko Version          37.0
Device Name            flame
Firmware(Release)      4.4.2
Firmware(Incremental)  eng.cltbld.20150413.040508
Firmware Date          Mon Apr 13 04:05:20 EDT 2015
Bootloader             L1TC000118D0
-------------------------------------------------
I will verify Flame v3.0 later.

Comment 36

[Shally Li（leave from Mozilla）]

Attachment: verified_v2.2.mp4

Comment 37

[Shally Li（leave from Mozilla）]

This bug has been successfully verified on latest Nightly Flame v3.0 (The usageTime >0).
Reproduce rate: 0/5

STR:
1. Install La Republica from Marketplace.
2. Open the app(using the Marketplace "Open" button)
3. Use the app for >5 sec.
4. Long press Home button to close the app.
5. Go to homescreen,uninstall the app
6. Wait for the app usage ping to reach the server.
**Actual results:
"https://marketplace.firefox.com/app/54f24b41-0bfd-4942-8651-6f397fb5614d/manifest.webapp": {"20150414": {
"usageTime": 0,
"invocations": 0,
"installs": 1,
"uninstalls": 0,
"activities": {}
}
---------------------------------------------------
"https://marketplace.firefox.com/app/54f24b41-0bfd-4942-8651-6f397fb5614d/manifest.webapp": {"20150414": {
"usageTime": 7,
"invocations": 1,
"installs": 0,
"uninstalls": 1,
"activities": {}
}
-------------------------------------------------------------------------
Device: Flame 3.0 user build (Pass)
Build ID               20150412160203
Gaia Revision          c8cb0c0ebb8dd1f5c0c9037e38f8e4b237beb77b
Gaia Date              2015-04-13 18:03:34
Gecko Revision         https://hg.mozilla.org/mozilla-central/rev/0a46652bd992
Gecko Version          40.0a1
Device Name            flame
Firmware(Release)      4.4.2
Firmware(Incremental)  eng.cltbld.20150412.192808
Firmware Date          Sun Apr 12 19:28:21 EDT 2015
Bootloader             L1TC000118D0