753195 - Pull app stats out of metrics

Status: RESOLVED FIXED

(Marketplace Graveyard :: Statistics, defect, P2)

Description

[Andy McKay]

Now that stats for the marketplace are going into metrics, we need to pull them back out again.

We'll need to create the appropriate tables in marketplace and have metrics write into them. Based on the spec:

"the  number of new installations of a particular app per day over time, and  should include information on the device and source that was used to  acquire it, e.g. Mozilla Market App’s Search on Android Ice Cream Sandwich."

How about:

webapp_stats_date
- app
- date
- count

webapp_stats_device
- webapp_stats_id
- device_type
- count

webapp_stats_source
- webapp_stats_id
- source
- count

Comment 1

[Xavier Stevens [:xstevens]]

Andy, Here's a data example of one of the items we received yesterday:

{"app-domain": "enjoy.your-clock.com", "locale": "ja", "src": "", "app-id": 372921, "user-agent": "Mozilla/5.0 (Windows NT 5.1; rv:12.0) Gecko/20100101 Firefox/12.0"}

Comment 2

[Andy McKay]

As per meeting yesterday, first step is to produce some text files of the output, we'll write a script to pull that data off the site and place it into the marketplace database.

So filing this to @xstevens for that first part, then we'll file bugs to get the next part done.

Comment 3

[Christopher Van Wiemeersch [:cvan]]

Would you like for us on the Marketplace to do some sniffing so we can guess on which device the app was installed and send that along? (Right now with our JS detection we can make a pretty good guess of whether you're on a phone vs. tablet vs. desktop.) Knowing on which device an app is being used - that's blocked by AITC - but this seems like a good first step.

Comment 4

[Andy McKay]

I was thinking it might be to do that at the metrics cluster since we'll need to repeat the same code for all data going into metrics, be it pings from marketplace, AITC or reading web logs.

However I think if we can get as much information into metrics, it can't hurt. So any additional information (touch, screen res, etc.) could be helpful in conjunction with the user-agent.

Comment 5

[Xavier Stevens [:xstevens]]

Andy, Here's some examples from my runs: https://metrics.mozilla.com/protected/marketplace/

Comment 6

[Andy McKay]

Looks good. 

Could we get app-id on this one, so we can correlate:

https://metrics.mozilla.com/protected/marketplace/webapp_stats_date-2012-06-03.txt

Is it possible to add in device type to:

https://metrics.mozilla.com/protected/marketplace/webapp_stats_device-2012-06-03.txt

I think we need to provide you regexes for that?

Comment 7

[Xavier Stevens [:xstevens]]

Shouldn't be a problem to add app-id.

And yes I think you were going to provide some regex code or some sort of specification for what the device type should be. I'm descent at regex but I need to know what you want.

Comment 8

[Xavier Stevens [:xstevens]]

Updated the webapp_stats_date with appid included.

Comment 9

[Andy McKay]

Thanks xstevens, until we can figure out device types, want to start that going nightly?

Anything else you want @ngoke, @cvan?

Comment 10

[Xavier Stevens [:xstevens]]

All set to run nightly and I'm running the backfill right now. Should be done in the next half hour or so. Files are currently located in the same directory as Comment 6.

Should I close this out and you can open up a new ticket when the device stuff is ready?

Comment 11

[Andy McKay]

Sounds good, thanks Xavier.

Comment 12

[Christopher Van Wiemeersch [:cvan]]

(In reply to Xavier Stevens [:xstevens] from comment #10)
> All set to run nightly and I'm running the backfill right now. Should be
> done in the next half hour or so. Files are currently located in the same
> directory as Comment 6.
> 
> Should I close this out and you can open up a new ticket when the device
> stuff is ready?

Yes, please do. Thanks for working on this!

Comment 13

[Xavier Stevens [:xstevens]]

Resolving this for now. New adjustments and issues can be filed separately.