973079 - Pull in APK Factory logs

Status: RESOLVED WONTFIX

(Marketplace Graveyard :: Statistics, enhancement, P4)

Description

[Andy McKay]

APK Factory is writing out logs for stats. We'll need to pull those into monolith. Exactly where they are will be determined in bug 973076.

Issues include:
- not everything that's in the APK factory will be in the marketplace
- not sure what identifier will be in the log, but I'm pretty sure it won't be the addon.id but something about the app manifest url

Comment 1

[Austin King [:ozten]]

There is some discussion about if this belongs in Kibana, Monolith or Heka. I don't care.

Requirements:

Create a "last 30 days" dashboard for each of the following:

1) Total number of APK Installs per day
2) ADU per day
3) Average number of Installed APKs during an update ping
4) Total number of times an out of date APKs reported

Data Source Examples:

The following three types of log lines are already being logged in monolith.log on apk-factory servers

apk-install (graph 1)

    apk-install,"https://ozten.github.io/misfeasance/manifest.webapp",2014-04-21T22:03:52.402Z

apk-update-apps-installed (graph 2 and graph 3)

    apk-update-apps-installed,4,2014-04-21T22:18:45.611Z

apk-update (graph 4)

    apk-update,f549b671-bb06-4f24-a03c-048a9e147ff4,"http://deltron3030.testmanifest.com/manifest.webapp",1398118689,2014-04-21T22:18:45.605Z

Graphs 2 and 3 get their data from apk-update-apps-installed. The presence of a log line indicates an ADU for graph 2. The second field "4" indicates the user had 4 APKs installed, averaging this data would produce graph 3.

Comment 2

[Rob Miller [:rmiller]]

Okay, I know almost nothing about monolith, nor about any of our operational constraints, but given what you've described I have a pretty good idea how I'd tackle this w/ Heka and Kibana, so I'll put that here and let those who know more about the other pieces whether there are better ways to go about it.

The only real work involved on the Heka side would be to write a custom decoder (some Lua code) to parse the log lines, extract the relevant data, and put it in a Heka message with desired field names. This would then be routed through Heka to an ElasticSearchOutput, where it would be converted into suitable JSON and sent on to ES. Once in ES it should be pretty easy to use Kibana's UI to build the graphs you want. Certainly 1, 2, and 4 would be straightforward. I haven't tried building an "average" graph in Kibana yet but I'd be surprised if it weren't possible.

Details to be worked out would be the exact path the data takes through Heka and into ES. Easiest would be to just run Heka on each apk-factory server, where it could slurp up monolith.log, do the parsing, and send it directly in to ES. There are other ways, though, if ops doesn't want to do that for some reason.

Comment 3

[Austin King [:ozten]]

I don't know Lua. What product/component could I file a bug to get help writing this conversion routine?

The logs are CSV, it should be trivial to convert them to the Kibana JSON format.

Comment 4

[Rob Miller [:rmiller]]

product: Mozilla Services (https://bugzilla.mozilla.org/enter_bug.cgi?product=Mozilla%20Services)

component: Server: Other

Assign it to me. :)

Comment 5

[Austin King [:ozten]]

> Certainly 1, 2, and 4 would be straightforward.

To get things moving, I've filed bug#1031403 with only 1, 2 and 4.
We can iterate again, after we've got things cooking end to end.

Thanks!