Pull in APK Factory logs



4 years ago
2 years ago


(Reporter: Andy McKay, Unassigned)


Mac OS X
Dependency tree / graph


(Whiteboard: [feature][marketplace-transition])



4 years ago
APK Factory is writing out logs for stats. We'll need to pull those into monolith. Exactly where they are will be determined in bug 973076.

Issues include:
- not everything that's in the APK factory will be in the marketplace
- not sure what identifier will be in the log, but I'm pretty sure it won't be the addon.id but something about the app manifest url


4 years ago
Priority: -- → P4


4 years ago
Severity: normal → enhancement
Whiteboard: [feature]
Blocks: 941926
No longer blocks: 936669
Blocks: 936669
No longer blocks: 941926

Comment 1

4 years ago
There is some discussion about if this belongs in Kibana, Monolith or Heka. I don't care.


Create a "last 30 days" dashboard for each of the following:

1) Total number of APK Installs per day
2) ADU per day
3) Average number of Installed APKs during an update ping
4) Total number of times an out of date APKs reported

Data Source Examples:

The following three types of log lines are already being logged in monolith.log on apk-factory servers

apk-install (graph 1)


apk-update-apps-installed (graph 2 and graph 3)


apk-update (graph 4)


Graphs 2 and 3 get their data from apk-update-apps-installed. The presence of a log line indicates an ADU for graph 2. The second field "4" indicates the user had 4 APKs installed, averaging this data would produce graph 3.
Okay, I know almost nothing about monolith, nor about any of our operational constraints, but given what you've described I have a pretty good idea how I'd tackle this w/ Heka and Kibana, so I'll put that here and let those who know more about the other pieces whether there are better ways to go about it.

The only real work involved on the Heka side would be to write a custom decoder (some Lua code) to parse the log lines, extract the relevant data, and put it in a Heka message with desired field names. This would then be routed through Heka to an ElasticSearchOutput, where it would be converted into suitable JSON and sent on to ES. Once in ES it should be pretty easy to use Kibana's UI to build the graphs you want. Certainly 1, 2, and 4 would be straightforward. I haven't tried building an "average" graph in Kibana yet but I'd be surprised if it weren't possible.

Details to be worked out would be the exact path the data takes through Heka and into ES. Easiest would be to just run Heka on each apk-factory server, where it could slurp up monolith.log, do the parsing, and send it directly in to ES. There are other ways, though, if ops doesn't want to do that for some reason.

Comment 3

4 years ago
I don't know Lua. What product/component could I file a bug to get help writing this conversion routine?

The logs are CSV, it should be trivial to convert them to the Kibana JSON format.
product: Mozilla Services (https://bugzilla.mozilla.org/enter_bug.cgi?product=Mozilla%20Services)

component: Server: Other

Assign it to me. :)

Comment 5

4 years ago
> Certainly 1, 2, and 4 would be straightforward.

To get things moving, I've filed bug#1031403 with only 1, 2 and 4.
We can iterate again, after we've got things cooking end to end.

Depends on: 1031403


2 years ago
Last Resolved: 2 years ago
Resolution: --- → WONTFIX
Whiteboard: [feature] → [feature][marketplace-transition]
You need to log in before you can comment on or make changes to this bug.