991828 - Query fxa-auth-server db for metrics stats once a day and send them to fxa heka

Status: RESOLVED FIXED

(Cloud Services :: Operations: Metrics/Monitoring, task)

Description

[Katie Parlante]

This has been discussed in the meta bug starting here: https://bugzilla.mozilla.org/show_bug.cgi?id=972105#c5

Opening a new bug to track this piece of the work.

We want to track two data points per day: total created and total verified, as of midnight GMT. (The query can run after that point in time, say 4am or whatever).

Eventually we want to store it on an ES index or some other data store; getting it to heka is both a good first step for that and good enough for our immediate needs (prevents Benson from having to run a manual query once a week).

Sample queries:
select count(*)
from accounts
where createdAt < UNIX_TIMESTAMP('2014-03-30 00:00:00')*1000

select count(*)
from accounts
where createdAt < UNIX_TIMESTAMP('2014-03-30 00:00:00')*1000 and emailVerified = true

Comment 1

[Katie Parlante]

Heka dashboard should look a *lot* like this one: https://heka.fxa.us-west-2.dev.mozaws.net/#sandboxes/FxaSandboxManager-FxaAccountCreated/outputs/FxaSandboxManager-FxaAccountCreated.DailyRunningTotal.cbuf

Comment 2

[Katie Parlante]

It would be great if we could move on this... I'd like to be able to do per-day stats on the statusboard when we go live, and having the data in heka would make everything a lot easier.

:gene, could we bump up the priority, or reassign if that makes sense?

Comment 3

[Gene Wood [:gene]]

Sure thing, working on it today.

Comment 4

[Gene Wood [:gene]]

I've put together this PR and will be working on tweaking it based on reviews today

https://github.com/mozilla-services/puppet-config/pull/418

Comment 5

[Gene Wood [:gene]]

This feature has been merged and will show up in the next production train that goes out

Comment 6

[Mike Trinkala [:trink]]

I am unclear on the use case.  Heka has the raw data and can calculate these in real time and from the PR it doesn't look like we are going to use the data to backfill the Heka graph.  

The only thing Heka has to estimate is the active daily users (because it cannot store the entire set in 8M of memory) that might be interesting to see the actual vs estimate but I am not sure that is really useful.

Comment 7

[Katie Parlante]

The use case is populating the geckoboard (or its replacement). I need the actual total. (The backfill data is important fwiw, but I have that in the spreadsheet).

Perhaps we can/want to compute this in real time, but we need to factor in the accounts that have already been created, accounts that get deleted, etc.

Querying the db is easy and reliable. At the end of the day though, I don't really care *how* we do it, I just want the numbers to be reliable and to persist. (Not that this gets us there, but at least it automates one step and sets us up to send it somewhere to perist & be graphed).

Comment 8

[Katie Parlante]

I should say, the immediate use case is for me to look at heka, grab the totals and put them in my spreadsheet (without bugging :mostlygeek daily). Hopefully soon we turn that into sending the data on to geckboard/geckoboard replacement.

Having a heka graph of total/verified/adu is also nice to have, but ultimately we need graphs for an audience that is not going to have access to heka.