578593 - Figure out a solution that can regularly produce update stats

Status: RESOLVED FIXED

(Camino Graveyard :: General, defect)

Description

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

We haven't had regular, or even pseudo-regular, stats updates for 6 months (Jan 18), with the exception of a week in late April.  This is because stats is not really bread-truck safe :-(

I now know where the logs are stored, and how to fetch them (but not how to trigger log rollover, which really needs to be done before they hit 800+ MB, nor the details of processing the logs), but we need to come up with a solution for running the stats regularly. This could take the form of:

a) Sam producing his scripts and any exclusion data/practices (i.e., what else besides Brazil guy is regularly excluded?) so someone else can run them

b) Stuart using the oracle to process data and just noting that from "this point" forward, data is not 100% comprable with the old data

c) Something else I've not thought of

We need some sort of solution, though, because 6 months without stats and 800+ MB log files aren't very helpful.

Comment 1

[Samuel Sidler (:ss)]

Option 'a' is in your Google Docs. Does that resolve this bug?

Comment 2

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

(In reply to comment #1)
> Option 'a' is in your Google Docs. Does that resolve this bug?

Yes, once you tell me what email address to use in the license block so I can check the script in (and then check in my subsequent changes, though there's probably some more polish that can be done).

I've successfully run the script against the stub file (pings from 21 Jan, and partial pings from 20 Jan and 30 Jan) and updated the spreadsheet with that data.

Comment 3

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Sam's script: http://hg.mozilla.org/camino/rev/44fedf5c6141

My first round of changes made using it last night: http://hg.mozilla.org/camino/rev/b646d770256f

There's still more to polish, but this makes it a little less painful for me to use ;-)  Thanks Sam!