916217 - Aggregate histograms by submissions date in addition to build date

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement)

Description

[Jonas Finnemann Jensen (:jonasfj)]

As of writing histograms are aggregated by build date, this enables us to see how different
versions behave...
But we should also have a set of JSON files where data is aggregated by submission date.
To see how the internet changes over time, so we can see if changes in the histograms are
due to changes on the internet...

Summary:
 * Add a `*-bydate.json` for each measure containing histogram by submission dates,
 * Extend map/reduce job (`dashboard.py`)
 * Update mr2disk.py to handke the `*-bydate.json` files too, and
 * Update client side javascript and UI...

Comment 1

[Jonas Finnemann Jensen (:jonasfj)]

Note to self (as I'll probably end up implementing this):

We probably only want to data by submission dates for the first 12 weeks following a release, otherwise, we'll end up with huge JSON files with too many submission dates to visualize.

As a heuristic we can discard entries where `submission date - build date > 12 weeks`.

Remark, we might want to consider gradually increasing submission date intervals.
So that for the first 9 weeks, we have an entry for each day, but for the 10 week we only have for every 2nd day.
11th week every 4th, 12th week every 8th day... or something like that.

Comment 2

[Jonas Finnemann Jensen (:jonasfj)]

With the deployment of the new dashboard on the spot based analysis framework, this data is now aggregated, and can be disabled by anybody who wants to hack a dashboard :)

Comment 3

[Jonas Finnemann Jensen (:jonasfj)]

s/disabled/displayed/

The data is aggregated and can be fetched using `telemetry.js`, we only need to update the UI to show histogram evolution by submission date.

Comment 4

[Jonas Finnemann Jensen (:jonasfj)]

With the deployment yesterday, calendar dates (submission dates) are now available in the dashboard.