1191677 - Extract V2 data for dashboard that combines v2 + v4 data

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Description

[Katie Parlante]

As per mreid's doc: https://docs.google.com/a/mozilla.com/document/d/1VzQHfzfA-S_lO2wpXDFjDzSJntJCMwP03TzefIj7RrE/edit?usp=sharing

This bug is for digesting v2 data to the "executive signal" as defined in the document. We can then make that data available to the combined dashboard.

Design bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1174912#c3
Implementation (meta) bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1175583

Comment 1

[Katie Parlante]

John, we'll need someone from the metrics team assigned to this.

Comment 2

[John Jensen]

Hi there, I scanned mreid's doc and based on my understanding I think Hamilton is the best bet here. Hamilton, could you look?

Comment 3

[Hamilton]

Reading through the docs now. Wondering how this project differs from bug 1179845.

Comment 4

[Katie Parlante]

(In reply to Hamilton from comment #3)
> Reading through the docs now. Wondering how this project differs from bug
> 1179845.

They're related. 

- The goal of bug 1179845 is to compare v2 to v4 data, to validate both data quality and how we're generating the v4 report. 

- The goal of the meta/parent bug (bug 1175583) is to build out the dashboard that will combine both v2 and v4 data over the period of time that we will be collecting data from both old+new clients. mreid's design has us collecting an "executive signal" from the v2 data. Instead of aggregating by date/dimension as we do with the rollup, we need to aggregate by clientId/date so that we can make some reasonable decision about clients that overlap in both v4 and v2 data sets.

We suspect that the data from this request may also help debug the discrepancies in bug 1179845

Perhaps we should set up a time to review mreid's proposal so that we're all on the same page.

Comment 5

[Mark Reid [:mreid]]

Executive signal based on v2 is available here:
s3://net-mozaws-prod-us-west-2-pipeline-analysis/mreid/exec_signal20150812/nightly/e/

Format is csv, fields are:
clientid, docid, country, channel, os, usageHours, default, google, bing, yahoo, other, buildid, crashes, plugin_crashes, activityDate