1349070 - Create views for Sync engine error data

Status: RESOLVED DUPLICATE of bug 1381641

(Data Platform and Tools :: General, enhancement, P1)

Description

[Alex Davis [:adavis]]

The Sync summary view includes engine error data, but extracting it in re:dash requires a `CROSS JOIN UNNEST`, which is impractical as we accumulate sync pings. This bug tracks creating a Spark view in a format that's more convenient to query.

Comment 1

[Thomas Huelbert]

as per Alex this goes to Kit. P2.

Comment 2

[Anthony Miyaguchi [:amiyaguchi]]

An initial notebook with a breakdown of the new view can be found in [1].

1. follow-up bug should be filed for importing initial data into athena/re:dash
2. dataset should be evaluated for usability

If (2) looks good:

3. notebook should be scheduled on ATMO
4. notebook should be ported to python_mozetl [2] with tests asserting dataset properties
5. job should be scheduled on airflow [3] from python_mozetl

In the near future steps 3-5 are done by the creator of the dataset, but the process could use some better documentation.

Has this dataset been evaluated against any queries yet? If it hasn't, I can help get the ball rolling. 

[1] https://gist.github.com/thomcc/5c1cb88eabc5897cd0fb32e0d928e02b
[2] https://github.com/mozilla/python_mozetl
[3] https://github.com/mozilla/telemetry-airflow

Comment 3

[Alex Davis [:adavis]]

Anthony, can you import that data into a table for the last 30 or 60 days so that we can double check it first? Better safe than sorry.

If all is good, you could flush it and then backfill and schedule it moving forward.

Comment 4

[Anthony Miyaguchi [:amiyaguchi]]

Once the data is generated, it should be easy to file a bug to get it into redash to query against (see bug 1347785 for a reference).

Comment 5

[Thom Chiovoloni [:tcsc] (ex-moco)]

Unnecessary given that 1381641 ended up working out.