Create a derived parquet dataset for the sync ping

RESOLVED FIXED

Status

()

Firefox
Sync
P1
normal
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: markh, Assigned: markh)

Tracking

(Blocks: 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [sync-metrics][measure-integrity], URL)

Attachments

(1 attachment)

(Assignee)

Description

2 years ago
The next step on the path to dashboards is to create derived parquet dataset.

Dexter tells me the steps are roughly:
- You have to build a new view for the Sync pings (i.e. as done for longitudinal and main summary). You can find an example for the MainSummary at [1] (the same repository also contains the code for the longitudinal).
- This should live in the https://github.com/mozilla/telemetry-batch-view repo.
- I suggest to use IntelliJ IDEA [2] to write your code, as you can easily import the SBT project from the repo and have it download all the dependencies, along with all its code completion cool features.

[1] - https://github.com/mozilla/telemetry-batch-view/tree/master/src/main/scala/com/mozilla/telemetry/views
[2] - https://www.jetbrains.com/idea/

I'll schedule a time to chat with Dexter soon, but getting this on file and so I don't lose the above steps a second time :)
Mark, do you have any suggestion or additional information about how to add Sync pings to re:dash?
Flags: needinfo?(mreid)
Priority: -- → P1
Whiteboard: [sync-metrics]
(Assignee)

Updated

2 years ago
Assignee: nobody → markh

Comment 2

2 years ago
There are 2 other smaller steps.

After developing and testing the Sync View code per your description, we'll need to schedule it to run from Airflow[1]. There are several examples in the "dags" directory.

We also need to import the new dataset into the Hive Metastore used by re:dash. This is a matter of adding a crontab entry on the re:dash server (at least until bug 1269781 lands) and should only take a few minutes.

[1] https://github.com/mozilla/telemetry-airflow
Flags: needinfo?(mreid)
Whiteboard: [sync-metrics] → [sync-metrics][measure-integrity]
(Assignee)

Comment 3

2 years ago
Created attachment 8792793 [details] [review]
Link to pull request
(Assignee)

Comment 4

2 years ago
This code was merged in https://github.com/mozilla/telemetry-batch-view/pull/114. Bug 1307317 and bug 1307318 are 2 followups that must be done by the pipeline team to finally get this data in re:dash - so all the work here is done.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.