1380737 - Write an airflow job to populate a derived database on hbase for TAAR

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement, P1)

Description

[mlopatka]

Job to populate only the important columns for making addon recommendation into hbase instance that TAAR will use.

Should be modelled after this job: 
https://github.com/mozilla/telemetry-batch-view/blob/647abfb7f014995267360a50f8bacee788737b2f/src/main/scala/com/mozilla/telemetry/views/HBaseMainSummaryView.scala

documentation here: 
https://docs.google.com/document/d/1c6S786IA88Ga0NOlZiEEQj071AOTwyVZUeZYyihZNw0/edit

Comment 1

[mlopatka]

depends on 1380713

Comment 2

[Alessio Placitelli [:Dexter]]

Other than the features outlined in bug 1380713, we also need to add the addonDetails column described here [1].

@aswan, as of now, the |addonDetails| field in telemetry seems to report information about installed legacy and disabled addons. Are you aware of any plan to not send this information in Firefox 57? Will we still have data about installed legacy addons when 57 goes live?

If you're not the right person to answer this, could you point me to who could?

[1] - http://gecko.readthedocs.io/en/latest/toolkit/components/telemetry/telemetry/data/main-ping.html#addondetails
[2] - https://dxr.mozilla.org/mozilla-central/search?q=setTelemetryDetails&case=true

Comment 3

[Andrew Swan [:aswan]]

In 57, legacy addons will remain installed but disabled.  Among other things, this means that the author of a legacy extension can create a WebExtension version and provide it as an update, and the browser will install it just like any other extension update.  I don't know of any plans to change anything that would affect the information that is available to telemetry about disabled addons.

Comment 4

[Alessio Placitelli [:Dexter]]

Attachment: HBase Scalar Job PR

Comment 5

[Alessio Placitelli [:Dexter]]

Attachment: Airflow PR

Comment 6

[Alessio Placitelli [:Dexter]]

The PRs were reviewed and merged and they are pushing data to the HBase instance.

Comment 7

[Firefox Bug Husbandry Bot]

Moved to new component, per bug 1425844.