1619020 - Provision pipeline family for Firefox Accounts

Status: RESOLVED INACTIVE

(Data Platform and Tools Graveyard :: Operations, task)

Description

[Wesley Dawson [:whd]]

The current plan is to name this family firefox_accounts, which produces the following pubsub topics:

firefox_accounts-raw (input topic we grant fxa permissions to write to)
firefox_accounts-decoded (decoded data)
firefox_accounts-decoded-doctype_PLACEHOLDER (republished doctype-specific stream for amplitude events)

The bigquery datasets and tables for the namespaces/doctypes in this pipeline family will not need to be provisioned specially since output tables are not directly related to pipeline families.

We plan to run the following jobs:
decoder - standard, but may require code changes to understand the input format (e.g. metadata parsing of the URI will likely need special cases)
payload_bytes_{raw,decoded_error} and live sinks - standard infrastructure with no changes needed, though there's some discussion of adding support for not sending certain data to bigquery
republisher (provisioned to republish amplitude events, no other changes needed)
amplitude-export (new code, reading from the amplitude events-specific doctype pubsub topic)

WIP with some placeholders here: https://github.com/mozilla-services/cloudops-infra/compare/data_fxa?expand=1&w=1.

This is the first time we'd be running a streaming job reading from a per-doctype topic, and there are some naming implications around that I'm still investigating. Another option would be to not republish specific doctypes for amplitude export and instead have the export job do filtering. This would simplify the ops infrastructure quite a bit but has potential semantic and cost implications. There are also considerations to be made regarding reusability of the export code for other pipeline families / namespaces and how that interacts with the namespaces, doctypes, and schemas we choose.

Comment 1

[Jeff Klukas [:klukas] (UTC-4)]

Note that I'm inclined to call the new namespace for these schemas firefox-accounts which matches the naming here (pipeline family firefox_accounts); it will be good to be consistent.

Comment 2

[Jeff Klukas [:klukas] (UTC-4)]

We no longer plan to support streaming FxA events to Amplitude via the data pipeline. We may instead pursue using scheduled queries down the line.