1596144 - Create a container to extract pings from the live activity_stream tables for CFR experiment

Status: RESOLVED FIXED

(Firefox :: Messaging System, task, P1)

Description

[Victor Ng [:vng] [:crankycoder]]

The new CFR experiment will require access to the activity_stream_live table to fetch pings out in near real-time.

The table we are targetting is moz-fx-data-shar-nonprod-efed.activity_stream_live.impression_stats_v1 which should have ~15 min old data.

The ETL will be a simple polling process that periodically checkpoints a milestone record to indicate the last processed record in the ETL job.

The job will run under the systems-research-group GCP project.

Output will be written out to a GCS bucket and synchronized with Remote Settings using a separate service.

Comment 1

[Victor Ng [:vng] [:crankycoder]]

GCP container has been built as of https://github.com/mozilla/messaging-system-personalization-experiment-1-numbermuncher/tree/20191213

Container is currently stored in gcr.io in a private container registry and will be passed to data-ops once we finalize the model.