1429955 - Create a prototype ingestion pipeline in gcp

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement, P1)

Description

[Daniel Thorn [:relud]]

at a glance it looks like this would be a good pipeline:

app engine (1) -> cloud pub/sub -> app engine (2) -> cloud storage -> cloud function -> big query

then set up big query to be accessible in redash.

app engine (1) validates json schema and forwards to pub/sub

cloud pub/sub makes sure we can connect other data warehouses and real-time analytics

app engine (2) batches up incoming data and writes to cloud storage (in avro format maybe)

cloud storage makes sure we don't run into limits when inserting to big query

cloud function executes on a regular basis and inserts data from cloud storage to data partitioned big query tables

Comment 1

[Daniel Thorn [:relud]]

date* partitioned big query tables

Comment 2

[Daniel Thorn [:relud]]

reference material https://cloud.google.com/solutions/architecture/optimized-large-scale-analytics-ingestion

Comment 3

[Daniel Thorn [:relud]]

Recording work here https://docs.google.com/document/d/1ENoZqLYBl-EyS9b8dZ-QDpsYLBZfGLKRdYO4NbpfMYk/edit#

Git repo here: https://github.com/relud/telemetry-sample

Comment 4

[Daniel Thorn [:relud]]

The google doc in comment 3 is up to date with current work on this. resolving fixed. Future work will be done in other bugs.