1553999 - Pub/Sub for discovery stream spoc impression and click data

Status: RESOLVED FIXED

(Firefox :: New Tab Page, task, P1)

Description

[Matt Miermans]

Setup a Pub/Sub stream with sponsored content ("spoc") click and impression events. This stream will be consumed by a Telemetry Proxy Server that I'll build in GCP, which will forward the events to AdZerk.

My understanding is that it is fairly easy to do IP geolocation in the data pipeline? That would be really helpful.

On the Firefox side we are adding a new field called "shim" that can be used to identify spocs and should be passed along to Pub/Sub.

Comment 1

[Jeff Klukas [:klukas] (UTC-4)]

Currently, we can support per-docType topics, but there seems good reason here to have a per-namespace topic instead, so that all pings under the activity-stream namespace end up in a single topic.

I've proposed a change in https://github.com/mozilla/gcp-ingestion/pull/619 to support this.

Once that's merged, we will want to add the following new options to the structured ingestion republisher job:

--perDocTypeEnabledList=activity-stream/*
--perDocTypeDestination=projects/moz-fx-data-shar-nonprod-efed/topics/structured-decoded-doctype_${document_namespace}_${document_type} 

And we'll need to create a topic: projects/moz-fx-data-shar-nonprod-efed/topics/structured-decoded-doctype_activity-stream_all .

Comment 2

[Jeff Klukas [:klukas] (UTC-4)]

PR is merged. As part of review, we settled on a slightly different interface:

--perNamespaceEnabledList=activity-stream
--perNamespaceDestination=projects/moz-fx-data-shar-nonprod-efed/topics/structured-decoded-namespace_${document_namespace}

So pubsub topic would become projects/moz-fx-data-shar-nonprod-efed/topics/structured-decoded-namespace_activity-stream

Comment 3

[Wesley Dawson [:whd]]

I'll update this bug when the relevant infrastructure is available.

Comment 4

[Jeff Klukas [:klukas] (UTC-4)]

Final pubsub topic name is projects/moz-fx-data-shar-nonprod-efed/topics/structured-decoded-namespace_activity_stream and this is now getting populated.

My understanding is that it is fairly easy to do IP geolocation in the data pipeline? That would be really helpful.

Yes, the content of this stream is the "decoded" pings where the IP address has been thrown away and geo lookup has been performed. You'll see geo content both in the attributes of the Pub/Sub messages (look for geo_country, geo_city, etc.) and inside the JSON payload (look at metadata.geo).

Last piece here should be around access and making sure the service account for the proxy server has privileges to read the topic. That can be a separate bug with Data Ops if it's not already covered elsewhere.

Comment 5

[Wesley Dawson [:whd]]

And a final note: the data in this topic will be all activity stream data, not just data with the new "shim" field. Your consumer will need to filter to the subset of pings you care about before forwarding them to other downstream consumers.

Comment 6

[Matt Miermans]

Love it! Many thanks!