1457221 - Copy JSON file from CEP to S3 (for Pioneer dash)

Status: RESOLVED FIXED

(Data Platform and Tools Graveyard :: Operations, enhancement, P1)

Description

[Josh Gaunt [:jgaunt]]

I'm unable to authenticate with https://pipeline-cep.prod.mozaws.net from a Databricks (or ATMO) notebook and thus can't remotely request some JSON I need to complete a job there. 

The JSON:
https://pipeline-cep.prod.mozaws.net/dashboard_output/analysis.moz_pioneer_opt_in_estimates.pioneer_opt_in_count.json

whd suggested copying the data from CEP to S3, where I could more freely request it to load in notebooks. 

Ideally, it would copy at a predictable time shortly after 0:00 UTC each day to s3://telemetry-private-analysis-2/jgaunt/pioneer/CEP.json but I'm uncertain what s3 buckets Databricks has access to

Comment 1

[Wesley Dawson [:whd]]

It occurs to me now that we have a mechanism for doing this already in the form of near real time reports. Here's an example, with the important part being to set the message Type to "report.daily":

https://github.com/mozilla-services/lua_sandbox_extensions/blob/master/moz_telemetry/sandboxes/heka/analysis/moz_shield_study_enrollment.lua#L83

This particular output produces data to s3 daily e.g.

s3://net-mozaws-prod-us-west-2-pipeline-analysis/reports/2018-04-26/0-analysis.moz_shield_study_enrollment.count.tsv

which is pretty much exactly what you want with the addition of history. Databricks has access to s3://net-mozaws-prod-us-west-2-pipeline-analysis and so does the CEP, so no additional configuration would be required.

So I think we should update your filter to output as a report, but in the shortest term I set up a cron to write to s3://net-mozaws-prod-us-west-2-pipeline-analysis/jgaunt/pioneer/CEP.json. The data is available now, and you can expect the data to be updated within 3 minutes after end of the UTC day.

Comment 2

[Josh Gaunt [:jgaunt]]

Outstanding!

Does the existing form of output need to be undone to output as a report, or could the latter be added in parallel?

I don't have any hard-coded dependency on the existing form, mainly curious and indecisive.

Comment 3

[Wesley Dawson [:whd]]

The format of the actual json data is opaque to the infrastructure and doesn't need to change. What we would be changing would be the surrounding heka message structure. We could even output both the current and the report-annotated message at the same time. From glancing at the source a report has Type = 'reports.daily', Fields[day], Fields[name], Fields[ext], with Fields[data] as what is currently the Payload.

Comment 4

[Josh Gaunt [:jgaunt]]

I believe the output of my flawed counter at 

https://pipeline-cep.prod.mozaws.net/dashboard_output/analysis.jgaunt_mozilla_com.pioneer_addons.pioneer_opt_in_count.json 

was copied, rather than output from the improved version trink wrote at 

https://pipeline-cep.prod.mozaws.net/dashboard_output/analysis.moz_pioneer_opt_in_estimates.pioneer_opt_in_count.json

Comment 5

[Wesley Dawson [:whd]]

I've re-run the s3 copy pointing at 
https://pipeline-cep.prod.mozaws.net/dashboard_output/analysis.moz_pioneer_opt_in_estimates.pioneer_opt_in_count.json and updated the cron job as well.

Comment 6

[Wesley Dawson [:whd]]

For now, I've set up the special case for this in https://github.com/mozilla-services/puppet-config/pull/2728. We can file a separate bug for refactoring to use the standard real time reporting machinery if that is desired.