1344960 - Consider using the moz_telemetry.s3 module or similar for direct-to-parquet output dimensions

Status: RESOLVED INCOMPLETE

(Cloud Services Graveyard :: Metrics: Pipeline, enhancement, P3)

Description

[Wesley Dawson [:whd]]

For standard s3 uploads we have object path schema files, maintained at https://github.com/mozilla-services/puppet-config/tree/master/pipeline/modules/pipeline/files/schema and used in our site-specific s3 output: https://github.com/mozilla-services/lua_sandbox_extensions/blob/master/moz_telemetry/sandboxes/heka/output/moz_telemetry_s3.lua#L20.

We may want a similar mechanism for direct-to-parquet outputs, which currently use the standard parquet_s3, which only does basic sanitizing: https://github.com/mozilla-services/lua_sandbox_extensions/blob/master/parquet/sandboxes/heka/output/s3_parquet.lua#L253. If we want to do this, we could create a moz_telemetry_s3_parquet that provides similar kinds of bucketing to our regular s3 output so that we don't see things like "channel=r--7---*_-7----/" in our partitions, and possibly more nefarious failures like those to be addressed for bug #1339024.

Comment 1

[Firefox Bug Husbandry Bot]

Closing abandoned bugs in this product per https://bugzilla.mozilla.org/show_bug.cgi?id=1337972