1357250 - Evaluate zstandard performance with telemetry data

Status: RESOLVED WONTFIX

(Data Platform and Tools :: General, enhancement, P3)

Description

[Wesley Dawson [:whd]]

For the ingestion pipeline, we know gzip on upload is the current bottleneck. We should verify that moving to zstandard improves the size and performance of our s3 uploads. If we can run fewer machines on ingestion we will have fewer total objects, which should increase the downstream Dataset API performance.

For said downstream analysis tools, when we moved to the new ingestion infra (and gzip), we measured a 10-15% performance decrease from our previous per-record snappy compression format (the cost of having smaller object sizes). With zstandard, we should expect to see a significant performance increase (> 15%) in our analysis.

The first step is to generate a day's worth of data from landfill into some test data sets in the canonical bucket (-zstd and -gzip), and compare both the amount of compute required to do so and the resultant object sizes. :mreid did work similar to this a long time ago when we were choosing compression formats.

The second step is to run some spark analysis (probably counts) on the data via the scala and python bindings, and compare the performance of the -zstd and -gzip data sets.