1632857 - Public data produced via bigquery-etl should not be gzipped from client's perspective

Status: RESOLVED FIXED

(Data Platform and Tools :: General, task)

Description

[William Lachance (:wlach)]

Currently public data in json format is gzip compressed. e.g.:

https://public-data.telemetry.mozilla.org/api/v1/tables/telemetry_derived/ssl_ratios/v1/files/000000000000.json.gz

Ideally it would not be gzipped from the client's perspective, as gzip decoding isn't available from the JavaScript/DOM space. It's fine to compress with gzip or brotli in transit, but that's handled in a different way. Also of course this doesn't preclude using any compression scheme you like on the server side of things.

Comment 1

[William Lachance (:wlach)]

The iodide notebook I mentioned in bug 1632851 still doesn't work as we can't easily decompress gzipped data like https://public-data.telemetry.mozilla.org/api/v1/tables/telemetry_derived/ssl_ratios/v1/files/000000000000.json.gz from JavaScript.

Based on this guide and a quick browse of the source code, I think there are three steps here:

1. Set the content type of the blobs to application/json
2. Set the content encoding of the blobs to gzip
3. (not actually necessary but probably best) remove the .gz post

This way we get the best of all worlds -- the files should be compressed on the server and in transit, but still accessible via JavaScript.

Anna, is this something you could do? I could probably muddle through but it's probably much faster for you.

Comment 2

[Anna Scholtz [:ascholtz]]

Sure!

Comment 3

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/bigquery-etl/pull/942

Comment 4

[Anna Scholtz [:ascholtz]]

Attachment: GitHub Pull Request

Comment 5

[Anna Scholtz [:ascholtz]]

https://public-data.telemetry.mozilla.org/api/v1/tables/telemetry_derived/ssl_ratios/v1/files/000000000000.json should now be accessible through JavaScript