661578 - Telemetry: Use a binary data packet + gzip

Status: RESOLVED FIXED

(Toolkit :: Telemetry, defect)

Description

[(dormant account)]

As we expand our use of telemetry it would be nice to minimize the size of the data packet.

Comment 1

[Justin Dolske [:Dolske]]

Can HTTP POSTs (which is what I assume we're using) do gzip compression on the fly? And do we? :)

Comment 2

[(dormant account)]

(In reply to comment #1)
> Can HTTP POSTs (which is what I assume we're using) do gzip compression on
> the fly? And do we? :)

HTTP supports it. Not sure on how to rig up our APIs to use that.

Comment 3

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

HTTP-level compression is not the best way to minimize the data size. I would have a pre-configured compression dictionary for each release, and avoid doing any binary encoding of the data. I think this will give better results than plain gzip + a binary encoding. I would like to discuss this further as I am considering a similar technique for our HTTP disk cache.

Comment 4

[Justin Dolske [:Dolske]]

Just how much data are we talking about for telemetry drops, relative to normal HTTP usage, the browser's existing update/blocklist pings, and crash reports?

My only real concern here is that binary data sucks to work with. Unless we're talking about some enormous volume of data, it seems like working with JSON (or XML ;) and compressing it for transfer should be fine.

Comment 5

[Chris H-C :chutten]

We use JSON and stream-compress it in flight. We also compress it before saving to disk. I think this can be considered fixed, though investigations into next-generation in-flight data formats are always in scope (and are currently taking place as part of the glean project)