1944316 - Investigate Remote Settings Compression When Delivering Translations Record Attachments

Status: RESOLVED FIXED

(Firefox :: Translations, task)

Description

[Erik Nordin [:nordzilla]]

Description

The Remote Settings documentation specifies that it relies on CDN Dynamic Compression. The CDN Dynamic Compression documentation lists the following MIME types:

application/x-javascript
application/x-sdch-dictionary
application/javascript
application/xml
application/csv
application/json
application/json+protobuf
application/signed-exchange
application/vnd.apple.mpegurl
application/wasm
application/x-plist
application/x-protobuffer
application/x-protobuf
application/x-nacl
application/x-pnacl
font/ttf
font/otf
font/eot
image/svg+xml
image/pwg-raster
image/x-icon
image/vnd.microsoft.icon
video/vnd.mpeg.dash.mpd
audio/mpegURL
application/dash+xml
application/vnd.ms-sstr+xml
Pattern match 	application/*+json
application/*+xml
application/*mpegURL
text/* 

The MIME types of our Translations records are:

application/wasm
application/octet-stream
text/plain

Ostensibly, the application/wasm and text/plain records should be compressed automatically, but it's not clear if our application/octet-stream records are being compressed for delivery.

We should investigate this, and we should also consider if manually using xz -9 --extreme on all of our files would provide a benefit over the default gzip that is used by the CDN algorithm.

Comment 1

[Greg Tatum [:gregtatum]]

It also states it won't be compressed if "it's larger than 10 MiB." So it looks like we'll need to compress ourselves. Regardless, it looks like application/octet-stream are not be compressed, but the wasm is using brotli compression. The vocab, lexical shortlist, and model are all application/octet-stream

➤ curl -L --head --compressed https://firefox-settings-attachments.cdn.mozilla.net/main-workspace/translations-wasm/ec4aab08-4e7c-4a62-a7a5-ee6874a5d1a2.wasm
HTTP/2 200
x-guploader-uploadid: AFIdbgTlKLqIOkVwvC8xIDLH_HlW9JjR2bpub8BcxxCqJppv724Uwn06J2B0YI_QcdWQFGOs2slgo0Q
date: Tue, 28 Jan 2025 20:26:41 GMT
cache-control: public,max-age=604800
last-modified: Mon, 16 Dec 2024 17:30:03 GMT
etag: W/"89aa5d0c09b97eb5b1e8ee40df21aaf5"
x-goog-generation: 1734370203906339
x-goog-metageneration: 1
x-goog-stored-content-encoding: identity
x-goog-stored-content-length: 4956176
content-type: application/wasm
x-goog-hash: crc32c=4eTdOg==
x-goog-hash: md5=iapdDAm5frWx6O5A3yGq9Q==
x-goog-storage-class: STANDARD
accept-ranges: none
server: UploadServer
vary: Accept-Encoding
content-encoding: gzip
alt-svc: clear

➤ curl -L --head --compressed https://firefox-settings-attachments.cdn.mozilla.net/main-workspace/translations-models/394361e9-26df-46d9-b092-91952b70350b.spm
HTTP/2 200
x-guploader-uploadid: AFIdbgSJxuvxBMifW8LaY3MduV--eEM6_XXeRSklzNVErHoqFqlPCa5S10dowv4UJD9xfkr2Y1PJU4k
x-goog-generation: 1700620284972644
x-goog-metageneration: 1
x-goog-stored-content-encoding: identity
x-goog-stored-content-length: 919745
x-goog-hash: crc32c=OPelVA==
x-goog-hash: md5=13fMheBj7Emlogx8zGx43g==
x-goog-storage-class: STANDARD
accept-ranges: bytes
content-length: 919745
server: UploadServer
date: Tue, 28 Jan 2025 19:26:26 GMT
cache-control: public,max-age=604800
age: 3749
last-modified: Wed, 22 Nov 2023 02:31:24 GMT
etag: "d777cc85e063ec49a5a20c7ccc6c78de"
content-type: application/octet-stream
alt-svc: clear

➤ curl -L --head --compressed https://firefox-settings-attachments.cdn.mozilla.net/main-workspace/translations-models/793ae4c6-6689-40c6-ac76-dea8afac39b6.bin
HTTP/2 200
x-guploader-uploadid: AFIdbgSG5K1O6rarNybD4wj0akqdKGGEk63pzA0ExBuiupXf7GUawsNz4XskUFZ8fgY9hC_9
x-goog-generation: 1700620289190609
x-goog-metageneration: 1
x-goog-stored-content-encoding: identity
x-goog-stored-content-length: 5607608
x-goog-hash: crc32c=qQnVnA==
x-goog-hash: md5=lz6TcEQan3KGhWg8rWc5/w==
x-goog-storage-class: STANDARD
accept-ranges: bytes
server: UploadServer
date: Tue, 28 Jan 2025 19:26:12 GMT
cache-control: public,max-age=604800
age: 3796
last-modified: Wed, 22 Nov 2023 02:31:29 GMT
etag: "973e9370441a9f728685683cad6739ff"
content-type: application/octet-stream
content-length: 5607608
alt-svc: clear

➤ curl -L --head --compressed https://firefox-settings-attachments.cdn.mozilla.net/main-workspace/translations-models/23faa0b3-fa97-4e8a-a1db-28b8f2f48ea8.bin
HTTP/2 200
x-guploader-uploadid: AFIdbgTbMYy8ycZvXB2KjQOr6ebazVfsTs5st6DJ8ZCVbsJV0VgHjOEhPhk-Rzsgwcr8Ax5m6gtQC6w
x-goog-generation: 1700620295577216
x-goog-metageneration: 1
x-goog-stored-content-encoding: identity
x-goog-stored-content-length: 17140899
x-goog-hash: crc32c=HirJVA==
x-goog-hash: md5=HBwzleEqQf10WEQF9ik8JQ==
x-goog-storage-class: STANDARD
accept-ranges: bytes
server: UploadServer
date: Tue, 28 Jan 2025 19:24:15 GMT
cache-control: public,max-age=604800
age: 3924
last-modified: Wed, 22 Nov 2023 02:31:35 GMT
etag: "1c1c3395e12a41fd74584405f6293c25"
content-type: application/octet-stream
content-length: 17140899
alt-svc: clear

Comment 2

[Erik Nordin [:nordzilla]]

I've put together a document analyzing the urgency and feasibility of shipping all compressed data with just-in-time decompression.

I believe that we should try to move forward with Bug 1947431 and collect further benchmarks to support the hypothesis that DecompressionStream with native support for zstd will allow us to efficiently decompress models as needed.

Comment 3

[Erik Nordin [:nordzilla]]

Marking this resolved now that Bug 1982594 has landed.