Closed Bug 1644298 Opened 5 years ago Closed 5 years ago

Allow GZIP compression for attachments in security-state-staging/intermediates

Categories

(Cloud Services :: Server: Remote Settings, enhancement)

Product:
Cloud Services
Component:
Server: Remote Settings
Type:
enhancement

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jcj, Unassigned)

References

Details

Intermediate Preloading is enabled for all Desktop platforms (shipped in 75 in bug 1603834), so all profiles download the (currently) 2076 intermediates (4.0 MB) over a period of several days. We're considering increasing the rate at which those are being downloaded to further reduce unknown issuer errors.

It appears via experimentation that the https://firefox-settings-attachments.cdn.mozilla.net/security-state-staging/intermediates path is not enabling content compression. For example, running:

http -vv https://firefox-settings-attachments.cdn.mozilla.net/security-state-staging/intermediates/663df175-b989-4842-abee-ef65aef3cdbf.pem

doesn't set the content-encoding header:

GET /security-state-staging/intermediates/663df175-b989-4842-abee-ef65aef3cdbf.pem HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: firefox-settings-attachments.cdn.mozilla.net
User-Agent: HTTPie/2.1.0


HTTP/1.1 200 OK
Accept-Ranges: bytes
Age: 45022
Connection: keep-alive
Content-Length: 2333
Content-Type: application/octet-stream
Date: Mon, 08 Jun 2020 10:01:39 GMT
ETag: "5eff769774a83fca1f90421282e9a1e0"
Last-Modified: Fri, 08 Feb 2019 17:37:11 GMT
Server: AmazonS3
Via: 1.1 6ae304c394ca48eaeac474c114a24c89.cloudfront.net (CloudFront)
X-Amz-Cf-Id: IUaFLj6kHDDB1PazMfUo3OEuDzEP2VWeN231b-62RAQCtcMese2rWg==
X-Amz-Cf-Pop: LAX3-C3
X-Cache: Hit from cloudfront
x-amz-version-id: bd52gEWhf4PJkIK_S1iW7eWfLKOcoBNA


-----BEGIN CERTIFICATE-----
....

As an example though, that particular certificate compresses from 2333 bytes to 1733 bytes via gzip. Comparing the major compression algorithms for the whole cohort:

2.9M	/tmp/zstd-pem/
2.8M	/tmp/brotli-pem/
3.0M	/tmp/gzip-pem/
4.0M	/tmp/uncompressed-pem/

(Note zstandard is not supported by Firefox, but being a bit exotic and part of draft-ietf-tls-certificate-compression, I thought I'd test it.) Since GZIP is close in performance to Brotli (which is supported by Firefox), I think either would be fine and an improvement for bandwidth over uncompressed data.

Sven, do you know if there's a way to update the mimetype of existing S3 files?

If we could set the mimetype of the .pem files (to text/plain for example), we could benefit from the GZIP compression on Cloudfront. Reuploading them would be annoying since it would oblige every client to redownload everything. And if I remember well, this costs a lot of money.

Flags: needinfo?(sven)
Assignee: mathieu → nobody

:leplatrem It's possible to change MIME types, both via the console and via the API. I vaguely remember that the aws CLI requires re-uploading the files with the new MIME type, but via the console it's possible to do that without re-uploading.

Clients would only need to redownload if we publish a new version via Kinto, right? Re-uploading the same file with the same name manually, outside of Kinto shouldn't trigger any redownloads.

How many files need to be fixed? If it's just a handfull, I can do that manually in the console, otherwise we'd need to script it in some way.

Flags: needinfo?(sven)

Clients would only need to redownload if we publish a new version via Kinto, right? Re-uploading the same file with the same name manually, outside of Kinto shouldn't trigger any redownloads.

Yes, you're right. Re-uploading without going through Kinto isn't going to change the hashes, and thus will be transparent for the clients.

How many files need to be fixed?

Something like ~2000. Depends on your patience ;)

I can't really say how urgent this is. Also, I don't have the data to estimate the cost-save if we download 25% less data on this collection. Unless you are available to work on it now, we can chat about it in our next standup.

If we need to adjust the mime-types, we need to make an adjustment to the CRLite side to set that mime type on new entries (which do happen over time). I've opened https://github.com/mozilla/crlite/issues/76 for that.

Checking code, Firefox doesn't check mime-type: https://searchfox.org/mozilla-central/rev/fac90408bcf52ca88a3dcd2ef30a379b68ab24e2/security/manager/ssl/RemoteSecuritySettings.jsm#776

Once https://github.com/mozilla/crlite/issues/76 is fixed and deployed, I can fix the MIME type for all existing PEM attachments. I will need a list of the files I need to fix, or a way of composing that list myself.

I thought I had it fixed, but it turns out that some aspect of kinto-attachments is smart enough that it is overriding my explicit mime-type to set pem, presumably based on the attachment filename. [0]

[0] https://github.com/mozilla/crlite/issues/76#issuecomment-646338470

We had to move the Kinto attachments to a different CDN (Akamai) due to some change in our CloudFront contract that we were not aware of, which resulted in significant extra cost. On Akamai, we are a lot more flexible with MIME types, so we can choose to compress any MIME type we would like to. I'll try to figure out whether we even need any further changes on the CRLite side.

I don't think we need any changes to the upload code for the intermediate certs, and I think it would be good if you could revert the change to omit the .pem file extension on uploads. We need to set the content type in Kinto when uploading to S3, and the current plan is to do that based on the file extension. :leplatrem Can you confirm that this is correct, please?

Yes this is correct!

There is no way to force a specific mimetype via the REST API when uploading attachments, we use the file extension.
We will use .pem extension to set the correct mimetype (application/x-pem-file).

Compression for Kinto attachments should now work in production, but not yet in stage. The latter can only be fixed once I've moved stage to Akamai as well. (I know it's an unusual order to do prod first, but we needed to act fast for prod since it was burning a lot of cash, and the risk was very low.)

I'll mark this ticket as fixed now. Moving the stage CDN to Akamai is tracked in bug 1645848.

Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.