1344572 - Teach beetmover to set Cache-Control header in latest directories

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, enhancement, P3)

Description

[Nick Thomas [:nthomas] (UTC+12)]

We replace files in directories like 
  firefox/nightly/latest-mozilla-central
  firefox/nightly/latest-mozilla-central-l10n
   mobile/nightly/latest-mozilla-central-android-api-15
   mobile/nightly/latest-mozilla-central-android-api-15-l10n
   mobile/nightly/latest-mozilla-central-android-x86
and similar for mozilla-aurora. We serve new installs out of these directories, and the default caching on archive.m.o is ~ 1 week, so we need to set a special header on the S3 key so that the CDN doesn't serve old content. This shows up like this:

$ curl -IL "https://download.mozilla.org/?product=firefox-nightly-latest-l10n-ssl&os=osx&lang=ar"
HTTP/1.1 302 Found
Cache-Control: max-age=60
Content-Length: 139
Content-Type: text/html; charset=utf-8
Date: Sun, 05 Mar 2017 20:29:06 GMT
Location: https://download-installer.cdn.mozilla.net/pub/firefox/nightly/latest-mozilla-central-l10n/firefox-54.0a1.ar.mac.dmg
Connection: keep-alive

HTTP/1.1 200 OK
Content-Type: application/x-apple-diskimage
Content-Length: 74982443
Connection: keep-alive
Date: Sun, 05 Mar 2017 20:29:08 GMT
x-amz-replication-status: COMPLETED
Last-Modified: Sun, 05 Mar 2017 12:55:46 GMT
ETag: "ce5df8d4ab633619274ec069dbf6b659"
Cache-Control: max-age=3600              <--------------------- this guy, right here
x-amz-version-id: KlmjjZYthqb.Uo6scBhzrR5zAG0pU6Xc
Accept-Ranges: bytes
Server: AmazonS3
Via: 1.1 b4479f202bc7050d8c458896bb43d335.cloudfront...
X-Cache: Miss from cloudfront
X-Amz-Cf-Id: cpLJ6ghvaXbrPiGlX3mObqiMSFBTgqxOAJqhxTdpRPTOsM4oCbKuKw==

The Cache-Control header is missing in requests for Firefox linux and linux64, and also for Fennec. Examples of places making these links:
* Firefox - https://www.mozilla.org/en-US/firefox/nightly/all/ to the download.m.o example
* Fennec - https://www.mozilla.org/en-US/firefox/channel/android/ pointing to
https://archive.mozilla.org/pub/mobile/nightly/latest-mozilla-central-android-api-15/fennec-54.0a1.multi.android-arm.apk.

So, we should teach beetmover to include CacheControl when adding new keys to S3 for paths including latest-*. FWIW, the old upload system (post_upload) uses a regexp of "^pub/.*/nightly/latest.*".

Comment 1

[Mihai Tabara [:mtabara]⌚️GMT]

This issue came up again today. I'll have a look into tweaking beetmover to make this work.

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bug 1344572 - add Cache-Control headers for beetmoved artifacts. r=nthomas

Comment 3

[Mihai Tabara [:mtabara]⌚️GMT]

@nthomas> mtabara: so that’ll be for everything we beetmove ?
13:09:48 <mtabara> yeah
13:10:11 <mtabara> wasn't sure if that's the case but looking at random artifacts for win32, both in latest* and outside, seems like they all have that Cache-Control header set
13:10:12 <mtabara> e.g. https://pastebin.mozilla.org/8984032
13:10:22 <mtabara> so I guess it couldn't hurt, could it?
13:10:27 <mtabara> s,could,can
13:11:10 <@nthomas> hmmm, interesting. that’s not how I thought it was set up
13:11:28 <@nthomas> maybe we did the same for candidates as nightly/latest-*
13:12:04 <@nthomas> except your example isn’t candidates, silly me
13:16:18 <mtabara> bah, you're right - scratch that PR. candidates dirs don't have that set - https://pastebin.mozilla.org/8984033
13:16:50 <mtabara> if we go on with the PR, we'll affect the candidates beetmoved artifacts as well.
New messages
13:17:26 — mtabara rushed to think this is one-liner fix, need to investigate more
@nthomas> that might be fine, might be not. Depends how much traffic giving out RC builds on beta update channel generates

Comment 4

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8853846 [details] [review]
Bug 1344572 - add Cache-Control headers for beetmoved artifacts. r=nthomas

on hold for now - might be slightly trickier to solve than the initial one-liner I had in mind.

Comment 5

[Mihai Tabara [:mtabara]⌚️GMT]

:catlee and :oremj discussed this earlier today again, some context is here:

@catlee> oremj: does this need to be set per object when uploading, or can it be set as a policy?
19:34:14 <oremj> catlee: it is per object. post upload is already set to add cache-control to those objects, but if they are uploaded in a different way, you'll need to add cache-control on the PUT
19:35:55 <@catlee> right, ok
19:36:53 <oremj> the default is one day
19:37:21 <oremj> so once you alter your tools, you'll upload and it will take ~1day to replace existing files, then from there on out it will be 1hr or whatever you specifcy
19:37:24 <oremj> specify
20:08:31 <@catlee> oremj: is 1h too short a lifetime for release objects?
20:39:32 <oremj> catlee: all or just nightly?
20:40:11 <@catlee> oremj: all
20:40:17 <@catlee> just hypothetically
20:40:35 <oremj> catlee: i'm pretty sure that would be fine with our traffic patterns
20:41:21 <oremj> no great way to see how it would affect the backend request rate unless we tried it out
20:41:52 <oremj> for anything that isn't firefox release, I think we should go for 1hr
20:41:58 <oremj> for firefox release, it would be probably better to taper
20:42:32 <oremj> maybe try 4 hr and then 1
20:42:36 <oremj> or something along those lines


So my take aways from this are:
1. ideally we'd have that cache control somewhat configurable and generic in beetmoverscript so that we can enforce different values for differrent locations/channels (candidats vs nightly, etc).

2. however, short term, it shouldn't hurt setting an enforced Cache-control of ~1h for all artifacts.

@catlee: do you agre with this? Should we go for now with 2) and see how that behaves in production or should I spend a bit more time and come up with a more elaborate but more elegant way to solve it via 1) ?

Comment 6

[Chris AtLee [:catlee]]

Maybe we try 4h first, and see how that impacts release. It will be an improvement for nightly in any case.

Comment 7

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8853846 [details] [review]
Bug 1344572 - add Cache-Control headers for beetmoved artifacts. r=nthomas

Adjusting to 4h and calling out for a second glance from :catlee.

Comment 8

[Chris AtLee [:catlee]]

Jeremy, FYI that we will be setting a 4h cache time for new nightly/release artifacts.

Comment 9

[Mihai Tabara [:mtabara]⌚️GMT]

https://hg.mozilla.org/build/puppet/rev/2917853968fb
Bug 1344572 - bump beetmoverscript in beetmoverworkers. r=trivial

Comment 10

[Jeremy Orem [:oremj]]

Sounds good, thanks for the heads up.

Comment 11

[Mihai Tabara [:mtabara]⌚️GMT]

I respun the "ro" locale with the new beetmoverscript that contains this Cache-control change.
Cache-Control is now set.

Closing this for now to see how it behaves. 
Will reopen if we need to adjust the timing.

$ curl -IL "https://download.mozilla.org/?product=firefox-nightly-latest-l10n-ssl&os=linux&lang=ro"
HTTP/1.1 302 Found
Cache-Control: max-age=60
Content-Length: 150
Content-Type: text/html; charset=utf-8
Date: Fri, 05 May 2017 17:27:40 GMT
Location: https://download-installer.cdn.mozilla.net/pub/firefox/nightly/latest-mozilla-central-l10n/firefox-55.0a1.ro.linux-i686.tar.bz2
Connection: keep-alive

HTTP/1.1 200 OK
Content-Type: application/x-tar
Content-Length: 64777125
Connection: keep-alive
Date: Fri, 05 May 2017 17:27:42 GMT
x-amz-replication-status: COMPLETED
Last-Modified: Fri, 05 May 2017 16:22:58 GMT
ETag: "1a4c117627f9c8b2da20773322e2266c"
Cache-Control: public, max-age=14400     <------------ this guy right here
x-amz-version-id: wPd8seJP1H0CoyGmXXxT30xDymyyt2fb
Accept-Ranges: bytes
Server: AmazonS3
Via: 1.1 b74a7a3f7ddfd685212e870d027c332d.cloudfront.net (CloudFront), 1.1 1f95d1a95c6f2df8073daccac07b7570.cloudfront.net (CloudFront)
X-Cache: Miss from cloudfront
X-Amz-Cf-Id: u1qGZ3v2wM39MUVoT1LLckNJF8beoa4s_ckV59GZfrVJbE1DbuYV3A==