887307 - Enable caching for releng-http on scl3 internal cluster

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

There are two vhosts on this virtual server which consume a huge amount of bandwidth.  Unfortunately, the traffic traverses the same NIC twice (once for the frontend, and once for the backend), but caching can at least cut down the backend traffic significantly.

The vhosts are
  runtime-binaries.pvt.build.mozilla.org (in use)
  tooltool.pvt.build.mozilla.org (releng will switch to this someday)

Both serve a small set (about 300) of large compressed files (averaging 100M), which should be easily cached.  There's already a bug to look at better client-side caching (bug 887301), but that approach is limited since there are ~5000 clients, each with their own cache.

I don't know enough zeus to set this up correctly.

Comment 1

[Jake Maul [:jakem]]

With coordination in IRC, we have enabled caching on the releng-http Virtual Server on the internal Zeus cluster in SCL3. Default/max TTL is 10 minutes. We may want to raise that.

We also raised the max-cacheable-filesize from 20MB to 200MB. This isn't enough to cache *everything*, but enough to cache the average, and enough to seriously dent the bandwidth usage coming in from the backend web servers.


The one potential caveat is that this affects all sites hosted behind the releng-http virtual server. Dustin spot-checked at least one of them, and it returns sensible Cache-Control headers already (which will override Zeus's 10min default), so we are hopeful this won't cause issues. If it does, the simple solution is to have Apache send proper Cache-Control headers for any content which is being improperly cached.