1593372 - Add proxy_cache and proxy_cache_path settings to searchfox nginx to cache dynamic content

Status: RESOLVED FIXED

(Webtools :: Searchfox, enhancement)

Description

[Andrew Sutherland [:asuth] (he/him)]

Our python and rust web servers use proxy_pass which on its own does not cache the things retrieved this way. We should add proxy_cache and proxy_cache_path settings and similar in order to cache these deterministic byproducts. See bug 1593367 for context.

Some docs on this:

• https://www.nginx.com/blog/nginx-caching-guide/
• https://docs.nginx.com/nginx/admin-guide/content-cache/content-caching/

Comment 1

[Andrew Sutherland [:asuth] (he/him)]

Okay, I have an initial attempt at this up at https://github.com/mozsearch/mozsearch/pull/252 after the usual hair-pulling that all configuration changes like this entail. It's currently up on both servers by manually running the directory creation steps from the script and then running web-server-setup.sh manually as well.

It takes a load like https://searchfox.org/mozilla-central/rev/95342c2fde91c6e365a7283ec974f80d89685129/dom/indexedDB/ActorsParent.cpp that is 54,710ms in the network panel down to 408ms in a separate firefox instance with its own profile, noting that the data was 11.3MB and I have a very fast network connection.

Note that I also extended the proxy timeout delays because https://searchfox.org/mozilla-central/rev/f372470e10c8cb0691681603a1d6324dee5b3b8a/dom/indexedDB/ActorsParent.cpp which was the problematic file in bug 1593367 was actually taking longer than 60 seconds to generate and so we were returning a 504 for that on the initial load before it finished processing.

In my manual testing, I am finding that doing "Show latest versions without this line." sometimes will end up with a "File Not Found" result, but I presume that's due to cases where we end up referencing a revision in which the file did not exist with that name (or at all? I'm not sure about if we do rename-following?).

Comment 2

[Andrew Sutherland [:asuth] (he/him)]

This landed in time for Sunday's run and results seem good.

If one runs:

grep -c "Cache:HIT" /var/log/nginx/searchfox.log; grep -c "Cache:MISS" /var/log/nginx/searchfox.log

Like I just did and got:

354
3214

One sees the number of cache hits and the number of cache misses. So that's a roughly 10% hit-rate. And running:

sudo du -sh ~/index/nginx-cache

We get:

156M	/home/ubuntu/index/nginx-cache/

Which is interesting at least.

Comment 3

[Andrew Sutherland [:asuth] (he/him)]

Uh, and note that the changes included adding a header to simplify diagnosis of whether this stuff is working, etc. Note that we serve dynamic stuff with "cache-control: max-age=120" and no etag still (bug 1593375).

So if you look at the response headers on something that was a miss, you'll see:

x-cache-status: MISS

And if it's a hit:

x-cache-status: HIT

Comment 4

[Andrew Sutherland [:asuth] (he/him)]

Final tallies for Monday's indexer that's been rotated out which gives us a full day of all time-zones usage:

grep -c "Cache:HIT" /var/log/nginx/searchfox.log; grep -c "Cache:MISS" /var/log/nginx/searchfox.log
1379
9594