279379 - use squirm to make app update service rdf more cacheable

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[(not currently active) Ted Mielczarek]

Firefox currently appends a random number to the App Update Service RDF urls to
defeat RDF caching, which currently defeats squid caching also.  (bug 279335) 
We should use squirm to rewrite these URLs, removing the random number and
relying on standard HTTP caching.

squirm: http://squirm.foote.com.au/
specifically: http://squirm.foote.com.au/squirm-1.23.tgz

my squirm.conf:
--
begin
network 0.0.0.0/0
pattern squirm.patterns get
end
--

my squirm.patterns:
--
abort .html
abort .jpg
abort .png
abort .gif
abort .php

regex  ^http://rawr.homeip.net:8000/~luser/redir/(.+\.rdf)
http://rawr.homeip.net:8000/~luser/redir/\1
^http://rawr.homeip.net:8000/~luser/redir/
--

Basically, quit immediately if you get a request for one of the specified
extensions.  Otherwise, anything that matches that regex gets replaced by the
second string.  The third string is the "quick match" string, which gets
compared via a normal string compare before doing regex matching, to further
speed up the process.

excerpt from my squid access.log:
--
1106413978.196     12 127.0.0.1 TCP_MISS/200 1309 GET
http://rawr.homeip.net:8000/~luser/redir/contents.rdf - DIRECT/24.229.107.58
application/rdf+xml
1106413991.233    682 127.0.0.1 TCP_REFRESH_HIT/304 324 GET
http://rawr.homeip.net:8000/~luser/redir/contents.rdf - DIRECT/24.229.107.58 -
1106414290.618    887 192.168.1.107 TCP_MEM_HIT/200 1315 GET
http://rawr.homeip.net:8000/~luser/redir/contents.rdf? - NONE/- application/rdf+xml
--

You might have to tweak squid.conf also, mine had these two lines in by default:
--
acl QUERY urlpath_regex cgi-bin \?
no_cache deny QUERY
--
which would seem to deny cacheability of any URL with a ? in it.

Comment 1

[Dave Miller [:justdave]]

OK, we have squirm installed on update now, and it works marvelous with a few
caveats.

cross-domain rewrites don't appear to work.  Squid continues to send the
original hostname the user requested to the destination server, regardless of
what we rewrite it to (we're in httpd_accellerator mode).

However, the regexp to lop the query string off the end of the rdf files is
working great, and appears to have dropped memory consumption from 6.8 GB down
to 1.8 GB.  We're now caching one copy for each language code instead of 256. :)

Comment 2

[alanjstr]

Can you paste or attach the config file so we can investigate the rewrite?

Comment 3

[Dave Miller [:justdave]]

squirm.conf:
------
begin
network 0.0.0.0/0
pattern squirm.patterns GET
end
------

squirm.patterns:
------
#regex ^http://update http://george.foobar/
#regex ^http://update\.mozilla\.org:8080/update/firefox/(.+\.rdf)\?
http://aus.mozilla.org:8080/update/firefox/\1
^http://update.mozilla.org:8080/update/firefox/
#regex ^http://update\.mozilla\.org:8080/update/firefox/(.*)
http://aus.mozilla.org:8080/update/firefox/\1
^http://update.mozilla.org:8080/update/firefox/
regex ^http://update\.mozilla\.org:8080/update/firefox/(.+\.rdf)\?
http://update.mozilla.org:8080/update/firefox/\1
^http://update.mozilla.org:8080/update/firefox/
regex ^http://update\.mozilla\.org:8080/plugins/(.*)
http://pfs.mozilla.org:8080/plugins/\1 ^http://update.mozilla.org:8080/plugins/
regex ^http://aus\.mozilla\.org:8080/update/firefox/(.+\.rdf)\?
http://aus.mozilla.org:8080/update/firefox/\1
^http://aus.mozilla.org:8080/update/firefox/
------

all of the stuff commented out was tried at one point or another.  the
george.foobar thing was an unsuccessful attempt to break it (it didn't break).