279335 - Reduce load on Update server by using less-random URL

Status: RESOLVED FIXED

(Toolkit :: Application Update, defect)

Description

[alanjstr]

When checking for application updates, nsUpdateService appends a random number.
 Instead, it should append something that will be cachable by the server. 
Perhaps YYYYMMDD.  Then it would only have to be cached once per day.

Comment 1

[alanjstr]

Attachment: Use YYYYMMDD from current date instead of random

Comment 2

[Brian Ryner (not reading)]

I'm not really familiar with u.m.o, so bear with me.  What is the random key
used for currently?

Comment 3

[Dave Miller [:justdave]]

(In reply to comment #2)
> I'm not really familiar with u.m.o, so bear with me.  What is the random key
> used for currently?

The files being retrieved from the server are static files in a directory, so
the random key does absolutely nothing on the server except clutter the log
files and prevent squid from caching them.

I'm told the random key is added within Firefox to prevent Firefox from using a
cached copy of the file, because the RDF API apparently doesn't allow you to
override the cache status.  Unfortunately it has exactly the same effect on the
squid caching proxy, which is adding unecessary load to the server actually
hosting the files.

Comment 4

[alanjstr]

Attachment: Use javascript instead of vbscript

/me blushes

Comment 5

[Mike Shaver (:shaver emeritus)]

Is there a reason we're not just adjusting the squid server's no_cache ACL to
not suppress caching of _all_ requests with ? in them?  Seems like something we
could probably do a pretty good job matching to our environment, and worth the
few hours of effort.

(I'd recommend not making that change to the source unless we determinine that
squid can't be made to do what we want here.  Which would surprise me, and even
then, let's just fix RDF finally.)

Comment 6

[alanjstr]

justdave implemented squirm because Firefox is DDoSing UMO this week.  But by
using squirm, he had to undo the domain split for now.  

As for suppressing caching of all requests with a ? in them ... you do realize
that the rest of the website is dynamic, right?  

Comment 7

[Mike Shaver (:shaver emeritus)]

(In reply to comment #6)
> As for suppressing caching of all requests with a ? in them ... you do realize
> that the rest of the website is dynamic, right?  

Were you addressing me?  I specifically and clearly suggested turning _off_ the
default squid config which _already_ suppresses caching of requests with ? in
them.  I think that suppression unwise for many aspects of the site, most
especially the update requests that are melting us down right now.

(You don't need ? in the URL for dynamism, of course, but that's an entirely
different architectural question.)

Comment 8

[Dave Miller [:justdave]]

Squid is and has been, since it was set up, configured with all of the no_cache
stuff ripped out.  It caches EVERYTHING, including URLs with query strings on
them, as the site would never be able to handle the load otherwise.

The issue with this bug is that by using a random number on the end of the URL,
squid is now caching hundreds of copies of each of those RDF files instead of
just one, which means it's hitting the host application server hundreds of times
instead of just once every time the cache expires, and also eating up all the
available memory on the server trying to keep all the "frequently accessed
files" in memory (which it can't, because there are too many of them).

However, we seem to have completely mitigated the problem on the server with the
use of Squirm to rewrite the query string out of the URL before squid sees it
for rdf files.  Eliminating the domain name split had nothing to do with
implimenting Squirm, and was only done as an additional load reduction action
since each client is now hitting the server once instead of three times (two of
them getting 301 responses) trying to get the files. (And squid doesn't cache
301s at all, so every one of those first two hits was hitting the host
application server)

Comment 9

[Daniel Veditz [:dveditz]]

Comment on attachment 172098 [details] [diff] [review]
Use javascript instead of vbscript

resolved server-side, not wanted on the branch

Comment 10

[Brian Ryner (not reading)]

Comment on attachment 172098 [details] [diff] [review]
Use javascript instead of vbscript

cancelling, if this is not needed

Comment 11

[alanjstr]

We'd still like to reduce the load on the servers.

Comment 12

[Axel Hecht]

To give an RDF API comment, as I was asked on my blog (?).

Is this an HTTP cache problem or an issue with RDF having singleton datasources?

We are currently thinking about redoing the singleton rdf data source story, or
rather, dropping it. This is a rather involved story, though, so I'm not 100%
confident that it wouldn't break stuff. This is not going to be done without
some research and a good deal of announcements to break as few extensions as
possible.

Comment 13

[Darin Fisher]

nsUpdateService.js could easily talk to the Necko cache to delete the en-US.rdf
file before loading the RDF datasource.  That would avoid needing the '?' in the
URL, and it would also avoid the need to modify the RDF subsystem.

Comment 14

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Comment on attachment 172098 [details] [diff] [review]
Use javascript instead of vbscript

I can't seem to just clear the review request since bugzilla goes crazy; but
-'ing as I believe ben rewrote most of this code.

Comment 15

[Benjamin Smedberg]

The RDF subsystem caches the datasource irrespective of the network cache, which
I don't think is the issue here.

Comment 16

[Axel Hecht]

The RDF/XML singleton bug is bug 297810, which turned out to be a good deal more
tricky than I hoped.

Comment 17

[Chase Phillips]

Assuming this bug is filed against the trunk, the new update service uses an
equally new schema for the update URL.  This URL contains no random string. 
This bug could be a candidate for ->INVALID at this point.

Comment 18

[Chase Phillips]

Or ->FIXED. :)

Comment 19

[Darin Fisher]

I think we can mark this fixed given the new update system.