User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.224 Safari/534.10 Build Identifier: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:220.127.116.11) Gecko/20101206 Ubuntu/10.04 (lucid) Firefox/3.6.13 I'm a web developer, and I'm experiencing a cache issue with Firefox at any platform (Windows, Mac and Linux), while other browsers tested (IE6 - WinXP, Safari - Mac and Chrome - Mac and Linux) are caching perfectly. I verified the issue after explicitly setting the Cache-Control header to max-age=BIG_NUMBER_HERE and verified that the Expires header was pointing to a very far date in the future. Then, I accessed the webpage and using firebug, I verified that the header was indeed right, and I got a 200 for the given content. If I navigate the website (clicking links, I'm not doing any sort of refresh or reload of the page), I can verify once again that the same request is made, and a 200 response is retrieved instead of just fetching the content from cache. I verified the request headers, and they don't mention max-age=0. I checked my about:config, and here is what I've got: http://i56.tinypic.com/ephtmf.png Here is a sample of HTTP header that is facing this sort of problem: http://i51.tinypic.com/25yu0ev.png and http://i55.tinypic.com/ndsyvm.png Both samples I got while navigating the website for the second time, using links inside the page. Same methodology applied at the other browsers mentioned and they managed to cache perfectly. If I go to my about:cache, this is what I see: http://i53.tinypic.com/2czeutw.png which is the exact match of the second content's header above. Even though, it doesn't make use of this cache because I can see the request through firebug, and I can see the request arriving at the webserver. The same doesn't occur with the other webbrowsers mentioned. Reproducible: Always Steps to Reproduce: 1. Serve a webpage with the headers I demonstrated 2. Access the page 3. Navigate to that page again, and verify that it isn't using the content from cache Actual Results: Firefox isn't respecting the cache-control headers, and it ends up making unecessary requests to the server, decreasing user experience by slowly loading the content that is already in cache. Firefox, instead, sends a non-conditional get request and getting a 200 everytime. Expected Results: Firefox should grab the content from its cache and pop it on the screen without making any request to the server. In the worse acceptable case, should send a conditional get request, accepting a 304 as response and then using the cache. I tested firefox with and without the BetterCache extension, and I could verify that BetterCache partially fixed the caching issue, by enforcing firefox to cache a lot of contents that wasn't cached before. Even with this extension, firefox isn't caching as much content as the other browsers are caching without any extension. I even tested this behavior in a firefox without firebug, to make sure it isn't its fault, and I verified the same amount of requests in my webserver.
Adriano, is there a public server that shows the problem that you could point me to? If not, can you please create an HTTP log following the instructions at https://developer.mozilla.org/en/HTTP_Logging while doing as few page loads as possible while still reproducing the problem? Then please attach that log to this bug using the "Add an attachment" link and mention which URI you expected to be cached that wasn't.
Oh, one more thing. The screenshot at http://i51.tinypic.com/25yu0ev.png shows us making a conditional request; the server is responding with a 200, not a 304... But again, an HTTP log would likely give us all the information (including the request sequence, and all the relevant headers) to figure out what's going on here.
I thought that a conditional request must send a Last-Modified header with the Last-Modified value the browser has from its cached content. Otherwise, how is the webserver supposed to know if the content was modified or not and return a 304? Perhaps I'm just missing something here. What is it in that request header that makes it conditional?
> I thought that a conditional request must send a Last-Modified header There are two kinds of conditional requests in HTTP for our purposes (there's a third one, but it can't result in a 304). If all a client has to work with is a date, then it can send an If-Modified-Since header. The server should then send a 304 if the resource has not changed since that date. But if a client has an ETag (which if you look at those headers it does) for the resource then it can send an If-None-Match header with that ETag in it. The server should then send a 304 if the ETag it has matches one of the ETags in the If-None-Match header. If the ETags don't match, then the resources are different and the server must respond with a 200 and send the new data.
Thanks for clarifying this! I didn't know about these other headers. I just setup a server where you'll be able to reproduce this issue: http://adriano.burble.net You don't really need to login to figure this out, but you can create an account if you want to. Please, let me know if I can help you further with this testing.
OK, so how do I reproduce the bug? I tried loading the page, clicking the "log in" link, clicking the "about" link, etc, and things look correct here (some things are cached; some are not due to Vary headers the site sends, that sort of thing). What file are you expecting to be cached that isn't and what should I be loading, in what order, to reproduce the problem?
What headers are screwing with the cache? Chrome is caching everything, but the html. Same behavior with the other browsers I tested. Everything that comes with a Cache-Control header of max-age=big_number should be cached.
> What headers are screwing with the cache? The only one I saw in my testing was Vary (for the stylesheets). In particular, the cookies for the site kept changing as I loaded different pages, and the stylesheet responses have "Vary: cookie" in them. But again, I'm not sure which urls you think should be cached but are not, so it's hard to answer your question... Would be easier if you would answer mine. As for "max-age=big_number", the stylesheets served by that site come with a 1-day max-age (and those Vary headers). None of the other responses had a max-age at all, though some of the HTML had that string in Set-Cookie headers, like so: Set-Cookie: sessionid=1abdffa27116d9b30dfb9a70cca53538; expires=Wed, 09-Feb-2011 20:52:10 GMT; Max-Age=1209600; Path=/
It is supposed to cache all css, js and images. Don't you think that even with the 1-day max-age it should cache for at least one day?
Yes, except the server is sending a Vary header with that CSS, with "Cookie" listed as one of the values. And then it's setting different cookies on differenet pages. So since the Cookie header we send is different, we can't use the cache entry, because the server told us that the CSS it returns depends on the cookies (by putting "Cookie" in the Vary header). _Does_ your server return different CSS depending on the cookies? If not, why is it setting that Vary header?
No, it doesn't. Try this new address instead: http://adriano.burble.net:8080/ I think that perhaps apache is messing up with the headers. While reproducing the issue, I wasn't running through apache. What I did was to set a max-age for all css, js and image files to make sure they're all cached. So, once you get to that address, if you see any of those not being cached, then this is the issue I'm talking about. Then, if you test with chrome, you'll verify that they're all cached as expected.
I managed to remove the Vary headers from static files, and now firefox is caching more aggressively, but still not caching everything. I just updated http://adriano.burble.net:8080 to help you verify this. Anyways, shouldn't the Cache-Control have a higher priority over the Vary header? I may be missing something here again, but if the server is saying that this content should be caches for some time, can't firefox presume that this includes even the varying content? I know it isn't sane to send both, but perhaps it is more logical to have the Cache-Control to take precedence over the Vary header. Besides, you may try to navigate further in the website using the user boris and passwd boris.
> Anyways, shouldn't the Cache-Control have a higher priority over the Vary No. Please see the HTTP spec. > if the server is saying that this content should be caches for some time, > can't firefox presume that this includes even the varying content? No. The time thing says that for that amount of time if you make the same request the response will be the same. The Vary says what makes a request not the same. If a header listed in Vary changed, that's equivalent to requesting a different URI altogether from the point of view of caching. Maybe I'm not making myself clear. What I want from you is a list of step-by-step instruction (load uri X, click link Y, click link Z, whatever) and for each one a list of which files you think should be cached but aren't. Then I can investigate what the issue is. Telling me "oh, some stuff is not cached that should be, but I won't tell you which" is not particularly useful....
Thanks, that's the sort of information I was looking for. If I load that url, then focus the url bar and hit "enter", I see a single HTTP request for jquery-ui-1.8.2.custom.min.js during the whole process. On the second pageload, I see it loaded from cache.
Ok, but do you see it using from cache if it is called by the html that is loaded when you access http://adriano.burble.net:8080 ? Moreover, can you confirm that you see all files that fits in that description being cached? I just reproduced here once again, and I saw about half of the files being used from cache, and half being requested everytime I navigate to the page mentioned.
> Ok, but do you see it using from cache if it is called by the html that is > loaded when you access http://adriano.burble.net:8080 ? Yes, that's exactly when I see it loading from cache. > Moreover, can you confirm that you see all files that fits in that description > being cached? Yes. I see 2 HTTP requests for the toplevel page itself, which is served with no useful cache headers and a Date of "right now", and 2 HTTP requests for favicon.ico. I see only one HTTP request for every single other resource. Note that I'm testing this with the current Firefox 4 beta, though. Are you testing with Firefox 3.6?
No, I'm not using Firefox 4 beta. I'm using the one I mention in this issue description. It is the first comment, in the User Agent section. Do you have a firefox 3.6 for testing?
Of course. ;) I did just try that; I see the same behavior as in comment 17.
That's weird. I'm trying the same page using firefox in different machines (a Mac, a Linux and a Windows), and also got two other people to test it and we consistently got the same behavior. Could it be some sort of configuration at your side that is different than ours? Is there anything else that firefox considers while deciding on whether to use data from cache or not?
Adriano, did you try this with a new profile? It may be that the cache in the profile you are using has stale data-entries (yes, there are a number of issues with this...) from the old versions of your test.
Adriano, I created a brand-new profile to test with..... > Is there anything else that firefox considers while deciding on whether to use > data from cache or not? Other than the headers and the state of the cache, no? But the headers can be affected by add-ons, etc, and the state of the cache might depend on previous browsing behavior, of course.
I asked another friend (who doesn't work at the same place I do, and haven't accessed this site previously) and also created a new profile. With both, we managed to get everything cached as expected. What could be causing this then? The media directory changes name everytime the server restarts, so why is old cache (pointing to a different location) is affecting the way new content is fetched/cached? Is there something I can do with my headers to overcome this issue?
> What could be causing this then? I don't know. What's different between the two profiles?
(In reply to comment #23) > I asked another friend (who doesn't work at the same place I do, and haven't > accessed this site previously) and also created a new profile. With both, we > managed to get everything cached as expected. Meaning that it works now..? :) > What could be causing this then? The media directory changes name everytime the > server restarts, so why is old cache (pointing to a different location) is > affecting the way new content is fetched/cached? Does the URL used by the browser change also (because this is what the browser uses to generate the cache-key) ? > Is there something I can do with my headers to overcome this issue? Not quite sure which issue you mean... However, it seems to me like you have managed to tune various headers (with substantial help from bz) to do what you want, no? If somebody have stale/bad entries in their caches, they will experience suboptimal performance for some time, until the cache is cleared or those entries are evicted. I think this is inevitable and not a big issue.
Hm... That's weird, because I tried the following: 1 - My firefox in a Mac, with my cache, profile, etc. and it didn't cache. Then, I was clearing the cache, and it still wasn't caching anything. 2 - My firefox in a Linux, same test as before, tried to clear my cache several times and it didn't work 3 - A clean firefox in a Win XP machine, I didn't clear the cache, but have never accessed the site from there before. It didn't work either. In the windows machine there is no extension installed. In the Mac and Linux, there is Firebug, YSlow and Live HTTP headers. The url always change. The media location in the filesystem is always the same, it is only that url that keeps changing. Eg. /site_media/static_1101251233/ changes to something like /site_media/static_1101261435 I agree that I can mess up with Firefox, and that doesn't mean that firefox is got an issue. But what keeps me thinking is why other browsers were doing fine with the cache while firefox wasn't. Besides, what would be your recomendation to avoid the stale cache? Should I always set a max-stale header, or something else?
Uhh... are you saying that this doesn't work anyway? Were any of the tests from comment #26 using the same profile which apparently worked in comment #23?
No, I was talking about what I tried before and that wasn't successful at all.
could you help me understand how my firefox profile could get into a state that would stop it caching correctly? creating a new profile seemed to resolve this issue for me but i'd like to know how this happened. It won't be acceptable for my users to keep having to create new profiles, i'd like to understand if this was a one-off issue or if this is something that might reoccur.
It might be a good start to answer bzs question in comment #24. If clearing the caches doesn't work for you, something else must be different,
In the old profile I had: * Aptana Debugger * ColorZilla * Email This! * Firebug * FireLogger * Flash Video Downloader * Header Spy * HttpFox * Image Zoom * Live HTTP Headers * Measure it * Modify Headers * SenSEO * SQLite Manager * Web Developer * Xmarks * YSlow And the new profile, I didn't have any extension installed. It was brand new. Two other colleagues experienced the same behavior, but they probably didn't have all those extensions. Is that what you wanted to know, or am I missing something that could help you solve this issue?
> 6.) Set your computer's clock ahead by one day. Once you do that, your computer and the server no longer agree on the time, and "has time X passed?" calculations suddenly depend on whose time you're measuring from. It looks like RFC 2616 has an algorithm for trying to handle this situation and we just don't implement it. That's a bug, but it's not THIS bug unless Adriano is also purposefully screwing up his clock. I filed bug 640445 to cover this issue. > I am *not* sure if I leave the clock alone and simply let a day pass if the > issue will still occur. It shouldn't.
No, I wasn't tweaking my clock. That wasn't necessary to reproduce the issue. John, do you think you can reproduce that without screwing the clock?
Here is the temporary file I was using yesterday for testing (this will be deleted once you guys have had enough time to test so I'll just leave it up for a few days or so)... (with www) jabcreations.com/scripts/test.js I loaded it this morning (maybe 12 hours later) and the first request was to my cache, then a HTTP 304 though it wasn't a constant 304 per request...it was pulling from my cache locally as I would hope.
Created attachment 544025 [details] Testcase for Vary header cache handling Requires PHP-enabled www server.
I have this (or very similar) problem on Firefox 5, on Linux. Did a bunch of testing, it seems to boil down to `Vary' header being mishandled by HTTP cache. Testcase attached in the comment above (sorry, no experience with bugzilla :P) The testcase makes five HTTP requests to the same URI, differing only in `X-Nnn' header. The www server returns different content depending on the X-Nnn header. This is indicated by the `Vary' header. It's best to observe the HTTP communication via Firebug or similar. Actual result: response: X-Nnn: text/foo response: X-Nnn: text/bar response: X-Nnn: text/bar response: X-Nnn: text/bar response: X-Nnn: text/bar First two requests are performed OK, but subsequent ones are served from cache with invalid content -- disobeying the Vary header. Expected result: first open (or Ctrl+F5) of the test page: response: X-Nnn: text/foo response: X-Nnn: text/bar response: X-Nnn: text/frob response: X-Nnn: text/knob response: X-Nnn: text/q-p subsequest open of the test page -- same output, but served from HTTP cache: response: X-Nnn: text/foo response: X-Nnn: text/bar response: X-Nnn: text/frob response: X-Nnn: text/knob response: X-Nnn: text/q-p
dexen, that seems unrelated to this bug. Could you please file a separate bug on that issue?
Could you try this with a recent nightly? I can not reproduce this with my local build, nor with a downloaded FF5. Hence, I guess I'm doing something wrong... (Your script loads the uri "/foobar" which I changed to make the script load itself - is this correct?)
Status: UNCONFIRMED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.