480487 - if-modified-since not working

Status: RESOLVED INVALID

(Core :: Networking, defect)

Description

[scott schmitz]

User-Agent:       Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10_5_6; en-us) AppleWebKit/525.27.1 (KHTML, like Gecko) Version/3.2.1 Safari/525.27.1
Build Identifier: 3.06

I have an ajax application which has been optimized to use if-modified-since to minimize the data traffic.  Firefox used to do the if-modified-since for my AJAX requests as designed.  But now, all it does is make the standard request.  IE7 works fine.

Here's the AJAX request:
https://www.scottschmitz.com/subscription/AJAX/GetList.php?userID=37&databaseName=Contact&format=Contact

and this is the response my server returns:
HTTP/1.x 200 OK
Date: Fri, 27 Feb 2009 04:48:16 GMT
Server: Apache
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: age=3600, private, must-revalidate
Etag: "851d51305a74417c5a0cfab39cf43689"
Last-Modified: Fri, 27 Feb 2009 04:19:22 GMT
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/xml; charset=UTF-8

I have tried changing header to:
Cache-Control: age=3600, public, must-revalidate
Pragma:cache

but that made no difference


You can repeat this using a test server by going to the URL:
https://www.scottschmitz.com/secure/SignIn.php
sign in using:
[e-mail]
test1

Click on the Contacts button after you have signed in.  The contacts will load from AJAX data.  Click on the Contacts tab at the left to reload that screen.  The second time the AJAX data loads the browser should make a not modified request to my server.  But doesn't.  I used Live HTTP headers to get the above header info.

Normally the application uses https, but you can reload the page (after signing in) using http as well.  In both cases the AJAX request does not do the if-modified-since thing.




Reproducible: Always

Steps to Reproduce:
see above instructions - easy to repeat.
Actual Results:  
makes a standard request

Expected Results:  
browser should realize that the URL that is being requested in is its cache and then make an if-modified-since request to the server.  But it does not.

Comment 1

[Frank Wein [:mcsmurf]]

What do you mean with "Firefox used to do the if-modified-since for my
AJAX requests as designed"? Do you mean this worked fine with an earlier version of Firefox?

Comment 2

[scott schmitz]

Firefox used to send the if-modified-since request when it had the data cached and http headers were included just so.  It no longer does this.  Instead, it simply makes the request again, regardless of the contents of its cache. 

Caching has always been finicky, with specific headers required to be set just so.  So, I suspect that there was a change within the last 12 months which changed how caching is handled in regards to if-modified-since.

The easiest way to see what I am talking about is to take a look at the example and turn on Live HTTP headers.  Do you have some kind of regression test for validating that if-modified-since works?

Comment 3

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #0)
> Date: Fri, 27 Feb 2009 04:48:16 GMT
> Expires: Thu, 19 Nov 1981 08:52:00 GMT

> http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.2.4
> 13.2.4 Expiration Calculations
>(snip)
> The max-age directive takes priority over Expires,
> so if max-age is present in a response, the calculation is simply:
>      freshness_lifetime = max_age_value
> Otherwise, if Expires is present in the response, the calculation is:
>      freshness_lifetime = expires_value - date_value

Because no "Cache-Control: max-age=xxx", cache expires anytime, unless date of user's PC is set "1981/11/19" or before. 

> Cache-Control: age=3600, private, must-revalidate
> I have tried changing header to:
> Cache-Control: age=3600, public, must-revalidate

scott schmitz(bug opener), are you testing cache control extension of "age=..."? (IE supports it?)
If not, check with cache directives defined in HTTP 1.1 only, please.
Explicit max-age in Cache-Control header is requested in some of HTTP 1.1 mechanism. Sorry but I don't know whether "no max-age" relates to "must-revalidate" processing or not.

> http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9
> 14.9 Cache-Control
>    Cache-Control   = "Cache-Control" ":" 1#cache-directive
>    cache-directive = cache-request-directive
>         | cache-response-directive
>    cache-request-directive =
>           "no-cache"                          ; Section 14.9.1
>         | "no-store"                          ; Section 14.9.2
>         | "max-age" "=" delta-seconds         ; Section 14.9.3, 14.9.4
>         | "max-stale" [ "=" delta-seconds ]   ; Section 14.9.3
>         | "min-fresh" "=" delta-seconds       ; Section 14.9.3
>         | "no-transform"                      ; Section 14.9.5
>         | "only-if-cached"                    ; Section 14.9.4
>         | cache-extension                     ; Section 14.9.6
>     cache-response-directive =
>           "public"                               ; Section 14.9.1
>         | "private" [ "=" <"> 1#field-name <"> ] ; Section 14.9.1
>         | "no-cache" [ "=" <"> 1#field-name <"> ]; Section 14.9.1
>         | "no-store"                             ; Section 14.9.2
>         | "no-transform"                         ; Section 14.9.5
>         | "must-revalidate"                      ; Section 14.9.4
>         | "proxy-revalidate"                     ; Section 14.9.4
>         | "max-age" "=" delta-seconds            ; Section 14.9.3
>         | "s-maxage" "=" delta-seconds           ; Section 14.9.3
>         | cache-extension                        ; Section 14.9.6
>    cache-extension = token [ "=" ( token | quoted-string ) ]

> http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9.6
> 14.9.6 Cache Control Extensions

Comment 4

[scott schmitz]

OK.  I have added the 'max-age' directive.  No change in behavior.

I also tried removing the 'age' directive with 'max-age' in its place and also removing the 'must-revalidate' directive as well.  Under no condition can I get the browser to make the if-modified-since request.

This used to work perfectly.  Perhaps you have a regression test for validating if-modified-since behavior?  I wonder if this is broken?

These headers currently work fine under IE7 and Safari.

I also did a about:config call in the browser and filtered by the keyword cache.  I made sure that all those settings were default value.


Below are some sample headers:

HTTP/1.x 200 OK
Date: Mon, 02 Mar 2009 04:18:04 GMT
Server: Apache
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: age=3600, max-age=3600, public, must-revalidate
Etag: "4ea232caab185a58e20b6078f183de47"
Last-Modified: Sat, 28 Feb 2009 21:50:00 GMT
Keep-Alive: timeout=5, max=99
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/xml; charset=UTF-8

Comment 5

[scott schmitz]

I did one additional test and added an expires header far into the future.  By adding an expires header far into the future, the information was cached and, for some reason, was never checked for validity at all.  So, the must-revalidate cache-control directive was ignored.  

My understanding of the specification is that max-age and must-revalidate would always override expires header.  But this is not what I found.

Comment 6

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #0)
> I have an ajax application which has been optimized to use if-modified-since to minimize the data traffic.

> Etag: "851d51305a74417c5a0cfab39cf43689"
> Last-Modified: Fri, 27 Feb 2009 04:19:22 GMT

Since Etag: is returned by your server, client can issue conditional request of If-None-Match instead of If-Modified-Since.
> http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.19
> 14.19 ETag
> http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.26 
> 14.26 If-None-Match

Firefox doesn't issue GET with If-None-Match? 

(In reply to comment #5)
> By adding an expires header far into the future, the information was cached
> and, for some reason, was never checked for validity at all.

Do you mean max-age=3600(1 hour) on Cache-Control: was ignored by Firefox?
What expiration is set in about:cache for the URL?
(Disk Cache Device, List cache entries)

Comment 7

[scott schmitz]

I did some more testing.  With returned headers of:

Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: age=3600, max-age=3600, public, must-revalidate
Etag: "4ea232caab185a58e20b6078f183de47"
Last-Modified: Sat, 28 Feb 2009 21:50:00 GMT

and a request URL of:
http://www.scottschmitz.com/forms/TBuyersForm.html?Buyer:::Buyers:37

there was no entry in the disk cache.  

On a hunch, I changed the Expires header to 1 hour into the future.  Once I did that, the item showed up in the cache.  However, Firefox never asked the server if the data was fresh, it just used it.

So, I see 2 problems:
1. max-age should override Expires, but does not
2. must-revalidate directive is being ignored

Comment 8

[scott schmitz]

I should mention that all of these requests to the server are via AJAX.

Comment 9

[scott schmitz]

I did some more research and found that there was a mention of Firebug causing interaction with headers.  I turned off firebug and everything worked fine!

My final headers were:

Cache-Control: private, must-revalidate
Etag: "8d990001296a7c134422fcb8758a9f52"
Last-Modified: Mon, 02 Mar 2009 22:11:38 GMT
Content-Type: text/xml; charset=UTF-8

which works fine for IE7, Safari and FireFox.

Comment 10

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #9)
> My final headers were: (snip)

No max-age/no Expires:? If so, Mozilla family usually holds data in cache longer time than Web developer's anticipation, due to current Heuristic Expiration algorithm. See Bug 328605 and bugs pointed in it. I recommend you to set explicit/appropriate max-age(and Expires: if required) always.

Comment 11

[scott schmitz]

Yes, I see.  What I want is for the data to be cached, but to be always revalidated, even if the data is only a few second old.  Given that condition,  If I am reading:

http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html

>> The request includes a "max-age=0" cache-control directive, 
>> which forces each cache along the path to the origin server to revalidate its own entry,
>>  if any, with the next cache or server.

Cache-Control: private, must-revalidate, max-age=0

might be the best solution?

Comment 12

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #7)
> 1. max-age should override Expires, but does not

I've met Bug 203271.
You are probably looking very old Bug 203271. (opened on 2003-04-24!!!).
I think "with max-age(=0)/without Expires:" is best solution for Firefox currently, although "no Expires:" may produce other problems on old browsers which don't support HTTP 1.1.

Comment 13

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #8)
> I should mention that all of these requests to the server are via AJAX.
(In reply to comment #9)
> there was a mention of Firebug causing interaction with headers.
> I turned off firebug and everything worked fine!

A problem with Firefox+AJAX+Firebug (Load incomplete, Reload required) has been reported to Bug 482293. If you still need Firebug and if you can easily test Bug 482293 with your site, join in Bug 482293, please.

Comment 14

[[e-mail]]

I noticed that with our own server, Firefox does not use if-modified-since correctly. It appears that for small files it simply request the file again without using if-modified-since although the file
is cached on the browser side. For .gif files, it was working properly, but not for .css and .js files.
It seems that 33KB is the magic number: below that, Firefox would not use if-modified-since, above it, Firefox would use it. (I have Firebug installed with my browser.)

Comment 15

[Patrick McManus [:mcmanus]]

comment 9