310656 - Firefox redownloads CSS styles every time I visit this site (i.e. cache is malfunctioning) - optimize the 'freshness' calculation to prevent 'very short freshness' issue

Status: RESOLVED WONTFIX

(Core :: Networking: Cache, enhancement)

Description

[Artem S. Tashkinov]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9a1) Gecko/20050921 Firefox/1.6a1
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9a1) Gecko/20050921 Firefox/1.6a1



Reproducible: Always

Steps to Reproduce:
1. Open http://slashdot.org in the trunk version of firefox
2. FF will load CSS style files from images.slashdot.org
3. Close FF
4. Again open http://slashdot.org 
Actual Results:  
FF will load CSS styles files from images.slashdot.org though the CSS style
files must be cached

Expected Results:  
FF uses its cache instead of reloading CSS style files.

Comment 1

[Jo Hermans]

Could you create a log as described at
http://www.mozilla.org/projects/netlib/http/http-debugging.html and attach it to
this bug?

possibly bug 290032

Comment 2

[Artem S. Tashkinov]

Actually I understood what caused this FF behaviour:

I opened about:cache?device=disk and I saw this (other slashdot CSS files have
the same entries):

           Key: http://images.slashdot.org/print.css
     Data size: 533 bytes
   Fetch count: 4
 Last modified: 2005-10-01 13:37:08
       Expires: 2005-10-01 18:50:04

So CSS style files are redownloaded every next day. 
BUT:

telnet images.slashdot.org 80
Trying 66.35.250.55...
Connected to images.slashdot.org.
Escape character is '^]'.
GET /print.css HTTP/1.1
Site: images.slashdot.org

HTTP/1.1 200 OK
Date: Sat, 01 Oct 2005 10:02:38 GMT
Server: Boa/0.94.14rc17
Accept-Ranges: bytes
Connection: Keep-Alive
Keep-Alive: timeout=10, max=1000
Content-Length: 1441
Last-Modified: Thu, 29 Sep 2005 03:27:03 GMT
Content-Type: text/css

No expiration date is set in HTTP headers. That makes me wonder why FF decides
that CSS files "Expires: 2005-10-01 18:50:04".

Anyway I'm attaching the log file you have requested.

Comment 3

[Artem S. Tashkinov]

Attachment: Firefox NSPR_LOG

This is the log file I got by
$ export NSPR_LOG_MODULES=nsHttp:5,nsSocketTransport:5,nsHostResolver:5
$ export NSPR_LOG_FILE=/tmp/log.txt
$ firefox

Comment 4

[James Ross]

If no expires is specified, a date is calculated based on the last modified of
the document, which I believe results in the document expiring about 1/10th of
it's age into the future (so if the document was last modified 10 hours ago,
Firefox would make it expire in 1 hour - Darin will correct me here). The
comment here shows the logic for calculating the "freshness lifetime":

http://lxr.mozilla.org/seamonkey/source/netwerk/protocol/http/src/nsHttpResponseHead.cpp#266

Comment 5

[Darin Fisher]

Yeah, this is working as designed.  See RFC 2616 where the Last-modified based
expiration time calculation is prescribed.

Comment 6

[Alfred Kayser]

RFC2616 says:
  "Also, if the response does have a Last-Modified time, the heuristic
   expiration value SHOULD be no more than some fraction of the interval
   since that time. A typical setting of this fraction might be 10%."
So, it 'might be' 10%, but possible more, but not more than the 'interval since
that time', so may be instead of just doing 10%, do 100% for the very short
intervals so prevent very very short freshness settings?

define age = 'interval since that time'
if (age < 1 day) freshness = age              // 100% of age
else if (age < 10 days) freshness = 1 day     // between 10 and 100%
else freshness = age/10                       // 10% age

the 'SHOULD be no more' even allows to have a minimum freshness threshold:
   if freshness < 1 hour freshness = 1 hour     
to prevent very short freshness 

Note, that this only applies to the
     // Fallback on heuristic using last modified header...
at
http://lxr.mozilla.org/seamonkey/source/netwerk/protocol/http/src/nsHttpResponseHead.cpp#300

Comment 7

[Darin Fisher]

Alfred: Yeah, I have often thought about improving our Last-modified heuristic.
 Something progressive probably does make the most sense.

Comment 8

[Artem S. Tashkinov]

Darin: From you point of view I can't understand whether the accident I have
reported is a bug or a feature request. May I hope someone will work on this in
order to improve caching policy? Maybe it's worth implementing simple rules for
caching different type of documents? I.e. don't cache HTML or alike documents
and use cache as much as possible for the types other than HTML.

Comment 9

[Alfred Kayser]

To clarify a bit, the plan is to comply with the RFC standards.
According to Darin the cache is working as it should (according to the standards). 

What seems to happen is that some files (such as css's) are so shortlived that
they disappear from the cache very quickly. What I propose to do, is to
fine-optimise the 'freshness' calculation within the bounds of the specification
to prevent this 'very short freshness' a bit.

Comment 10

[Darin Fisher]

confirming, and reassigning to alfred ;)

Comment 11

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #0)
> 2. FF will load CSS style files from images.slashdot.org
> 3. Close FF
> 4. Again open http://slashdot.org 

What is you cache option setting?
If "When the page is out of date"(browser.cache.check_doc_frequency=3, default), I think problem is simply that "Heuristic Expiration" is larger than one user is convinced, and "improving" by Darin Fisher in Comment #7 may work.
But if "Once per session"(browser.cache.check_doc_frequency=0), problem of Bug 221036 can also be involved because your scenario has restart step of FF.

Comment 12

[Elmar Ludwig]

No, this is actually the opposite case of bug 221036:

This issue is about extending the heuristic expiration time in specific cases (e.g. as described in comment #6) in order to keep files without an explicit expiration time longer in the cache than we do now. Bug 221036 is a request to not use (or revalidate) a cached resource under the same circumstances when it is first accessed in a session.

Setting severity to enhancement (this is definitely not major).

Comment 13

[Artem S. Tashkinov]

(In reply to comment #11)

> What is you cache option setting?

I have browser.cache.check_doc_frequency=3 in my settings. BTW, as for #1 - closing FF is not necessary at all. Just reopening ./ in around an hour makes FF redownload all CSS and JS files.

Comment 14

[Patrick McManus [:mcmanus]]

not going to mess with the heuristic over one site when explicit options are available to it.