831025 - Provide way to use HTTP cache for etag/if-modified checks, but without storing data

Status: RESOLVED WONTFIX

(Core :: Networking: Cache, defect)

Description

[Jason Duell]

The b2g folks want to use etags as a way to check if app manifests have changes, but w/o storing the data of the manifest (it can be large).  Their current hack is to tell the cache not to store (INHIBIT_CACHING), but look at the etag header the server sets, and then manually set the if-etag-modified header on the next HTTP request (they get a 304 if the server replies that the content is not modified).

We could encapsulate this model within necko more cleanly by providing a new flag (can't think of a good name yet: INHIBIT_CACHE_DATA?) that tells the HTTP cache to not store any data, just the last-modified and etag info.  Clients would then pass the (or another new) flag to following HTTP requests for the same URI, and the channel would only return data if the resource was modified, otherwise it would return an empty 304 request.

Michal says this doesn't require cache changes, just HTTP channel mods:  create a new cache session for the data-less entries (so they don't conflict with regular requests for the same URI, which expect normal semantics), and then store only the headers, not the data, in cache entry.  And return empty 304 status for not-modified resources.

Not a blocker--the current hack appears to be working fine.  I'm not sure how high to prioritize this, actually--we can live w/o it so the risk is more that we'll forget about it...

Main risk in change to doing this in necko is that when cache gets corrupt (10% or so of the time currently: we'll see how often it happens for B2G), we'll get a thundering herd of manifest requests that will be fully downloaded.  Fabrice is not worried about it, though.

Comment 1

[Honza Bambas (:mayhemer)]

Sounds good, I had similar idea.  The patch could be simple, however it may need some patching when we fail to open the data stream to load from the net (bug 812483?)

Advantage here is that the conditioning code doesn't need to be duplicated somewhere in JS and conditional requests will be handled 100% correctly by Necko using the existing code.  That is always good.

INHIBIT_PAYLOAD_CACHING ?

Comment 2

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to Jason Duell (:jduell) from comment #0)
> The b2g folks want to use etags as a way to check if app manifests have
> changes, but w/o storing the data of the manifest (it can be large).  Their
> current hack is to tell the cache not to store (INHIBIT_CACHING), but look
> at the etag header the server sets, and then manually set the
> if-etag-modified header on the next HTTP request (they get a 304 if the
> server replies that the content is not modified).

I think their existing implementation is reasonable. If they already have this working, why do we need to do anything new?

Comment 3

[Honza Bambas (:mayhemer)]

(In reply to Brian Smith (:bsmith) from comment #2)
> I think their existing implementation is reasonable. If they already have
> this working, why do we need to do anything new?

They are duplicating code for issuing conditional HTTP request.  That is actually the only concern.  This request may be coming from my comment https://bugzilla.mozilla.org/show_bug.cgi?id=823648#c7 where I was trying to suggest rather use flaging of xhr's http channel then manually building conditional request.  I was not aware one of the requests was to not store the data it self.  If there is no cache entry then the B2G OWA manifest checking code is working as expected even when no etag/last-mod headers have not been provided (we won't try to count expiration time and potentially prevent server request).

Comment 4

[Patrick McManus [:mcmanus]]

there are separate efforts underway to allow storing stuff in the cache other than response entities (tied to the meta data of the response entitiy)