21313 - Need to cache document charset

Status: VERIFIED FIXED

(Core :: Networking: Cache, defect, P1)

Description

[bobj]

Bob Jung wrote:

  Do you have a spec or some sorta doc on your cache work?
  For I18N, we would like to cache the charset last used to render a page.
  We talked about it, but never did this in previous (4.x and before) browsers.

There is a very detailed description of the cache API, in the form of the
commented IDL files (attached).  However, the information that you're
talking about (charset) is at the protocol-level or above, which is at a
higher level than the the cache manager deals with.  It is possible to
store opaque information in each cache entry to be associated with a URL
using the setProtocolPrivate() API.  It sounds like this is what you
would want to do.

-Scott

See:
http://lxr.mozilla.org/seamonkey/source/netwerk/cache/public/nsINetDataCacheMana
ger.idl
http://lxr.mozilla.org/seamonkey/source/netwerk/cache/public/nsICachedNetData.id
l

Comment 1

[Scott Furman]

See bug 20939, which also addresses the issue of storing URL metadata in the
cache.

Comment 2

[Scott Furman]

Note that the cache method name was changed to SetAnnotation() from
SetProtocolPrivate().

Comment 3

[leger]

Bulk move of all Cache (to be deleted component) bugs to new Networking: Cache
component.

Comment 4

[bobj]

See bug 20939 for an attachment which contains
    "code to store image dimensions in cache (untested)"

We should be able to study that for the implementing charset caching.

Comment 5

[Teruko Kobayashi]

Added beta1 in keywords.

Comment 6

[bobj]

Reassigned to jbetak for M15.

Comment 7

[leger]

Putting on [nsbeta2+][5/16] radar.  This is a feature MUST complete work by 
05/16 or we may pull this feature for PR2.

Comment 8

[bobj]

Added perf keyword

Comment 9

[leger]

Putting on [nsbeta2-] radar. Missed the Netscape 6 feature train.  Please set to 
MFuture.

Comment 10

[[e-mail] (away - not reading bugmail)]

This is really easy to do - it pretty much maps whatever we did for the 
bookmarks. The only difference is that we do not need to change the cache code 
to achive document charset persistency in the cache. The cache service provides 
a neat interface (getAnnotation, setAnnotation) for opaque attributes 
associated with a URL.

http://lxr.mozilla.org/seamonkey/source/netwerk/cache/public/nsICachedNetData.id
l#162

All we need to do is to get the cache service in nsHTMLDocument.cpp (look how 
we are getting the bookmarks service), use the document URL as key to access 
the stored doument charset. Once a new document charset has been determined in 
nsHTMLDocument.cpp we need to store it using the URL as key. That's pretty much 
it. I already included a new member in the enum struct in nsIParser.h to 
reflect the neccessary priority for the charset determination algorithm in 
nsHTMLDocument.cpp.

Comment 11

[Frank Tang]

performance issue. mark nsbeta3

Comment 12

[Frank Tang]

set it to P1 M18

Comment 13

[Frank Tang]

nsbeta3+ per i18n bug meeting. this may also help 45408

Comment 14

[Frank Tang]

take out ETA.
I probably can now work on this since other important problem are all waiting 
for some other stuff.

Comment 15

[Frank Tang]

Here are some example how to get the nsINetDataCacheManager and nsICachedNetData

from 
http://lxr.mozilla.org/seamonkey/source/netwerk/protocol/jar/src/nsJARChannel.cp
p#383
381     nsCOMPtr<nsINetDataCacheManager> cacheMgr;
382     nsXPIDLCString jarBaseSpec;
383     nsCOMPtr<nsICachedNetData> cachedData;

423     cacheMgr = do_GetService(NS_NETWORK_CACHE_MANAGER_PROGID, &rv);
424     if (NS_FAILED(rv)) goto error;
425 
426     rv = mJARBaseURI->GetSpec(getter_Copies(jarBaseSpec));
427     if (NS_FAILED(rv)) goto error;
428 
429     rv = cacheMgr->GetCachedNetData(jarBaseSpec, nsnull, 0, 
nsINetDataCacheManager::CACHE_AS_FILE,
430                                     getter_AddRefs(cachedData));

Comment 16

[Frank Tang]

Here is the patch http://warp/u/ftang/tmp/fix21313.txt

Index: src/nsHTMLDocument.cpp
===================================================================
RCS file: /m/pub/mozilla/layout/html/document/src/nsHTMLDocument.cpp,v
retrieving revision 3.263
diff -u -r3.263 nsHTMLDocument.cpp
--- nsHTMLDocument.cpp  2000/08/18 00:47:00     3.263
+++ nsHTMLDocument.cpp  2000/08/18 08:09:05
@@ -103,6 +103,8 @@
 #include "nsIDocumentCharsetInfo.h"
 #include "nsIDocumentEncoder.h" //for outputting selection
 #include "nsIBookmarksService.h"
+#include "nsINetDataCacheManager.h"
+#include "nsICachedNetData.h"
 #include "nsIXMLContent.h" //for createelementNS
 #include "nsHTMLParts.h" //for createelementNS
 
@@ -486,6 +488,7 @@
     }
   }
 
+  nsCOMPtr<nsICachedNetData> cachedData;
   nsresult rv = nsDocument::StartDocumentLoad(aCommand,
                                               aChannel, aLoadGroup,
                                               aContainer,
@@ -501,6 +504,10 @@
   
   nsAutoString lastModified;
   nsCOMPtr<nsIHTTPChannel> httpChannel = do_QueryInterface(aChannel);
+
+  PRBool bTryCache = PR_FALSE;
+  PRUint32 cacheFlags = 0;
+
   if (httpChannel) 
   {
     nsXPIDLCString lastModHeader;
@@ -572,6 +579,22 @@
       }
     }
 
+    if(kCharsetFromCache > charsetSource )
+    {
+      PRUint32 loadAttr = 0;
+      rv = httpChannel->GetLoadAttributes(&loadAttr);
+      NS_ASSERTION(NS_SUCCEEDED(rv),"cannot get load attribute");
+      if(NS_SUCCEEDED(rv) )
+      {
+         // copy from nsHTTPChannel.cpp
+         if(loadAttr & nsIChannel::CACHE_AS_FILE)
+             cacheFlags = nsINetDataCacheManager::CACHE_AS_FILE;
+         else if(loadAttr & nsIChannel::INHIBIT_PERSISTENT_CACHING)
+             cacheFlags = nsINetDataCacheManager::BYPASS_PERSISTENT_CACHE;
+         bTryCache = PR_TRUE;
+      }
+    }
+
     // Don't propogate the result code beyond here, since it
     // could just be that the response header wasn't found.
     rv = NS_OK;
@@ -778,6 +801,9 @@
     nsXPIDLCString scheme;
     aURL->GetScheme(getter_Copies(scheme));
 
+    nsXPIDLCString urlSpec;
+    aURL->GetSpec(getter_Copies(urlSpec));
+
     if (scheme && nsCRT::strcasecmp("about", scheme) && (kCharsetFromBookmarks 
> charsetSource))
     {
       nsCOMPtr<nsIRDFDataSource>  datasource;
@@ -786,8 +812,6 @@
            nsCOMPtr<nsIBookmarksService>   bookmarks = 
do_QueryInterface(datasource);
            if (bookmarks)
            {
-                 nsXPIDLCString urlSpec;
-                 aURL->GetSpec(getter_Copies(urlSpec));
 
                  if (urlSpec)
                  {
@@ -805,6 +829,34 @@
        }
     } 
 
+    if(bTryCache)
+    {
+       nsCOMPtr<nsINetDataCacheManager> cacheMgr;
+       cacheMgr = do_GetService(NS_NETWORK_CACHE_MANAGER_PROGID, &rv);       
+       NS_ASSERTION(NS_SUCCEEDED(rv),"Cannot get cache mgr");
+       if(NS_SUCCEEDED(rv) && urlSpec)
+       {
+          rv = cacheMgr->GetCachedNetData(urlSpec, nsnull, 0, cacheFlags, 
getter_AddRefs(cachedData));
+          NS_ASSERTION(NS_SUCCEEDED(rv),"Cannot get cached net data");
+          if(NS_SUCCEEDED(rv)) {
+              bTryCache = PR_TRUE; // since we can get cachedData;
+              if(kCharsetFromCache > charsetSource) 
+              {
+                nsXPIDLCString cachedCharset;
+                PRUint32 cachedCharsetLen = 0;
+                rv = cachedData->GetAnnotation( "charset", &cachedCharsetLen, 
+                      getter_Copies(cachedCharset));
+                if(NS_SUCCEEDED(rv) && (cachedCharsetLen > 0)) 
+                {
+                   charset.AssignWithConversion(cachedCharset);
+                   charsetSource = kCharsetFromCache;
+                }
+              }    
+          }
+       }
+       rv=NS_OK;
+    }
+
     if (kCharsetFromParentFrame > charsetSource) {
       if (dcInfo) {
         nsCOMPtr<nsIAtom> csAtom;
@@ -856,6 +908,13 @@
 
   rv = this->SetDocumentCharacterSet(charset);
   if (NS_FAILED(rv)) { return rv; }
+
+  if(bTryCache)
+  {
+       rv=cachedData->SetAnnotation("charset",charset.Length()+1,
+               NS_ConvertUCS2toUTF8(charset.GetUnicode()));
+       NS_ASSERTION(NS_SUCCEEDED(rv),"cannot SetAnnotation");
+  }
 
   // Set the parser as the stream listener for the document loader...
   if (mParser) 

Comment 17

[Frank Tang]

fix and check in.

Comment 18

[Tom Everingham]

verif.

Comment 19

[Pulsebot]

Pushed by ffxbld-merge:
https://hg.mozilla.org/releases/mozilla-beta/rev/7cbda623cccd
[fenix] For https://github.com/mozilla-mobile/fenix/issues/21313: renewing performance awesomebar metrics (https://github.com/mozilla-mobile/fenix/pull/21315)
https://hg.mozilla.org/releases/mozilla-beta/rev/015a86d7ca8d
[fenix] For https://github.com/mozilla-mobile/fenix/issues/21313: Renew fission metrics expiring in December
https://hg.mozilla.org/releases/mozilla-beta/rev/9c39d991b9e2
[fenix] For https://github.com/mozilla-mobile/fenix/issues/21313: Renew metrics for December - never expire updates (https://github.com/mozilla-mobile/fenix/pull/21788)
https://hg.mozilla.org/releases/mozilla-beta/rev/262b38923085
[fenix] For https://github.com/mozilla-mobile/fenix/issues/21313: Remove expiring/unused metrics for December (https://github.com/mozilla-mobile/fenix/pull/21789)
https://hg.mozilla.org/releases/mozilla-beta/rev/923d8c5c90ce
[fenix] For https://github.com/mozilla-mobile/fenix/issues/21313: Renew product telemetry expiring in December (https://github.com/mozilla-mobile/fenix/pull/21316)