533144 - cache not fully used and about:cache reports wrong sizes

Status: RESOLVED INCOMPLETE

(MailNews Core :: Networking, defect)

Description

[Eric Moore]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2b4) Gecko/20091124 Firefox/3.6b4
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.5) Gecko/20091130 Thunderbird/3.0

I'm using Thunderbird 3.0 RC2. I have no offline folders and message synchronizing is disabled. I used about:cache?device=disk in the start page to report whats in the disk cache. It never gets more than 1/4 full and about:cache reports the wrong sizes. 

About:cache says:

Number of entries: 	880
Maximum storage size: 	40960 KiB
Storage in use: 	10198 KiB
Cache Directory: 	C:\Users\eric\Profiles\Thunderbird\tanstaafl\Cache

It contains hundreds of "keys".

Windows Explorer reports:

Size: 9.78 MB (10,258,513 bytes)
Size on disk: 9.92 MB (10,412,032 bytes)
Contains: 72 files, 0 folders

10198 KiB does not equal either size reported by windows explorer.

I had expected each entry to be stored in a separate file. How are entries mapped to files?

I have read more than enough messages to fill the cache but it seems like there is some hidden limit on how much of the cache gets used for IMAP messages. I have very few HTML messages so most of the cache is being wasted.

I opened a small plain text message in a remote folder and then opened its 7651 KB PDF file attachment. I closed it, opened another small plain text message and then selected the prior message and opened the attachment again. It fetched both the message and the PDF file again. I exited Thunderbird and restarted it so I could look at about:cache. It hadn't changed.

Even if it didn't save it in the disk cache I would have expected it to save it in the memory cache. I thought that was increased for 3.0.

What can I do to get better use of the disk cache by my IMAP account without turning on message synchronization?







Reproducible: Always

Comment 1

[Ludovic Hirlimann [:Usul]]

About:cache is something that comes from Firefox, and probably doesn't take into account the way Thunderbird uses cache. This would explain your numbers.

>What can I do to get better use of the disk cache by my IMAP account without
>turning on message synchronization?

Good question , can you elaborate a bit more on your use case ? (ie why no auto-sync while still wanting to have your emails cached)

Comment 2

[Wayne Mery (:wsmwk)]

(In reply to comment #0)

> What can I do to get better use of the disk cache by my IMAP account without
> turning on message synchronization?

Comment 3

[David :Bienvenu]

about:cache reports disk cache usage; it shouldn't need to know how TB is using it to report the correct size.

thunderbird won't put messages in the cache that aren't fully downloaded. If you turn off mime_parts_on_demand, that will make clicking on a message with external attachments (like pdf files) download the whole message, and put it in the disk cache. You might try toggling mail.server.default.mime_parts_on_demand

Comment 4

[rsx11m]

This may be related to the observations in bug 529210 on loading of MIME parts into cache not finishing up correctly, and bug 345832 comment #78 on an unclear expiration policy for cached IMAP content. Either way, setting this preference to false now gets a longer initial download as the full message is grabbed, but it's still in the cache (and retrieved from there) when revisiting the same message shortly after the initial visit.

Comment 5

[rsx11m]

Eric, is this still reproducible on trunk after bug 565852 was fixed and with some initial patches for bug 531033?

Comment 6

[Eric Moore]

I don't know. I've stopped using both the disk cache and offline folders. 

I just ran into a problem with both 3.1.6 and 20101128 Thunderbird/3.3a2pre where despite mail.imap.mime_parts_on_demand being set false it displayed "This body part will be downloaded on demand." for two small HTML attachments in a message. That seemed due it to also having a 3356KB power point attachment that was incorrectly (no idea why) designated as Content-Type: application/octet-stream; . I don't see this problem in a message with a 86KB ZIP attachment, one with a 280KB JPEG attachment and another with a 15.6MB PDF attachment for example. 

I want to figure out whats going on with it first so that I can accurately predict which messages should make it into the cache. Once I do I'll retest using a disk cache for a couple of days.

Do I need to add mail.server.default.mime_parts_on_demand (see comment 3) to the config editor and set it false? I don't have any server specific mime parts on demand settings  (despite this example where it actually used mime parts on demand) so I don't understand why it exists and whats its relationship to mail.imap.mime_parts_on_demand .

Comment 7

[rsx11m]

(In reply to comment #6)
> I just ran into a problem with both 3.1.6 and 20101128 Thunderbird/3.3a2pre

The bug 565852 patches were only checked into trunk so far, bug 531033 was checked in for 3.1.7pre nightly builds as well.

> Do I need to add mail.server.default.mime_parts_on_demand (see comment 3) to
> the config editor and set it false?

For the purpose of this test, leave mail.server.default.mime_parts_on_demand on its default "true" value to see the effect of these patches. I'm actually not sure if mail.imap.mime_parts_on_demand is doing anything (I've always set the server-default pref only).

Comment 8

[WADA:World Anti-bad-Duping Agency]

If Tb's Disk Cache is similar one to Fx/Mozilla's "Disk Cache", and if Necko's code is used for it, Bug 175600 may be applicable to Tb's Disk Cache too.
> Bug 175600 Only 8192 objects (entries) can be stored in disk cache.
Eric Moore(bug opener), how many files do you see in Tb's Cache directory?

FYI.

As stated in bug 105843, Mozilla's "Cache" is conceptually similar to real cache of CPU or swap file for paging which is for performance/response-time while running. It's never for offline-use like usage, even though data in Disk Cache can be used after next restart, if shutdown fortunately completed normally/successfully.
> Bug 105843 Cache lost if Mozilla crashes

After fix of Bug 212316, number of reports like Bug 212251 declined very very much, and number of complaints added to Bug 105843 declined, because Cache loss by other than System Crash/Power Failure is reduced by fix of Bug 212316 .
> Bug 212251 Cache cleared if Windows is restarted / rebooted / shut down while mozilla is open
> Bug 212316 Mozilla must handle WM_ENDSESSION message to cleanly unload in case of exiting or restarting windows

Comment 9

[Eric Moore]

I haven't had a chance to do any retesting yet. However I always had less than a thousand objects. 

I was not interested in using it as a offline folder, I was looking for increased performance. I did not get it because it tried to optimize the wrong use case. Most of my messages are plain text and almost all of the messages that I re-read are plain text. The most noticeable performance hit I run into is fetching small attachments. 

Comment 1 asked " can you elaborate a bit more on your use case ? (ie why no
auto-sync while still wanting to have your emails cached)"

The classic IMAP model of remote folders that can be accessed from multiple clients does a good job of meeting my needs. I have multiple PCs (though I mainly use one) and sometimes want to access my mail from remote locations without carrying a flash drive.

The overhead and complexity of auto-sync buys me nothing since I have no interest in reading stuff offline and I don't need it as a backup solution. I also have no interest in global search since even though I have 10 IMAP accounts I organize almost all of my mail into a nested hierarchy of folders in my main IMAP account. I almost always know exactly what folder has the messages I'm looking for and the IMAP server has good performance searching a single remote folder.

Things that would help me is a better memory cache and some way to tweak it since I sometimes bop back and forth a lot between a small number of messages, and a disk cache that caches the messages that I most often reread. Thunderbirds disk cache doesn't appear to learn what I re-read and it seems optimized to cache a browsers web pages.

Comment 10

[Wayne Mery (:wsmwk)]

this should be retested on trunk. feel free to reopen with those results.

about:cache issues need to reported against mozilla Core:Networking