Closed Bug 439731 Opened 16 years ago Closed 15 years ago

Reloading of read messages when using IMAP account (enable disk cache)

Categories

(MailNews Core :: Networking: IMAP, enhancement, P2)

enhancement

Tracking

(Not tracked)

RESOLVED FIXED
Thunderbird 3.0b3

People

(Reporter: olaf, Assigned: rsx11m.pub)

Details

Attachments

(1 file, 1 obsolete file)

931 bytes, patch
Bienvenu
: review+
Bienvenu
: superreview+
Details | Diff | Splinter Review
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14
Build Identifier: version 2.0.0.14 (20080421)

Thunderbird doesn't cache IMAP messages between sessions.

Reproducible: Always

Steps to Reproduce:
1. Open Thunderbird with an IMAP account
2. Open a message - it is downloaded from server
3. Close Thunderbird
4. Open Thunderbird again
5. Open the same message as in step 2 - it is downloaded from server again 
Actual Results:  
The message is downloaded every time I want to see it.
It happens also without closing Thunderbird after watching several other messages.

Expected Results:  
The message is read from a disk cache.
This is the way e.g. Evolution/Outlook etc. works.

It looks that Thunderbird uses some small memory cache for IMAP messages and no disk cache.

This is a real showstopper for IMAP users.
I use 512 kbit/s link to connect to my company. For small messages it is just an annoyance but I often work with several megs attachments. It makes Thunderbird totally useless.

Waiting 2 minutes to open 3 MB message is a real pain if you know that it has been downloaded already.

When I work with big attachments it is even irritating on LAN.
Version: unspecified → 2.0
For IMAP, Thunderbird has a small memory cache but no disk cache by design. However, you could select the folders for offline use, thus the folder
contents is mirrored locally and available without additional connections
to the server (there is however bug 405437).

See http://kb.mozillazine.org/Offline_folders for details on this, and http://kb.mozillazine.org/Entire_message_fetched_when_opening_a_IMAP_message

If you nevertheless want Thunderbird to have a disk cache in addition to
that as an intermediate level between the memory cache and offline folders
for temporary caching of recently visited messages (also between sessions),
this would have to be confirmed as an enhancement request since it's not a
bug per se.

Moving from Thunderbird General to Networking: IMAP component.
Assignee: nobody → bienvenu
Component: General → Networking: IMAP
Product: Thunderbird → Core
QA Contact: general → networking.imap
Version: 2.0 → Trunk
We're planning on turning on offline caching by default in an other bug for 3.0
I couldn't find this, do you have the bug number?
Reply to Comment #1:
Setting folder for offline use is of no use for me.
I don't want to download all messages. This is hundreds of megabytes.

The thing I need is to cache already downloaded messages on disk. And have accass to them even if I close and open Thunderbird.

If this is an enhancement or a bug is simply matter of viewpoint in my opinion :)
David, if that bug in comment #2 is about caching recently downloaded messages
in the sense of Olaf's suggestion, this here would be a duplicate of such bug, otherwise it appears to be a valid RFE request.
(In reply to comment #2)
> We're planning on turning on offline caching by default in an other bug for 3.0

Found it, bug 385502.

Confirming as an enhancement request [Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.2pre) Gecko/2008071701 SeaMonkey/2.0a1pre]. This is different from bug 385502 in that only a small disk cache is requested to keep recently accessed messages and their attachments, not a complete copy of a folder for offline use. It is also different from bug 345832 on attachment preloading, a persistent cache is asked for to retain downloaded messages between sessions.
Severity: major → enhancement
Status: UNCONFIRMED → NEW
Ever confirmed: true
OS: Windows 2000 → All
Hardware: PC → All
Summary: Reloading of read messages when using IMAP account → Reloading of read messages when using IMAP account (want limited disk cache)
For Thunderbird, this depends on bug 450456 to get the disk cache included.
Adjusting bug summary after the disk cache is now included in Thunderbird
builds. Per bug 450456 comment #9, using that cache for IMAP would have to
be enabled in nsImapService::GetCacheSession(), currently only HTTP pages
are cached on disk. Maybe bug 345832 can be solved in the process.
Summary: Reloading of read messages when using IMAP account (want limited disk cache) → Reloading of read messages when using IMAP account (enable disk cache)
This sure could be a big win if it's not too much work.
Flags: wanted-thunderbird3+
Flags: blocking-thunderbird3?
Priority: -- → P2
I don't think we'd block on it, but it sounds like a good bug.
Flags: blocking-thunderbird3? → blocking-thunderbird3-
Keywords: helpwanted
Whiteboard: [good first bug]
Attached patch Simple patch — — Splinter Review
This looks (almost) trivial, but apparently that may be all what is needed.
It would change the policy setting for IMAP messages from STORE_IN_MEMORY
to STORE_ANYWHERE, thus removing the current restriction to keep them in non-persistent storage.

I've tested this on 64-bit Linux (for SeaMonkey, since it has a complete "about:cache" implementation to examine its contents). With offline folders disabled and browser.cache.memory.enable set to false, all cachable data - including IMAP messages - end up in the disk cache now. They remain accessible after closing and restarting SeaMonkey. Interestingly, images as part of an
HTML message are stored with the message in encoded form, whereas attachments are decoded and cached as separate entries (that's the same for memory too).

With browser.cache.memory.enable reset to true, IMAP messages are preferably stored on the disk cache, while browser.cache.disk.enable set to false
enforces keeping all data in memory. I didn't observe any unusual behavior
with IMAP-offline mode enabled again, so this appears to be transparent.

The same policy applies for IMAP which also applies for other entries (e.g., HTTP remote content). I don't think a separate configuration option is needed,
a user who has privacy concerns would likely disable both offline storage as well as the disk cache as a whole, both options are provided by the current
user interface and preferences.
Attachment #351024 - Flags: superreview?(bienvenu)
Attachment #351024 - Flags: review?(bugzilla)
this is not needed if we're storing imap messages in the offline store by default, which we are now.
Correct, but this is for cases where the user does not want offline storage being activated for a potentially large set of folders, but wants to utilize a limited disk cache for the most current messages anyway (comment #4).
I'm having a little trouble understanding all the modes, but the behavior I am looking for is offline-on-demand, meaning if I have displayed a body, I will be able to see it offline (and also not reload in another session).  If I have opened an attachment, I will be able to see it offline.  But I do not want the overhead of automatically downloading messages that I have not looked at.
Thus I see it as not a caching issue but a incremental offline issue.
If the user is using an offline store, then isn't turning on the disk
cache for imap messages going to mean that those messages are stored in the
disk cache and the offline store?

Hence this will potentially push out the web page data that we do want to use
the disk cache for.
Standard8, you're exactly right. So the proposed patch would make us double up every message you read, by default. We could tweak the patch so that it only uses the disk cache if we're not going to store the message in the offline store, or we could add a hidden pref to enable this non-default behavior.
(In reply to comment #16 and comment #17)
> If the user is using an offline store, then isn't turning on the disk
> cache for imap messages going to mean that those messages are stored in the
> disk cache and the offline store?

Good point, I'm unable though to verify that this is actually happening. I have emptied the disk cache, marked my account for offline storage, and waited until all messages were synced. Opening such messages, even if not read previously, did not increase the number of disk-cache items, nor did they show up in the detailed list. In contrast, new messages not yet downloaded were cached. Thus, it appears that this problem is only present to a certain extent (at least as far as I can tell from my bit of testing here). Do you see anything different?
Attached patch Alternative patch (obsolete) — — Splinter Review
Based on comment #17, the trade-off appears to be either letting the cache algorithm decide what's optimum or introducing another pref setting to let the user decide explicitly what to do. I'm posting this second patch as a possible solution for the latter option.

This version adds mail.imap.cache.disk.enable for the advanced user to activate disk caching, while retaining the memory-only default (I'd expect to see this "true" with offline storage disabled). Note that changing this pref won't be effective until a new IMAP cache is established (e.g., upon restart).

> We could tweak the patch so that it only uses the disk cache if we're not
> going to store the message in the offline store

Given that offline storage can be enabled or disabled for each individual IMAP folder, it may be tricky to determine the global cache behavior based on this. So, that option would call for a more complex solution than just modifying the cache policy...
A local copy in offline storage means it is never removed.
A disk cache means it is magically removed when some capacity limit is reached.

I think the best improvement here would be to just add to the offline behavior:
1. (as now) no offline storage, just whatever cache might be used.
2. (as now) Total download offline storage: download everything in folder
3. (new) As viewed/opened (body or attachment) offline storage.

Since there is already a button for "download now", what is needed is a second (mutually exclusive) option in the offline dialog: "Save all viewed messages and opened attachments for offline viewing".  Then the "download now" does the heavy lifting of downloading everything that has not already been saved due to viewing.
(of course the UI change is optional...it could be a preference).

Finally, it doesn't seem like there is any way to remove the offline storage copies for a folder. (separate feature request?).  If there was, then I think the "save for offline as viewed" could be the default...
(In reply to comment #17)
> Standard8, you're exactly right. So the proposed patch would make us double up
> every message you read, by default. 

Since 50M of mail is a bit much to get through, is that really a problem?

FWIW, I hope we don't have to have a pref for this. It's something people expect to just work.
I have confirmed the behavior described in comment #18 with a nightly Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b3pre) Gecko/20081203 SeaMonkey/2.0a3pre build, one folder of an account marked for offline use and another one not. Only messages not currently present in the offline storage end up in the (here) memory cache. Thus, it seems that an occasional redundancy of a new message fetched before it was downloaded for offline storage should not pose a big problem, I'd agree with comment #21 to prefer the simpler patch to just enable the disk cache without adding the overhead of a preference.

As for Jim's suggestion, finding a solution to improve offline storage beyond the current all-or-nothing scheme IMO may be useful (e.g., also specifying a maximum cap of the offline store, then cache-like removing least-used entries once that cap has been reached). I think though that this should be a separate RFE, involving quite some extensions to the offline implementation.
I actually think the "offline" concept should *not* be subject to capacity controls (which is indeed the current situation).  My only enhancement would be to be able to store offline as bodies and attachments are specifically viewed or otherwise retrieved.  If I have enabled a folder for offline, I expect it to eventually be a mirror of what is on the server.  The only missing feature is to do it incrementally rather than all at once.  My primary usage model is intermittently connected via 3g or evdo, where bandwidth is still scarce.  I *never* want my bandwidth used up by background downloading, but I also *never* want to download something I have already "paid for" by viewing.  When I happen to be connected on a high speed link, I want to be able to explicitly complete the total folder download.  I cannot do this today since explicit downloading only puts it in the offline store if I have enabled the folder for offine usage - which forces the total download whenever I connect!
(In reply to comment #23)
> I actually think the "offline" concept should *not* be subject to capacity
> controls (which is indeed the current situation).

Jim, if you read comment #4 and comment #7, you will notice that providing but limiting the volume of IMAP data mirrored locally is exactly what this bug had been filed for. If you want to change the behavior of the initial download to offline storage, I'd definitely suggest to file a separate enhancement request as it is a different scope than the discussion here.

> (comment #9) Maybe bug 345832 can be solved in the process.

Further testing shows that some issues mentioned there are resolved by the
disk cache without needing offline storage, also because of its larger volume compared with the smaller memory cache (4MB on Thunderbird/Shredder). Reopening a <1MB message I've read yesterday with both inline images and attachments was nicely grabbed from the disk cache, including when saving the attachments. Thus, worked as intended.

It still appears a bit arbitrary though how specific messages are cached and when they are revalidated, possibly because IMAP cache items don't have an explicit expiration date. Another test message with 8MB in size was revalidated first and recached on the first opening after restart, despite being in the cache, but then accessed from cache on the second shutdown/restart cycle. This may be part of follow-up work for bug 345832, as expiration/revalidation is in general independent of the location (disk or memory) of a cached item.
(In reference to comment 20)
>I think the best improvement here would be to just add to the offline behavior:
>1. (as now) no offline storage, just whatever cache might be used.
>2. (as now) Total download offline storage: download everything in folder
>3. (new) As viewed/opened (body or attachment) offline storage.

I agree and 3. is possible now (sort of). The new auto-sync feature supports strategies for downloading messages. You can exclude large messages or even all messages until you manually select them. However with default mime_parts_on_demand prefs messages containing non-inline attachments will not get stored in the offline folder. This leads to an unintuitive situation as a user where messages you have read and viewed attachments are not available to view offline.

IMO this problem is not *cleanly* solved by caching or auto-syncing folders by default. The problem is that we have a conflict between two strategies to minimise bandwidth usage:
- mime_parts_on_demand: download body (and inline parts) first then non-inline attachments on demand
- offline folders: only download messages once after which they are retrieved from the offline storage

(from comment 23)
>My only enhancement would be
>to be able to store offline as bodies and attachments are specifically viewed
>or otherwise retrieved.  If I have enabled a folder for offline, I expect it to
>eventually be a mirror of what is on the server.  The only missing feature is
>to do it incrementally rather than all at once.

So is there a way to have our cake and eat it too. Is it possible to both download mime parts on demand and have those same mime parts stored for offline folders? Or once all mime parts have been cached can we reconstruct the message?

I'm developing an extension called Incommunicado to make TB more friendly in unreliable/low-bandwidth conditions and would love to be able to solve this.
In hindsight my last post should probably have gone on bug 405437. Looking at the comments there and also for bug 345832 there are quite a few people (me for one) who would like offline storage to support partially downloaded messages.

I'm interested in developing this for my extension. Would it would be worth making this an RFE?
(In reply to comment #25 and comment #26)
I think the target audience for this bug is a user with a limited-bandwidth connection who has limited disk resources as well (e.g., with a laptop where using 7GB of redundant offline messages on a 40GB disk is not acceptable), thus both reducing server downloads and limit disk usage are relevant aims here.

If you find that your suggestions are not covered by any of the other bugs yet, you can/should file them as a new RFE, yes.
I will happily and clearly position my perspective as being limited-bandwidth, and NOT limited disk resources.  Any solution should clearly keep these issues distinct, and not combine them, in order to satisfy all three possible "constraint situations".  If there were simply to check-boxes in the UI:
1. Conserve bandwidth by ensuring that message contents are never transferred more than once.
2. Conserve disk space by caching only XXXMB of message contents.
3. Force download of all message contents when connected to server.

Then all the other complications would be moot.
I agree. This should be separate things.
Ok, I was trying to motivate a user scenario where one would be interested in both avoiding reloading of a message but also need to limit how much local space is dedicated to this.

While there may be a more comprehensive solution for comment #28 point 2 based on extending the offline implementation, this may not be available for the next major release due to additional work that would be involved. On the other hand, the disk cache is readily available now and could just be switched on as proposed, without preventing any larger solution in the future.
I agree with rsx11m (comment 30). Could we at least support a preference that will turn on disk based caching for TB3.0.

I plan to implement offline storage of mime parts for the Incommunicado extension I am working on. But I imagine it is going to be non-trivial and will not make the TB3.0 release.

For more info on the motivation and plan for the Incommunicado extension:

  https://wiki.mozilla.org/MailNews:Minimizing_Bandwidth_Usage
Mark, David: Can we give this a try for 3.0b2?
Product: Core → MailNews Core
Keywords: helpwanted
Whiteboard: [good first bug] → [has patch]
I wonder how it would impact online/offline filtering/searching if some messages are available and some not, in one folder.
I'd expect this to be transparent to any search, it shouldn't matter whether the cache is located in the memory or on the disk. If the search or filtering is conducted on the offline copies, the server (and therefore the cache) wouldn't be consulted at all. So, if something goes wrong here, it would be a bug independent of the question which type of cache is used.
Comment on attachment 351024 [details] [diff] [review]
Simple patch

Ping for reviews, this patch has been up for three months now.

If this is not wanted as the default behavior, then please consider/comment on attachment 351221 [details] [diff] [review] to make it optional.
Comment on attachment 351024 [details] [diff] [review]
Simple patch

so, no, I don't want to always use the disk cache...
Attachment #351024 - Flags: superreview?(bienvenu) → superreview-
Attachment #351024 - Attachment is obsolete: true
Attachment #351024 - Flags: review?(bugzilla)
Attachment #351221 - Flags: superreview?(bienvenu)
Attachment #351221 - Flags: review?(bugzilla)
Comment on attachment 351221 [details] [diff] [review]
Alternative patch

Ok, let's give the variant with a pref a try then.
Note that this is off by default.
Attachment #351024 - Flags: superreview- → review?(bienvenu)
Comment on attachment 351024 [details] [diff] [review]
Simple patch

actually, if we don't create cache entries when we're storing the message for offline use, then this may be fine. I'd need to double check that, though.
Attachment #351024 - Attachment is obsolete: false
Comment on attachment 351024 [details] [diff] [review]
Simple patch

it turns out that we do write messages to the mem cache even when storing them for offline use - this seems like a bug to me since we're never going to look in the memory cache if we have them in the offline store. If that bug turns out to be easy to fix, then I think I would be OK with this patch. I'd rather do that than introduce yet an other pref as the second patch does...
This looked fine during my testing per comment #22, but I'll leave the requests for both patches active until someone has figured that out then.
this comment in the code gives me a bit of pause:

http://mxr.mozilla.org/comm-central/source/mailnews/imap/src/nsImapProtocol.cpp#903

the implication is that setting security info on things in the disk cache is more dangerous somehow. Maybe bz has some thoughts?
I think it's just that at the time security info only worked in the memory cache because it was just an object.  Nowadays disk cache serializes and deserializes it, I think... Or at least we support disk caching of SSL.  Worth double-checking, though.
Comment on attachment 351221 [details] [diff] [review]
Alternative patch

>+    if (diskCache)
>+        rv = serv->CreateSession("IMAP-anywhere", nsICache::STORE_ANYWHERE, nsICache::STREAM_BASED, getter_AddRefs(mCacheSession));
>+    else
>+        rv = serv->CreateSession("IMAP-memory-only", nsICache::STORE_IN_MEMORY, nsICache::STREAM_BASED, getter_AddRefs(mCacheSession));
>+

I'm happy to go with David's judgment on this. Both patches are technically correct, however if this one gets submitted, then you'll want 2-space indentation, not 4.
Attachment #351221 - Flags: review?(bugzilla) → review?(bienvenu)
Thanks, formatting should be a trivial fix.

David, I assume that you are currently looking into the issues raised in
comment #39 and comment #41 and then decide one way or the other.
Comment on attachment 351024 [details] [diff] [review]
Simple patch

ok, let's give this a try.
Attachment #351024 - Flags: review?(bienvenu) → review+
Attachment #351221 - Flags: superreview?(bienvenu)
Attachment #351221 - Flags: superreview-
Attachment #351221 - Flags: review?(bienvenu)
Attachment #351221 - Flags: review-
Comment on attachment 351221 [details] [diff] [review]
Alternative patch

I'd rather not have an other hidden pref for this - but if the other patch has issues, we might have to go with this approach.
(In reply to comment #45)
> (From update of attachment 351024 [details] [diff] [review])
> ok, let's give this a try.

Thanks, is this ready to be checked in then or does it need another review?
Mark has approved it in comment #43 but didn't formally plus the patch.
Comment on attachment 351024 [details] [diff] [review]
Simple patch

yes, this is ready to go in...
Attachment #351024 - Flags: superreview+
Attachment #351221 - Attachment is obsolete: true
Assignee: bienvenu → rsx11m.pub
Status: NEW → ASSIGNED
Keywords: checkin-needed
Whiteboard: [has patch] → [c-n: Comment #47+, comm-central]
Checked in: http://hg.mozilla.org/comm-central/rev/aadcf678e456
Status: ASSIGNED → RESOLVED
Closed: 15 years ago
Keywords: checkin-needed
Resolution: --- → FIXED
Whiteboard: [c-n: Comment #47+, comm-central]
Target Milestone: --- → Thunderbird 3.0b3
Looking at the options in the b3 build, it doesn;t look like this is implemented yet... at least, not what Jim Kulp (and myself) would really like to see...

Could someone explain how this is supposed to work?

What I would consider the most sensible is instead of an all or nothing per folder option, have an 'on-demand' sub-choice...

Consider the following:

1. I have some IMAP folders that have many GB's of messages stored (I really do)

2. I do NOT want to have to download the entire folder when I set up my account on a new computer,

3. I want to only download messages/attachments once on any given PC I'm using

There really needs to be a sub-option under 'Tools > Account Settings > Synchronization and Storage', labeled something like 'On-Demand (only syncs messages when read/clicked on)'
if you set mail.server.default.autosync_offline_stores to false using options | advanced | config editor, and configure your folders for offline use, then you'll get basically on-demand storage for offline use.
Ah... ok, well, thats good for me...

But is there a good reason an option isn't provided in  the UI?

In fact, imho, this should be the DEFAULT behavior, not a hidden option. It could be further enhanced by some kind of 'First-use' detecion, which would prompt the user with a choice of syncing the entire folder (and maybe even give an estimate of how much time after the headers have downloaded), or just syncing messages on demand...
Note that there are also now bug 482476 and bug 506024 to establish policies based on disk space or age, and on download bandwidth for the autosync process. Those will limit the amount on offline storage used and how quickly new e-mails are synchronized (this would follow a most recently received scheme in contrast to a most recently used policy for the cache). Which set of options eventually will be represented in the user interface is probably a question of which ones cover the most frequently encountered cases.
Ok, thanks... I added an attachment to bug 482476 (see comment fifteen) of how the GUI could be modified to provide this as an option...
You need to log in before you can comment on or make changes to this bug.