470624 - On-demand message AutoSync

Status: ASSIGNED

(MailNews Core :: Backend, enhancement)

Description

[Gerry Healy]

User-Agent:       Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.0.5) Gecko/2008120121 Firefox/3.0.5
Build Identifier: Thunderbird 3.0b1

Implement on-demand auto-sync (a great oxymoron) to provide more control over when syncing takes place.

Reproducible: Always




The default behaviour of auto-sync is to pre-emptively download *all* messages in __offline__ folders. This may be undesireable in unreliable and low-bandwidth conditions or if you are paying by the kilobyte. I am developing an extension (Incommunicado) that will give a user more control over the use of bandwidth and thus syncing. Using auto-sync strategies most of the features can be implemented but I will also need the ability to trigger auto-sync on-demand. More information can be found here:

  https://wiki.mozilla.org/MailNews:Minimizing_Bandwidth_Usage
  https://wiki.mozilla.org/MailNews:Better_Faster_IMAP_Plan

I've spoken to Emre (auto-sync developer) and he has proposed the following implementation:

- Add a new operation mode (lets say on-demand) to nsAutoSyncManager.
- Add new method to nsAutoSyncManager to explicitly tell to re-prioritize the queue.
- When running in on-demand mode, it should ignore idle events.
- When running in this mode, nsAutoSyncManager should keep downloading all pending messages until either it is all done, or the user explicitly says STOP.

Comment 1

[Gerry Healy]

I'll be working on this from today (21 December 2008) and I'm hoping to to have a patch committed for the Thunderbird 3.0b2 release. What are my chances and is there anything I have to do other than submit the patch?

Comment 2

[Magnus Melin [:mkmelin]]

Yes, you should request review (and superreview) for the patch.
See http://developer.mozilla.org/en/docs/Getting_your_patch_in_the_tree

Once it has reviews, add the checking-needed keyword.

Comment 3

[David :Bienvenu]

How would this be different from the existing download and sync now, which syncs all folders configured for offline use, and it is cancellable?

Comment 4

[Gerry Healy]

The existing 'download/sync now' behaviour doesn't support strategies, which means you will get inconsistent behaviour between auto-sync and "download/sync now". I will also need to sync on-demand using strategies to get the "only download what I ask for and only download it once" behaviour I am aiming for in the Incommunicado extension. 

I made a post about 'download/sync now' behaviour to the dev-apps-thunderbird mailing list, but didn't get a reply so ended up contacting Emre directly to discuss it. In our offline discussion it also came up that because auto-sync is quite effective the "download/sync now" menu item may be removed from TB3.

http://groups.google.com/group/mozilla.dev.apps.thunderbird/browse_thread/thread/4c3927d9894b857b

I'll summarise our offline discussion here when I next have the opportunity.

Comment 5

[Gerry Healy]

Reading through the emails I think I've already summarised the most relevant discussion. What may be of interest is why I seem to be trying to take the auto out of auto-sync.

Here in Nepal I often have to deal with low-bandwidth (dial-up) and unreliable internet connections (frequent dropped packets). To deal with this I want to limit the messages that Thunderbird auto-syncs and manually 'Sync Now' when I have a better internet connection. Although at first this seemed like an edge case for myself (most people in Nepal use webmail) I can see from posts to related bugs (e.g.  bug 439731 #28 and bug 345832 #73) that it is probably an issue for a significant number of people and may even increase in relevance with the uptake of netbook/smart phones on expensive per kb data plans.

I was originally planning to add features similar to auto-sync strategies to the old offline syncing code, however since it has already been implemented very cleanly in auto-sync it seemed wise to build on it. Unfortunately using auto-sync this way means that the above mentioned bugs, which were worked around somewhat by auto-sync, raise their ugly heads again.

I'm planning to tackle bug 405437 'IMAP mail are not cached for offline use if message size is bigger than mail.imap.mime_parts_on_demand_threshold' after this which I hope will have the knock on effect of fixing bug 439731 and bug 345832.

Comment 6

[Gerry Healy]

Attachment: WIP patch v1 on rev 1564

This patch implements a fairly solid on-demand sync that can optionally override the auto-sync folder and message strategies.

To test it you will also want to download incommunicado-dev-20090109-tb.

NOTE: it currently produces a lot of debug output if DEBUG_me is defined

Comment 7

[Gerry Healy]

Attachment: WIP incommunicado-dev-20090109

This is a very early dev version of the Incommunicado extension. It provides preferences to limit the max message size downloaded by auto-sync (note currently only global settings are honoured).

To test it you can add a "Sync Now" button to the toolbar that allows you too test on-demand sync using auto-sync strategies (Sync Now) and with not max message size (Sync All Now).

Comment 8

[Gerry Healy]

Some problems with the current patch:
1) if on-demand sync is cancelled before completion it can leave the auto-sync queues in a logically invalid state
2) on-demand sync doesn't pick up messages TB doesn't already have headers for

The current implementation requires auto-sync to know quite a bit about on-demand sync. With the user-mode level implemented in bug 471208 it will hopefully be possible to solve these problems *and* implement on-demand sync more simply.

Comment 9

[Gerry Healy]

Apologies

I haven't got much further with this patch and probably won't for quite a while. I'm living in Nepal and we currently only have a few hours of power per day. It won't get any better before the monsoon in May/june.

I have ported the code on top of bug 471208, but it still feels a bit like trying to fit a square block into a round hole in that auto-sync is designed to work in the background and failure is non-critical whereas on-demand sync should really operate more in the foreground (at least the user should know when it has finished and if there were any errors).

I'll post the patch as-is as soon as I can so someone else can look at it if they are interested.

Comment 10

[Dan Mosedale (:dmosedale, :dmose)]

I suspect the activity manager and pop-tarts UI pieces that making their way into the tree would be likely to help with the interaction design here.  Thanks for the work you have done on this, and I have to say that you've got a very compelling reason for not continuing to make progress.  :-)