Closed Bug 534072 Opened 15 years ago Closed 14 years ago

Automatic synchronization with IMAP servers by default cause recursive downloading of all messages

Categories

(MailNews Core :: Backend, defect)

Product:
MailNews Core
Component:
Backend
Version:
1.9.1 Branch
Platform:
x86
Windows XP
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED DUPLICATE of bug 569161

People

(Reporter: stathis, Unassigned)

Details

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5 (.NET CLR 3.5.30729)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091204 Thunderbird/3.0

Right after upgrading to 3.0 and using existing IMAP accounts, Thunderbird starts by default to recursively download and index all messages on the server, causing massive bandwidth consumption and on my maildir, which is above 10GB, effectively makes the client unusable as I have to wait for all messages to be downloaded.

I tried to disable the synchronization and indexing features, but I was unsuccessful:

(a) I unchecked Tools > Options > Advanced > General > Advanced Configuration > Enable Global Search and Index
(b) For every IMAP account in Tools > Account Settings > %account > Synchronization & Storage, I unchecked the "Keep messages for this account on this computer"

None of the two options seems to have stopped the process, even restarting the client did not help. 

I managed to stop it by going to the Config Editor and setting:

mail.server.default.autosync_offline_stores to false

Restarting the client then behaved like the old TB 2.x. Keeping all the account sync settings disabled and the Global Search and Indexer unchecked and setting back the mail.server.default.autosync_offline_stores option to true on a subsequent TB session seems to not have caused the problem again and I can use TB 3 as expected.







Reproducible: Always

Steps to Reproduce:
1. Upgrade from Thunderbird 2.x
2. Open client
3.
Actual Results:  
The client started automatically downloading and indexing mails from all accounts.

Expected Results:  
I would have expected that the automatic syncing and indexing features are disabled by default on all accounts, simply because IMAP is supposed to keep mail remotely and users typically may only want to selectively enable this. Also from a user perspective the behavior of TB 2 should have been preserved, especially since it is very intensive for large maildirs.

The IMAP server account I used connects via port 993 and uses SSL/TLS, use secure authentication is disabled.

This only seems to happen right after an upgrade from TB 2.x to 3.0 and it appeared on two different Windows XP systems that accessed that were accessing the same IMAP accounts.
Actually, it appears that setting the mail.server.default.autosync_offline_stores back to true, despite keeping disabled every related option as I described previously, causes message downloading to take place again, so one really has to keep that option to false.
Component: General → Backend
Product: Thunderbird → MailNews Core
QA Contact: general → backend
Version: unspecified → 1.9.1 Branch
(In reply to comment #1)
> Actually, it appears that setting the
> mail.server.default.autosync_offline_stores back to true, despite keeping
> disabled every related option as I described previously, causes message
> downloading to take place again, so one really has to keep that option to
> false.

Tb 3 now respects per folder "offline use" setting upon autosync, and unchecking of "Keep messages for this account on this computer" sets "offline use=off" for all folders of the IMAP account.
So, default.autosync_offline_stores=true of Tb 3 won't force auto-sync of any IMAP folder, if "Keep messages for this account on this computer" is unchecked via UI and Tb 3 is restarted.
"default.autosync_offline_stores=false is needed to disable auto-sync" is Tb 2's behaviour.

> Steps to Reproduce:
> 1. Upgrade from Thunderbird 2.x

How did you upgrade?
Did you do clean install of Tb 3? Or overrided Tb 2's program directory by Tb 3?
(Off-Topic)
(In reply to comment #0)
> Expected Results:  
> I would have expected that the automatic syncing and indexing features are
> disabled by default on all accounts, simply because IMAP is supposed to keep
> mail remotely and users typically may only want to selectively enable this.
> Also from a user perspective the behavior of TB 2 should have been preserved,
> especially since it is very intensive for large maildirs.

This part is dup of bug 505759(WONTFIX'ed).
> Tb 3 now respects per folder "offline use" setting upon autosync, and
> unchecking of "Keep messages for this account on this computer" sets "offline
> use=off" for all folders of the IMAP account.
> So, default.autosync_offline_stores=true of Tb 3 won't force auto-sync of any
> IMAP folder, if "Keep messages for this account on this computer" is unchecked
> via UI and Tb 3 is restarted.

It appears that this is not the behavior I'm experiencing. With  "Keep messages for this account on this computer" on all accounts unchecked, mails are still downloaded if default.autosync_offline_stores=true, the default. Yes, I see that folder synchronization is enabled on my folders, now that you have mentioned it. 

> > Steps to Reproduce:
> > 1. Upgrade from Thunderbird 2.x
> 
> How did you upgrade?
> Did you do clean install of Tb 3? Or overrided Tb 2's program directory by Tb
> 3?

I used the existing profile directory from TB2; I used the update feature of TB2, the installer ran and I followed the migration procedure. I also should mention that while trying to fix the problem I removed several times the offline local mail store as it was filling up my disk. 

I will scrap my profile and start anew with a fresh install.
(In reply to comment #4)
> I will scrap my profile and start anew with a fresh install.

You don't need to scrap your profile(s). (A) If you need to check with profile from scratch, create new profile by "thunderbird.exe -profilemanager" or "thunderbird.exe -P".
Important thing is: (B) Clean install of Tb(install Tb 3 in empty directory).
                    Download ...Setup3.0.exe and execute it.
Do (B) first and check with current profile, and check with (A) if required, please.
I have the same issue.  I have thousands of users and do NOT want them to default to downloading all their email for offline use due to performance and disk quota issues.    I tried to fake thunderbird out by adding

user_pref("mail.server.server2.offline_download", false);

to the users' prefs.js file before starting thunderbird 3 for the first time.

It seems to happily ignore that setting (though if you look in the GUI, the offline synchronization setting IS unchecked) and downloads everything anyway.  If you right click on each folder you will see that they ARE enabled for offline sync on an individual folder basis, which apparently overrides the global setting.

The only way I can turn this off is by resetting every single folder or by checking the global offline setting and then turning it off again.  This will then change the settings for each individual folder to NOT sync.

Where are these individual folder settings stored?   How do I default during an upgrade from thunderbird 2 to 3 to NOT turn on imap offline synchronization?  Why was this made the default with no apparent way to automatically override it?

This is a deal breaker for me and I had to do an emergency downgrade for hundreds of machines (thousands of users) today.  Now I get to clean up 100's of full AFS volumes due to quotas that overfilled...  :(
setting mail.server.default.autosync_offline_stores to false will turn off autosync.
Bryan, ^^ just want to make sure the new migration assistant will have an option to turn off autosync by setting the above pref to false.
I don't believe we are doing that right now.  I'll make a note in bug 545563
 about it.
(In reply to comment #6)
> I tried to fake thunderbird out by adding
> user_pref("mail.server.serverN.offline_download", false);

This option is for default of "offline use=on"(if true) or "offline use=off(if false) when an IMAP folder is created or an IMAP folder is subscribed.
Confusing but current UI does do next:
  Synchronization & Storage
    Message Synchronizing
      [ ? ] Keep messages for this account on this computer
                                                           <Advanced>
  Uncheckd => Checked :
    set mail.server.serverN.offline_download=true   
    set "offline use=on" of all folder of this IMAP account
    enable <Advanced> button to change "offline use" setting of IMAP folders  
  Checkd => Unchecked :
    set mail.server.serverN.offline_download=false   
    set "offline use=off" of all folder of this IMAP account
    disable <Advanced> button to change "offline use" setting of IMAP folders  
So, if user wants to disable auto-sync of all IMAP accounts by changing prefs.js entry only without using UI, user needs to set mail.server.default.autosync_offline_stores=false by himself.
I don't know per account mail.server.serverN.autosync_offline_stores exists or not. But I don't believe such setting exists, because David never reffered to such per account setting.

Warning on use of same profile with Tb 3.0.x and Tb 2.x alternatively:
If you use Tb2 after you set mail.server.default.autosync_offline_stores=false, mail.server.default.autosync_offline_stores is reset because "false" is default of Tb2 and mail.server.default.autosync_offline_stores is removed from prefs.js. So, if you use same prefs.js by Tb3.0.x again, auto-sync is automatically becomes effective again because "true" is default of Tb3.0.x.
To avoid this issue, mail.server.default.autosync_offline_stores=false in user.js is required.
(In reply to comment #10)
 
> Warning on use of same profile with Tb 3.0.x and Tb 2.x alternatively:
> If you use Tb2 after you set mail.server.default.autosync_offline_stores=false,
> mail.server.default.autosync_offline_stores is reset because "false" is default
> of Tb2 and mail.server.default.autosync_offline_stores is removed from
> prefs.js.

That's weird, because there is no default in 2.0 for autosync_offline_stores, since the pref didn't exist.

Yes, autosync_offline_stores is a per-server pref (otherwise, there would be no use/need of having a server.default pref). I usually only tell people to toggle the default pref because they're violently opposed to autosync in general.

Re migration assistant, I don't think it has per-server/account ui, but if it does, it would want to set the per server pref (and maybe the default as well, because the user might add new accounts).
This ALMOST does what I want.  (Thanks so much for the suggestions!)  However, there is still an issue/bug.

If I set 

user_pref("mail.server.default.autosync_offline_stores", false);

in the user's thunderbird 2.x prefs.js BEFORE running thunderbird 3, it does NOT go ahead and download everything for offline use (even though the migration assistant claims that folders are setup for offline synchronization).  YES!!!!   This is exactly what I wanted.  However, it still downloads messages that a *viewed*, but for INBOX only.    That is, I can browse all the other folders and get nothing but a .msf file, but when I browse the INBOX I get both the INBOX.msf file AND an INBOX file containing all the messages I've looked at.   So, it appears to still download the INBOX messages but now it's only the ones I've viewed, which is a huge step forward since some of my users have gigabytes of messages in their INBOX and now it will only download the ones they actually view...still it would be nice if INBOX followed the same rules as the other folders and wouldn't download *any* messages with this preference set.    If I right click on inbox, click Properties, click the Synchronization tab, uncheck the box for "Select this folder for offline use", then it works as expected (the INBOX.msf file is updated but the INBOX file is not).  I still have to *remove* the INBOX file after changing that setting.   Note that the "Select this folder for offline use" box is still checked for EVERY other folder, but the mail.server.default.autosync_offline_stores setting seems to ignore that, except for INBOX.

When you use the UI to turn off offline synchronization, all these check boxes for the individual folders get unchecked as well.  I still don't have an answer on how that works.  Where are the individual settings stored?  I still need to apparently turn off the setting for Inbox if I want *no* messages to be downloaded.

This is all pretty unintuitive that a prefs.js setting that apparently doesn't show up in, nor can be set from the gui(?), actually overrides the individual folder settings that still say the folder is setup for offline mode...when they aren't.  :(

Thanks for all your suggestions.
Jerel, we have several settings that are related but independent. If you have a folder set for offline use, then we store messages you read in the offline store. That per-folder setting is a flag on the folder, and is stored in the .msf file. If you turn on autosync, then we automatically download messages to folders that are configured for offline use. In other words, autosync ignores folders that aren't set for offline use, so you can have a mix of folders setup for offline, and folders that aren't. The per-folder offline flag is more fundamental setting, if that helps.
there is a third setting, per server, which specifies the default for newly discovered imap folders, whether they should be set for offline use or not. Yes, it's confusing to have all these settings, but some of our users want a fairly high level of control.  What's really needed is documentation for all these settings, and how they interact. I'm not sure if a document like this has been started or not...it would either be MDC or SuMomo...
David and everyone else who's helped, thank you again for the information.  I'm half way to getting this the way I want it and it all makes a little more sense.   A fine level of control is great, but administrators also need a way to override settings and it seems as though the migration assistant is being a bit overzealous in changing settings.  :)

Thanks to the "mail.server.default.autosync_offline_stores" prefs.js setting I can put in a thunderbird 2 profile, I can prevent the auto syncing of *all* folders when thunderbird 3 is run for the first time, but it seems as though the migration assistant also turns ON offline use for all folders, even though they were all set to OFF under thunderbird 2, so each email read is still saved for offline use.   The current thunderbird 2 profiles have all folder file caching turned off and would like these settings to remain OFF during the migration.

Note that I also set mail.server.serverN.offline_download=false for the imap server in question but I'm not sure if that one actually does anything.

Is there also a setting to prevent the migration assistant from turning each individual folders' offline mode to on?    

Thanks again.
mail.server.serverN.offline_download is the third pref I referred to earlier. It sets the default for newly discovered/created folders. Setting it to false is what you want.

The migration assistant is being reworked for 3.1 to be less overzealous and more flexible.
(In reply to comment #12)
> I can browse all the other folders and get nothing but a .msf file,
> but when I browse the INBOX I get both the INBOX.msf file AND an INBOX file
> containing all the messages I've looked at.  
> So, it appears to still download the INBOX messages
> but now it's only the ones I've viewed, (snip)

It's another confusing behaviour for users who don't distingush (a) offline-store file("Inbox" in your case) as local mail data repository for "Work Offline" and (b) offline-store file as local mail data repository for auto-sync(from mail accessing view, pre-fetch of mail data) and for further Global Search and Indexer.
Per folder "offline use=on" setting and offline-store file is used for:
(a) Original offline-store use by "Work Offline";
  Automatic download(or ask for download) is executed upon entering "Work
  Offline" mode for folder of "offline use=on". Manual download to
  offline-store file can be executed on both folder of offline-use=on and
  offline-use=off, (Folder Properties, Synchronization, Download Now),
  and data in offline-store is viewed while working in "Work Offline" mode.
(b) New offline-store use by auto-sync mainly for Global Search and Indexer;
  If auto-sync is enabled(autosync_offline_stores, per account or default,
  is set to true), auto-sync downloads mails in offline-store file of folder
  of "offline use=on". 

There are several bevaiours I can't understand why:
 - If auto-sync is enabled and "offline use=on", "Download Now" looks to do
   nothing. It seems "schedule base auto-sync for the folder" ignores
   request of "download NOW" by user.
 - Because offline-store file is originally for "Work Offline",
   AFAIK there is no way to remove offline-store file via UI. 
   Once mails are downloaded into offline-store file, mail data in
   offline-store file(pointed from .msf) is used according to design of
   IMAP mail folder, "Work Offline", auto-sync, Global Search and Indexer, ...
   So, phenomenon of "offline-store file remains forver once used" occurs.

I think "Compact" is better to remove ofline-store file if IMAP folder of "offline use=off" in order not to produce user's confusion like "it appears to still download". ("Clear Now" button at right of "Download Now" button may be a better way for understanding behaviour. I think "Compact Now" at here is also not-so-bad idea for understanding.)
"Loss of data in offline-store file by Compact for user's view" occurs only on data downloaded manually by user's "Download Now" operation into folder of "offline use=no" which is user's choice. I believe it's never problem.
(In reply to comment #15)
> but it seems as though the migration assistant also turns ON offline use for
> all folders, even though they were all set to OFF under thunderbird 2,
> so each email read is still saved for offline use.
> The current thunderbird 2 profiles have all folder file caching turned off
> and would like these settings to remain OFF during the migration.

It's issue reported as Bug 541209.
> Bug 541209 Profile Update should honor existing offline folder settings

Jerel Mackay, please read thru WONTFIX'ed Bug 505759 which I already pointed in comment #3, and please see many bugs pointed in that bug, before adding comments about your new findings around "defaulted auto-sync=enabled for all IMAP folders" and before adding comments of improvement request for issues you experienced.
> Bug 505759 Upgrading Thunderbird 2.0 -> 3.0beta3 marks all folders as
> "Select this folder for offline use" and accounts as "Keep messages for this
> account on this computer"
duping to close bug 569161.
Status: UNCONFIRMED → RESOLVED
Closed: 14 years ago
Resolution: --- → DUPLICATE
You need to log in before you can comment on or make changes to this bug.