Open Bug 517425 Opened 15 years ago Updated 9 months ago

Thunderbird does store local IMAP mail copy in AppData\Roaming not AppData\Local of profile

Categories

(Thunderbird :: OS Integration, defect)

Product:
Thunderbird
Component:
OS Integration
Platform:
x86_64
Windows
Type:
defect

Tracking

(Not tracked)

People

(Reporter: blaubaer, Unassigned, NeedInfo)

References

(Blocks 1 open bug)

Details

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/530.5 (KHTML, like Gecko) Chrome/2.0.172.43 Safari/530.5
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 6.0; de; rv:1.9.1.1) Gecko/20090715 Lightning/1.0pre Thunderbird/3.0b3

We are a company which uses roaming profiles in a windows domain.
In this case all files of the profile, of a user will be tranferd to the file server. Thunderbird stores all its downloaded data of ImapMail in the AppData\Roaming directory and not in the AppData\Local. So a email box with a hughe number of emails will be also transfered to the file server. My profile for example tooks 3GB and this alone in ImapMail but I have no problem, when I log onto another workstation - Thunderbird try to load all the emails on demand from the server.

Reproducible: Always
Component: General → OS Integration
Version: unspecified → 3.0
QA Contact: general → os-integration
Do you realize that if you don't store the emails in the roaming profile, you'll have to download the emails again from the IMAP server if you use a different workstation ? Neither solution is ideal.
Yes but this is no problem. I choose IMAP because I don't depend on the workstation. My emails always on the server stored.
I don't know the average company, but at least the few companies I know about run their IMAP servers from the same server room as their file servers. Some even run the two on the same server, in which case you download gigabytes of data, just to upload it again to the same server. Also the whole point of all of this indexing by Thunderbird locally instead of querying the IMAP server each time, seems to be to speed things up, and constantly transferring gigabytes over the network doesn't seem like it would help.

Also for me as a private person this is annoying. Thunderbird currently takes up 72% of my Roaming folder after just installing Thunderbird 3 recently. That is a lot more data for me to take backup of, which could be avoided if Thunderbird's cache and index files were correctly stored in the Local AppData folder.

Fixing this bug shouldn't be very hard. Just see how Firefox does it from bug 291033:
http://mxr.mozilla.org/mozilla-central/source/netwerk/cache/src/nsCacheService.cpp#430
Then DON'T synchronize your IMAP account. If you do this, a local copy of the maps is stored, so that you can read it while being offline, and you can also search through the content.

If you're using Thunderbird 3, see the "Synchronization & Storage" section in the Account Settings. You can configure it in various ways.
(In reply to comment #4)
> Then DON'T synchronize your IMAP account. If you do this, a local copy of the
> maps is stored, so that you can read it while being offline, and you can also
> search through the content.
> 
> If you're using Thunderbird 3, see the "Synchronization & Storage" section in
> the Account Settings. You can configure it in various ways.

But I do want to be able to read messages offline and to search them. Thunderbird just shouldn't store the cache, offline storage and search indexes in a place intended for synchronization with a file server and for backup. This is the same for both IMAP and newsgroup messages, and I guess also global-messages-db.sqlite or at least most of that file.
I think the offline messages are a part of the "Local" storage and not the "Roaming" storage. It is the same like in the Firefox. 
But also if I deactive the offline storage mode the Thunderbird builds an index in the roaming profile. It is to large.
(In reply to comment #1)
> Do you realize that if you don't store the emails in the roaming profile,
> you'll have to download the emails again from the IMAP server if you use a
> different workstation ? Neither solution is ideal.

We have the same situation here.

Granted, it seems best to do searches on the local imap server. Though, to my understanding (??), the new search feature of TB3 does not work without enabling offline storage.
An idea could be to put the ImapMail folders on a shared network drive that is not backed up (perhaps a cheap san box) - provided the local network is fast.
(Though a bit awkward for it was meant to be an "offline cache".)

Probably (idea2!), this network drive could be made "available offline" by the native windows offline-files-mechanism. That would
1.) make it local again (increse performance) and
2.) enable laptop users to have their emails awailable offline.

Anyone brave enough to test this? ;-)
(Sorry for posting again) Though, beware if a user logs on to two machines and runs TB, at the same time? Can a merge of the imapmail folders damage consistency?
(ps. of course I meant nas, rather than san)
A good and simple solution for all parties could be: Make it possible for the user to select the behavior of storing temporary data with the following options:
a) Roaming profile (default on all non-windows-systems)
b) Local profile (default on all windows-systems because storing of temporary or cache data is default on windows to store in locale profile)
c) Set your own location (..if not present create it or fallback to system default)

And remember: The using of the locale profile is a standard on windows systems for temporary and cache data! And this index data and offline data are only cache data. Not more.
That would be a clean way to do it, but it does not solve the problem completely (as pointed out by Jo in comment 1).

Having a local IMAP server, I would probably switch off local caching at all.

Sadly, Gloda indexes only the offline cache. So without the local cache, I lose gloda.

In my case, for a 15 GB IMAP space, gloda takes 50MB. I can afford to have 50MB in the roaming profile.
So my wish would be that gloda can work without the offline cache. Different approach, but it would solve the problem.
Status: UNCONFIRMED → NEW
Ever confirmed: true
The best answer in a windows corporate environment is to use redirected folders for AppData (and Desktop & My Documents) folders - this avoids the copying back and forth for *all* apps.

One thing about moving the IMAP store to local settings is that all customizations to the message pane view for all folders - columns enabled/disabled/moved - are stored in the .msf files, so if you lose those (move to another PC, etc), you have to recustomize your message pane all over again for *every* folder - this is a major annoyance.
(In reply to comment #11)
> One thing about moving the IMAP store to local settings is that all
> customizations to the message pane view for all folders - columns
> enabled/disabled/moved - are stored in the .msf files, so if you lose those
> (move to another PC, etc), you have to recustomize your message pane all over
> again for *every* folder - this is a major annoyance.

This data should of course be stored in the Roaming location, so if it is in files mixed with Local data, then these files would need to be split up first.
Normally you are right. But redirected folders are just a "workaround" for the problem.

Normally we don not want to synchronized the most of the data to the DomainController because it is just a copy. Please just do not store the content of the messages, headers etc. in the roaming profile!
I agree with Charles that in a "professional" environment, it might be good practice in a windows domain to redirect also the appdata folders. Though (for instance in smaller businesses), administrators might not want to take that effort for some reasons. F.i., they might not want to waste disk (and backup) space for cache data.

I second that its a good idea to separate config data (customizations) from cache-type data - as Jesper suggested in comment 12.
The default location of the cache should be in local settings, which I think would be what everyone expects. (I even saw Outlook 2003 putting the pst file into local settings!) There should be a prefs option, by which each admin is free to decide whether they want to explicitly put the cache into the roaming profile.

My other concern is: If the cache is in local settings, and users log on to various machines alternatingly, probably not waiting for the cache to fully update - TB will alternatingly encounter various incompletely updated states of the cache. Is it rugged enough for that?
(In reply to comment #14)
> I agree with Charles that in a "professional" environment, it might be good
> practice in a windows domain to redirect also the appdata folders. Though (for
> instance in smaller businesses), administrators might not want to take that
> effort for some reasons. F.i., they might not want to waste disk (and backup)
> space for cache data.

Why would anyone redirect cached data? I wouldn't. %AppData%, Desktop and MyDocs are the only folders I redirect.

The point of this bug is IMAP offline storage should be similar to 'cached data' - but, as I pointed out, this isn't entirely true.

Using redirected folders solves this problem nicely, it isn't a 'workaround'.

> I second that its a good idea to separate config data (customizations) from
> cache-type data - as Jesper suggested in comment 12.

Agree... that's why I opened bug 543963, please feel free to vote for it... :)

> The default location of the cache should be in local settings, which I think
> would be what everyone expects.

Are e talking *cache* or IMAP offline storage? These are 2 different things to TB.

> (I even saw Outlook 2003 putting the pst file into local settings!)

Its always done that.

> There should be a prefs option, by which each admin is
> free to decide whether they want to explicitly put the cache into the roaming
> profile.

I do believe that the *cache* is in the Local Settings folder. I think you are talking about the imap offline store, which is simply the location of the mail store.

I agree that this should be configurable... I wonder if that's what:

mail.imap.server_sub_directory pref is for...

I guess I'll find out, since I just opened bug 545243 for this...

> My other concern is: If the cache is in local settings, and users log on to
> various machines alternatingly, probably not waiting for the cache to fully
> update - TB will alternatingly encounter various incompletely updated states > of the cache. Is it rugged enough for that?

Sure - that's a local machine state thing...

But just to be clear - you aren't meaning *cache*, you're meaning imap offline store...
(In reply to comment #15)
> I think you are talking about the imap offline store, which is simply
> the location of the mail store.
> 
> I agree that this should be configurable... I wonder if that's what:
> 
> mail.imap.server_sub_directory pref is for...

Answering myself - this is, of course, configurable, but only manually...

> I guess I'll find out, since I just opened bug 545243 for this...

Bug 545243 is to provide the ability to define a 'imap offline parent directory', so all new imap accounts are stored under there - could be Local Settings, or somewhere else entirely.
Charles, sorry I was using the wrong term. Yes, by "cache" I was actually having in mind the local offline storage of imap acounts - since it seems like a sort of cache to me.
Probably this will straighten out some misunderstanding!

If it were clearly separated from user data, I would probably prefer to not have the imap offline storge in %appdata%, even if it were redirected, because it takes server disk space (and probably backup tape space).

Thanks for mentioning mail.imap.server_sub_directory - seems worth investigating...
(In reply to comment #17)
> Charles, sorry I was using the wrong term. Yes, by "cache" I was actually
> having in mind the local offline storage of imap acounts

No worries... I mostly just wanted to clarify this for others who might be reading.

> If it were clearly separated from user data, I would probably prefer to not
> have the imap offline storge in %appdata%, even if it were redirected, because
> it takes server disk space (and probably backup tape space).

Agreed 100% (as I said above)...

> Thanks for mentioning mail.imap.server_sub_directory - seems worth
> investigating...

Googling didn't reveal its purpose... so maybe someone who knows can comment?
(In reply to comment #18)
> > Thanks for mentioning mail.imap.server_sub_directory - seems worth
> > investigating...
> 
> Googling didn't reveal its purpose... so maybe someone who knows can comment?

I guess you are off track here. It seems it refers to Tools -> Account Settings -> Server Settings -> Advanced -> IMAP server directory
Ahh... ok, thanks for finding that...

Ok, so, if you're so inclined, please feel free to add a vote for Bug 545243...

I just changed  the description to be more precise:

"Need a pref to define ImapMail directory location"
(In reply to comment #20)
> Ok, so, if you're so inclined, please feel free to add a vote for Bug 545243...
> "Need a pref to define ImapMail directory location"

Charles, could you do an uber-brief summary of the implications of bug 545243?
follow up to IRC talk today, partly in the context of profiles on corporate networks (but there's nothing useful to summarize yet), some references ...

the only recent chatter I found is http://groups.google.com/group/mozilla.support.thunderbird/browse_thread/thread/58494a5ee1b690a4/705d2325695548b9?lnk=raot 

nothing found on mozillazine or getsatisfaction (take that with a grain of salt)

in November benb, asuth and others posted in  http://groups.google.com/group/mozilla.dev.apps.thunderbird/browse_thread/thread/715dd86feb73be26 hits some things mentioned today on IRC.  This thread mentions:
 https://bugzilla.mozilla.org/show_bug.cgi?id=345070
   Move cache outside of the regular profile folder
  (which got some support in the thread)
 https://bugzilla.mozilla.org/show_bug.cgi?id=147344
   Breaking up the profile for roaming, sharing and performance
 https://bugzilla.mozilla.org/show_bug.cgi?id=408156
   Ability to split prefs into multiple files 

Jotting of possible issues related to what gets put where:
* disk space
* backup size
* synchronization
* integrity
* performance
* migration
Summary: Thunderbird does store temporary files in AppData\Roaming not AppData\Local → Thunderbird does store temporary files in AppData\Roaming not AppData\Local of profile
I missed http://groups.google.com/group/mozilla.dev.apps.thunderbird/browse_thread/thread/d3d96c5a7c88549e from March "local settings\appdata vs appdata" where this bug is referenced.
And I totally missed the newer (March) part of "TB2 -> TB3 in corporate environment", where it seems an adequate airing might be taking place
http://groups.google.com/group/mozilla.dev.apps.thunderbird/browse_thread/thread/1008d5224aea0e09/30ff0b5ad3437cb3
(In reply to comment #21)
> (In reply to comment #20)
>> Ok, so, if you're so inclined, please feel free to add a vote for Bug 545243...
>> "Need a pref to define ImapMail directory location"

> Charles, could you do an uber-brief summary of the implications of bug 545243?

Sure... its simple really.

Currently TB3 stores ImapMail offline store in %AppData%, which is a part of the roaming profile in a corporate environment.

I believe that this default should be changed to the "Local Settings" folder, so that Imap Offline stores are not a part of the roaming profile.

The main problem with changing this now, manually or otherwise, is, since the message pane column view settings are currently stored in the .msf files, these customizations (which I *always* apply) will no longer roam.

UI customizations should always roam, 'temporary' data - which is what Imap Offline Stores are - should not.

A separate but related issue is the location of Gloda's index file, which - since it can be very large for large mail stores - also needs to be moved to the Local Settings folder.
I completely agree with Charles! :-)
Note that this will not work for everybody - I often (almost daily) have to use different pc's, so I really want to use roaming profiles. It's already bad enough that the urlclassifier3.sqlite file in Firefox has to be downloaded again - I do not want to to this with the Imap mailboxes or global-messages.sqlite.

And yes, this means that I have download several hundreds of megabytes when I switch pc's, but that's done by the operating system. Pulling the data out of the mail server or re-indexing is much slower than out of the fileserver. Bandwidth is virtually unlimited in my workplace, so we don't bother with the network load.
(In reply to comment #27)
> Note that this will not work for everybody - I often (almost daily) have to
> use different pc's, so I really want to use roaming profiles.

I think it would work fine for you - much better in fact than it would if you had to transfer the entire offline mail store every time you logged in/out. Or are you saying that you routinely have GBs of new/deleted/moved email in a day or two's time? More than likely the difference is more measured in the 10's, or at most a few hundred MBs, if you deal with a lot of large attachments (we do)...

> It's already bad enough that the urlclassifier3.sqlite file in Firefox has
> to be downloaded again - I do not want to to this with the Imap mailboxes
> or global-messages.sqlite.

??? But that's the point. Currently, all of these files must be syncd when you log in/out, unless you are using redirected folders and/or storing your profile on a network share, in which case I doubt very seriously you're using TB3 right now, or else your TB would be mostly unusable because of bug 539389.

> And yes, this means that I have download several hundreds of megabytes when I
> switch pc's, but that's done by the operating system. Pulling the data out of
> the mail server or re-indexing is much slower than out of the files server.

It would only be a real problem if you were downloading the entire mail store every time. Seriously, if you are switching PCs multiple times per day, how much mail will be new/deleted/moved between each session? Certainly not enough to worry about as far as re-indexing and/or re-syncing the mail stores goes.

> Bandwidth is virtually unlimited in my workplace, so we don't bother with the
> network load.

To me its not as much about network load as it is usability. I'd much prefer to have a user profile that loads virtually instantaneously (like roaming profiles do when used in combination with redirected folders) - the best of both worlds.

Imo you are solving your problem in the wrong way. The ideal way to accomplish what you want is to store the entire profile on a network share, whether using redirected folders, or whatever. This is how we will do it when our new DC is rolled out - and yes, this means ours will be configured differently from what is being discussed here as a new default.

Regardless, you can always still choose to configure it however you want manually.
My roaming profile (Application Data) is 900MB currently, 600MB of which is used for Thunderbird. My Local Settings (Application Data only) is 1.2 GB, of which 15MB is used by Thunderbird.

It's not a problem downloading the roaming profile when I log in Windows (a few minutes at most), but downloading the IMap data from the mail server (we sync all data) and indexing them takes more than an hour. Downloading the urlclassifier3.sqlite takes DAYS last time I tried (it's throttled by Google). See also bug 550722.

Syncing when you log out takes only a few seconds even if you have changed a few dozen of megabyte (global-messages.sqlite for instance. If the roaming profile is already on your hard disk, logging in takes only a few seconds. The reason our company does it like this is that we often switch between fixed pc's and laptop, that need to carry the entire profile. And we're doing it from different countries (we don't carry laptops on the airplane, thanks to the US Customs & Homeland Security). Bandwidth is not a problem for us (I build routers for a living).

The real problem is ofcourse Windows, and the fact that it insists on having all data local, even if it has to be downloaded from a roaming profile.

What Thunderbird needs is a configurable place for the IMap data - it's up to the user if he wants to store it in the roaming profile, or in Local Settings, on a network share, or somewhere else on the hard disk. Or a USB key or whatever.
See bug 550722 what can happen if you loose the Local Settings - if T-bird will use Local Settings by default, then we have to deal with users that complain that all their data is downloaded again. Not everyone has an unlimited bandwidth (this whole bug is proof of that). I want to argue that moving the default from roaming to local settings is not a solution, you just move the problem to different people. A default location on the harddisk (away from the profile) would be much better for most people. But it needs to be configurable, because not everyone wants that configuration.
I think this bug about making storing all local data local as Microsoft suggest by default. While I do agree we need separate bug for changing this in prefs. Feel free to submit bug about that and mark it depend on this one
there is already a way to define this - see discussion in bug 545243.
(In reply to comment #30)
I just tried what will happen if the ImapMail folder becomes unavailable (like it is when I set it to a local folder, and the user roams to a different machine).
It actually means some kind of data loss:

1.) Virtual folders are lost
2.) Keywords (coloring of messages) are lost
3.) Column settings are lost

(Virtual folders remain lost even when ImapMail becomes available again.)
So, the minor evil seems to keep ImapMail in the roaming profile, and switch off whatever TB-options are needed to keep this folder small...
I'm encountering this issue because I work on 4 different machines, and basically want similar or identical config settings for a number of applications on each of the machines.  I'm often able to accomplish this using Dropbox, either by changing the application to store its config in my dropbox folder, or by creating a softlink from the dropbox folder to the app config folder.  This is difficult if not impossible for Thunderbird because it mixes low bandwidth infrequently changed user preferences with high bandwidth frequently changed files like the local imap caches.  Such caches are definately not what the roaming folder is for.  Witness the Firefox functionality of placing config settings under AppData/Roaming and putting its own cache under AppData/Local.  Since it looks like this is blocked on breaking out config data from the imap caches, is there a seperate bug for tracking that issue?
Depends on: 543963
(Report from 609876):

Windows separates user profiles into two parts - "Roaming" (%APPDATA% env var)
and "Local" (%LOCALAPPDATA% env var). This separation exists to permit apps
that create large caches, indexes, etc to store them in a part of the user's
profile that isn't synchronized to the login server - and often isn't backed
up.

Thunderbird stores easily-regenerated data, namely the IMAP cache and the
global index files, in the roaming profile. It should store these in the local
profile, as it regenerates them if they are missing, they can be removed
without data loss, and they're huge.

Thunderbird should notice and respect the %LOCALAPPDATA% environment variable,
or use SHGetKnownFolderPath( FOLDERID_LocalAppData ) to find where to store its
caches and indexes.
BTW, Group Policy's:

User Configuraton\Administrative Templates\System\User Profiles\Exclude Directories in Roaming Profile

... option could be used to exclude the Thunderbird ImapMail sub directory from the roaming profile, except for the fact that Thunderbird's profile directory randomization means there's no stable path relative to the user's homedir to specify in Group Policy.

An installer option that made Thunderbird create exactly one profile per user account, and do so in a stable location, would make this issue less painful, but would probably be harder than just moving the ImapMail cache to %LOCALAPPDATA%.
I second Craig.
It more logical, and more elegant have the cache the proper location.

Admins are busy people, and probably dont like to make "Extra-Würschtel" for each installed app.

Besides, part-time admins that are familiar with group policies probably are not familiar with Thunderbird, and vice versa (depending on whether they come from the "open source world" or from the "MS world").
(In reply to comment #37)
> Thunderbird stores easily-regenerated data, namely the IMAP cache and the
> global index files, in the roaming profile. It should store these in the local
> profile, as it regenerates them if they are missing, they can be removed
> without data loss, and they're huge.

Not entirely correct, but this reminded me to add bug 543963 as a blocker to this one, because as it stands now, the fact that UI the message pane folder/column view settings (modifications made to the UI) are stored directly in the .msf files... bug 543963 is about separating these modifications out from the locally cached stuff (ie the .msf files)...

> Thunderbird should notice and respect the %LOCALAPPDATA% environment
> variable, or use SHGetKnownFolderPath( FOLDERID_LocalAppData ) to find
> where to store its caches and indexes.

Agree, but - this can be a real load problem on the IMAP server(s) in environments that have large IMAP stores and actual roaming users who log onto different machines in the office.

As for the huge indexes created/needed by GLODA - what I'd really like to see is some formal IMAP INDEX enhancement to the IMAP RFCs for storing mail indexes directly on the imap server and letting the server take care of it. Dovecot is working on this, and its indexes dramatically speed up almost everything but body searches - it currently has two options for this, Squat and Solr, but both are still a work in progress...
Can someone who has access please add bug 543963 as a blocker to this one?
bug 543963 it's already there
Cyrus IMAPd does server side indexing and search of headers and body text using squat, and has for at least six years (when I started using it). It's part of why the global search & indexer is so frustrating in my environment - it's on by default, creates monster files and lots of IMAP server load, and is largely unnecessary.

As for server load - header downloads aren't a big deal unless your server is pretty **** (ie: mbox/maildir based with no header cache/index) and there should be no need to download entire bodies where server-side indexing is in place. The IMAP cache should be small and easily regenerated. Anyway, if you don't have IMAP server load but still need local caches you have roaming homedir server load instead, which I don't see as a win.

I'm periodically temped to grab the Thunderbird sources and see if I can hack in support for Group Policy extensions to set user.js values (so I can turn indexer off, etc) via Group Policy. Then I realize how horrific the win32 Group Policy stuff is on the programming side and run screaming back to my scary login script hacks.

In truth, a lot of this would be solved by providing a profile-less install option that got rid of profiles.ini and the whole randomized subdir thing, making Thunderbird (and Firefox) user config paths predictable and managable. We could even (finally) preconfigure the user's IMAP account on login! But I guess that's a bit off topic for this bug.
(In reply to comment #43)
> Cyrus IMAPd does server side indexing and search of headers and body
> text using squat, and has for at least six years (when I started using
> it).

Yeah, its a lot more mature - but dovecot is much easier to install/config.

I don't have much use/need for searching email bodies, so its immaturity in this regard doesn't bother me...

> It's part of why the global search & indexer is so frustrating in my
> environment - it's on by default, creates monster files and lots of
> IMAP server load, and is largely unnecessary.

So script the install

> As for server load - header downloads aren't a big deal unless your server is
> pretty **** (ie: mbox/maildir based with no header cache/index) and there
> should be no need to download entire bodies where server-side indexing is in
> place.

Agreed, although I do like the fact that TB3 can now be configured to download the bodies *once* on demand and stores the full message in the local IMAP store (not the cache).

But like you I cannot wait for full support for GPOs.

> The IMAP cache should be small and easily regenerated.

And shouldn't lose anything, which it does currently, which sux big time.

> Anyway, if you don't have IMAP server load but still need local caches
> you have roaming homedir server load instead, which I don't see as a win.

Which is why I use Redirected Folders for %AppData% (and My Docs and Desktop).

> In truth, a lot of this would be solved by providing a profile-less install
> option that got rid of profiles.ini and the whole randomized subdir thing,
> making Thunderbird (and Firefox) user config paths predictable and managable.
> We could even (finally) preconfigure the user's IMAP account on login! But I
> guess that's a bit off topic for this bug.

And besides the devs don't have any interest in doing this (losing the 'salted' profile paths) either (the stated reason is security, but I disagree vehemently with 'security through obscurity' arguments - unless things have changed since the last discussion I recall reading about this which was admittedly a while ago.
Has any progress been made on addressing this bug?
Summary: Thunderbird does store temporary files in AppData\Roaming not AppData\Local of profile → Thunderbird does store local IMAP mail copy in AppData\Roaming not AppData\Local of profile
Blocks: tb-enterprise
I would like to know too. We are using TB in our company for some time and to be honest I never look at the size of roaming data which TB use, but this starts to be problematic somehow as some users report the login process takes ~15 minutes and this is definitely not good. The TB uses ~80% of the whole profile size for some of them in our AD server and to be honest I would like to avoid manually change each account to store IMAP cache locally :(
Maybe a "in-the-middle-cache" would help: The roaming profile should only save the mail headers, so when changing the workstation, the headers are "instantly" updated, and message bodies could be added selectively to offline store upon opening messages.
See Also: → 535137
My solution (netlogon.bat):

@echo off
setlocal EnableDelayedExpansion

C:
if exist C:\Users\%USERNAME%\AppData\Roaming\Thunderbird\NUL (
  cd "C:\Users\%USERNAME%\AppData\Roaming\Thunderbird\Profiles"
  for /D %%i in ("*") do (
    set "profile=%%~nxi"
    mkdir "C:\Users\%USERNAME%\AppData\Local\Thunderbird\Profiles\!profile!\ImapMail"
    for %%j in ("!profile!\ImapMail") do (
      set "attribute=%%~aj"
      if "!attribute:l=!" == "!attribute!" (
        robocopy /E /MOVE "C:\Users\%USERNAME%\AppData\Roaming\Thunderbird\Profiles\!profile!\ImapMail" "C:\Users\%USERNAME%\AppData\Local\Thunderbird\Profiles\!profile!\ImapMail"
      )
    )
    rmdir /Q /S "C:\Users\%USERNAME%\AppData\Roaming\Thunderbird\Profiles\!profile!\ImapMail"
    mklink /D "C:\Users\%USERNAME%\AppData\Roaming\Thunderbird\Profiles\!profile!\ImapMail" "C:\Users\%USERNAME%\AppData\Local\Thunderbird\Profiles\!profile!\ImapMail"
  )
)
See Also: → 1399793
Severity: major → normal
OS: Windows Vista → Windows

Again if someone has 10GB+ of email, those would be moved back and force between the client computer and server at each login and logout of end-user set with roaming profile on a Windows domain that is a real issue and is just not possible, it does not work well in practice...

This is really a "bug" in TB that shall be addressed... or should I better say a by default misconfiguration on Windows platform? Which may be more accurate!

@Thomas,

If Thunderbird want to make headway in organisations such as business, gov, etc... this bug really need to be addressed. I don't know if you have way to raise awareness about the issue within the team and put it on the roadmap somehow... I am happy to discuss the matter if need be... There is already a functional usable solution which is just to change manually* setting in TB (but cumbersome!) that would not take much time or resources to TB team to implement by default, while making a huge difference to end-users... More detailed info in bug 1399793 (now marked as duplicate of this bug).

All it would require to fix it seems, is a change in one default setting within TB (a path) upon setup of IMAP account(s)!

This change shall make no difference for average users, but would drastically improve situation by default for roaming profile users on Windows!
I am not trying to promote Windows or business/organisation setup but to prevent TB getting causing issues and damaging its own reputation for bad side effects it may introduce for those users. If that make sense!

*cumbersome as it requires manual setup each a user login to new machine as well as many steps to be done in a careful way per user, per machine!

@Wayne,

It is really not clear nor understandable what are the technical issues that prevent TB team to change a default setting in TB (already available in IMAP account preferences) for IMAP cache to be relocated on the client PC outside the windows user roaming profile into the windows user local profile. I don't see how that can break anything for the application itself, search being done locally (on client computer) or server side (on IMAP server) anyway... from my testing I haven't found any issues...

If there is an open discussion organised about this matter, I would be glad to participate to it and explain the problematic, because I don't think the problematic (and simple solution) is clearly understood by the dev team as it may require knowledge and experience with Windows domains and roaming profiles.

As a explained at multiple occasion, local cached data shall reside in the Window user local profile not in the roaming one (which shall solely be used for configuration/preferences) as IMAP cached data is not meant to be shared on multiple computers, because TB would in any case reload/update everything upon use from the IMAP server anyway...

Hope that make sense.

Regards,
Richard

Flags: needinfo?(bugzilla2007)

Even if Windows domain is not of matter, moving the locally saved messages to AppData/Local on Windows or ~/.cache on Linux would have the advantage for daily local update software, as this would severely reduce the footprint of the daily backup chunks of personal local data.

But only the mbox files should be moved to AppData/Local on Windows or ~/.cache on Linux, but not the .msf files, as they contain local user data such as local keywords or thread pane view/column settings which are not part of the data on the foreign IMAP server. So this can not be achieved by just manually changing the current IMAP account settings.

Ok, now I understand a bit better the problematic with this issue which due to user TB preferences (pane view/column settings) being tangled with IMAP cached data (within .msf files)! Correct me if I am wrong here!

I still think:

  • short term the default IMAP server path (IMAP cache) shall be changed to local area, because it is more problematic for people to have to wait 10mn+ to login/logout to/from a machine than having to reset pane view/column settings first time when moving to another computer...
    and
  • then long term having user preference untangled from IMAP cache (unless handled by IMAP server?) and properly set separately on their own! Is msf file really the proper way to store user preferences in the first place, I wonder?!

@Alessandro,

I put you in copy of this comment/bug because separating user preference from local cache IMAP data may be something to consider improving while revamping the message list UI which is on your blueprint/timetable... for sometime in the near future...

@all,

Windows has two locations (three in practice but two mostly used by third party apps) in the User profile:

  • AppData/Local (~/.cache on Linux it seems, as per previous comment)
    -- used for cached data (for quick access/offline use) that can be re-generated/synced

  • AppData/Roaming (supposedly ~/ on Linux except ~/.cache)
    -- one for preference data (for roaming use - set once by end-user used everywhere)

Both contains local user data but not for the same purpose, and it seems Linux as well...

Thunderbird already setup a user profile in both location... and use both location... but not necessarily adequately for all data it handles...

Local shall contained any local data mostly locally cached data on the client computer... that can be rebuilt/reconstructed from server info... so both mbox files (containing local cached message content) and msf files (containing index and msg header info) shall be placed in this location... basically anything that can be rebuild from IMAP server upon first startup...
--> Not meant to be file system copied back and forth between Windows client computer and server upon sign in/out on the computer... as other sync mechanism (IMAP) are in place to keep the data up-to-date and backed up/synced between client<->server

Roaming shall contained any roaming data mostly user preferences... that are set by end-user as fit best and shall remain the same on all computer when roaming... thread pane view/column settings fall into that category but I am not sure to understand why such settings were placed in msf file instead of sqlite pref file, probably by convenience originally as they may be set on a per folder settings (unless those are handled via the IMAP server somehow but I am not aware :-)
--> Meant to be copied back and forth between Windows client computer and server upon sign in/out on computer...

All those are user local data that is why they are in the user profile, but their purpose differ greatly that is why there are stored in different location on Windows... and on Linux as well...

Not sure what is meant by local keywords in the previous comment, may need clarification... but a saved search (the settings only) would fall in user preferences (aka AppData\Roaming location), but an saved search or auto-completion search cache would fall as local data (aka AppData\Local location)... if that make sense...

It shall be the same for any pref/cache data over all in TB... for example I would expect an offline cached network calendar stored in the appData\Local on a Windows computer... and so on... but that may be pushing it I guess :-)

Regards,
Richard

Flags: needinfo?(alessandro)

I put you in copy of this comment/bug because separating user preference from local cache IMAP data may be something to consider improving while revamping the message list UI which is on your blueprint/timetable... for sometime in the near future...

Thanks for the ping.

Flags: needinfo?(alessandro)

(In reply to Richard Leger from comment #53)

Ok, now I understand a bit better the problematic with this issue which due to user TB preferences (pane view/column settings) being tangled with IMAP cached data (within .msf files)! Correct me if I am wrong here!

I still think:

  • short term the default IMAP server path (IMAP cache) shall be changed to local area, because it is more problematic for people to have to wait 10mn+ to login/logout to/from a machine than having to reset pane view/column settings first time when moving to another computer...

The lost of pane view/column settings may be bearable, but not the lost of message linked additional properties which can not be saved on the server such as "local keywords".

  • then long term having user preference untangled from IMAP cache (unless handled by IMAP server?) and properly set separately on their own! Is msf file really the proper way to store user preferences in the first place, I wonder?!

Good point!

Not sure what is meant by local keywords in the previous comment, may need clarification...

In messages list you can right-click on a message and choose that option to colourize items to your own taste. This data is not backed by the IMAP server, so should be saved in the Roaming space. I guess the same applies for the junk status flag and maybe some more TB specific properties, which currently are all stored in the .msf files.

but a saved search (the settings only) would fall in user preferences (aka AppData\Roaming location), but an saved search or auto-completion search cache would fall as local data (aka AppData\Local location)... if that make sense...

It shall be the same for any pref/cache data over all in TB... for example I would expect an offline cached network calendar stored in the appData\Local on a Windows computer... and so on... but that may be pushing it I guess :-)

Correct!

Engineering-wise, we now we have the infrastructure already in place in Gecko to place profile files in local. We just need to use it. We didn't have this 13 years ago when this bug was filed. This is now long-handing fruit.

The ImapMail/ directory in TB profile should not contain the filters file (I've accidentally deleted all my filters recently, for this very reason).

The same problem exists under Linux. Having ImapMail/ in the home dir requires me to add special exceptions in the backup strategy, just for Thunderbird. Whereas there is a specific folder for this kind of data already: ~/.cache/thunderbird/<profile>/ , which can be exempt from roaming, across all applications. This matches the "local" profile on Windows. So, this change wouldn't be Windows-specific, but makes sense cross-platform.

See Also: → 1780738
Severity: normal → S3
Flags: needinfo?(bugzilla2007) → needinfo?(vseerror)
You need to log in before you can comment on or make changes to this bug.