881966 - TB should never delete IMAP Offline store without warning / should provide solution to save offline store

Status: NEW

(MailNews Core :: Backend, defect)

Description

[Ulf Zibis]

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0 (Beta/Release)
Build ID: 20130511120803

Steps to reproduce:

follow e.g. one of these scenarios:
- bug 701688
- bug 878764
Maybe there are other scenarios.


Actual results:

IMAP Offline store is *silently* deleted


Expected results:

- user should be warned, that Offline store will be deleted on next TB restart, accompanied by some words for the reason.
- user should be provided a solution to avoid the deletion, e.g.:
-- suggest to make server connection working.
-- suggest replacement of invalid SSL-certificate or import one if missing.
-- offer to import correct SSL-certificate from Windows certificate store.
-- offer to "repair" folder cache i.e. panacea.dat.
-- offer switch to offline mode without deleting offline store

Comment 1

[Ulf Zibis]

Worse:
- user manually switches to offline mode
- user composes some emails and saves them to draft folder
--> after restarting TB, saved drafts are deleted
--> dataloss

Comment 2

[Ulf Zibis]

Another problem:
User doesn't have a backup of TB profile, because it was moved instead copied from original location.
--> message pane folder view/column settings are lost
--> dataloss

Comment 3

[Ulf Zibis]

I think, TB never should *silently* delete offline Mbox and .msf files. Instead it should rename them to *-n* if online sync is not possible for what ever reason (like for other reasons), because:
- Mbox and/or .msf files hold some data, which could not be retrieved from the server.
- Even if bug 883645 would be solved, panacea.dat could become deleted manually as it is suggested in many articles.

Clean-up of old/invalid *-n* files could be established by ThunderPlunger addon.

Comment 4

[renedon]

Other scenario causing local IMAP silently wiped:

- Syncing a Windows Live IMAP account
- TB gives error message "Select Command is not permitted in current state (NotAuthenticatedServerError)"
- Local IMAP storage is wiped!

See: https://bugzilla.mozilla.org/show_bug.cgi?id=1091562

Comment 5

[Ulf Zibis]

(In reply to renedon from comment #4)
Hi renedon,
thanks for adding another scenario.
Please add your vote on this bug :-)

Comment 6

[WADA:World Anti-bad-Duping Agency]

It's not phenomenon of "wipes Offline-Store file" nor "*silently* delete Offline-Store file".
It's merely phenomenon of "start of re-synchronization because server's state is changed to not-accessible state by server himself or by you". :-)
Basic design in imap is "all is held at imap server always" and "access to imap mail data from any where, at any time".
If you want POP3 like local mail data for imap, please hold your imap mail data under "Local Folders" which is designed for "holding mail data locally". "Local Folders" is always available in any your Thunderbird on any your PC.

If construction of msgstore/maildirstore will be finished, "manual change of mail.server.server#.type from imap to pop3" can be used as pretty easy "per folder Work Offline mode in case of loss of server access".
    By maildirstore, we can get rid of "4GB file size limit in BerkleyStore".
    By maildirstore, we can get rid of "From  line problem which was produced by bad design in Unix Mbox format."

Comment 7

[WADA:World Anti-bad-Duping Agency]

Sorry, typo. I wanted to say "per account(server) Work Offline mode".

Comment 8

[renedon]

WADA you still don't get it. We don't care. The end result is still that TB without user interaction wipes the IMAP local storage, which is bad. If there is no good reason for it.

Basic design in TB is the user option "Keep messages for this account on this computer". I don't see how that means "TB go ahead, delete them all whenever you feel like it", but you seem to think so.

You may right in pointing in the direction of the fact that TB has a SYNCING problem with IMAP. A good start for fixing this issue is to implement some debugging/error message informing the user of the "need" to delete the IMAP local storage, and the reason why.

I don't want to use POP3, because I want to keep my local pc completely synced with my mailbox on the server (folders, moved messages, read/unread messages and the whole shebam). Sometimes I access locally, sometimes I access through Outlook webaccess, or through Outlook.com website, depending on the mailbox. POP3 just creates one gigantic mess. That's why I need IMAP for all.

Comment 9

[renedon]

By the way, TB works great that way on IMAP (incl. Exchange access through DavMail). It's just this little annoying phenomenon that leads to an unpleasant surprise sometimes.

Comment 10

[Kent James (:rkent)]

While it would be a worthwhile feature to allow local storage of IMAP emails as a backup of what is on the server, that is not the current design philosophy of Thunderbird IMAP. Offline storage is primarily a performance enhancement. There are a few items of data that are stored in the .msf file and not recoverable from the server (such as the hidden junk score percent, and depending on the IMAP server capability message tags), but those are generally not critical. When it is known that the .msf file is about to be blown away, there is a mechanism to attempt to copy that .msf file to a backup database, and these metadata recovered based on the message ID of the message.

So far though some of the comments on his bug seem to view the various caches that TB provides, including the .msf file, the offline storage mbox files, and panacea.dat, as critical data. Caches are not critical data.

When you reach a situation internally where the program detects that its local representation of the message is seriously out of sync with what is on the server, there is currently no mechanism to reconcile the two, other than to replace the local copies with what is on the server. So it would be a feature request to allow that capability, which is sort of what is implied when you want to keep the local out-of-sync files. Otherwise, you are asking the user to keep local files, which means practically that the local information now has no known correlation with what is on the server. That scenario is generally considered unacceptable, which is why it makes no sense to present the option to the user.

Appropriate bugs would be specific cases, such as some of the items in comment 0, where an out-of-sync condition exists, but could be fixed by some reasonable action by the user. Other appropriate bugs would be cases, if any, where the offline storage is blown away while offline, since you cannot know you are out of sync if you cannot contact the server. But those would be specific requests in separate bugs.

But a general, vague request to give the user a choice in ANY case where an out-of-sync condition exists, and will result in a resync, is not something that is reasonable within the current design philosophy of Thunderbird.

Comment 11

[Wayne Mery (:wsmwk)]

It would be helpful to cite other mail client(s) that function as requested by this bug report, and do it without violating the expected imap client<>server framework mentioned by kent.

Comment 12

[WADA:World Anti-bad-Duping Agency]

(In reply to renedon from comment #8)
> WADA you still don't get it.

You doesn't seem to undestand this bug...
1. This bug is opened with Severity=critical. This means "flaw in Tb's code"(== "Tb's code is different from design") exists.
    However, Tb is currently working as designed, as implemented.
    If it's Severity=Enhancement, .I also think some improvements are needed for Tb users.
2. Even if prompted to user, what can Tb currently do when user replied "Don't clear my Offline-Store file!!!".
    To not touch  Offline-Store file in Tb, only possible way is "go Work Offline mode forever, until the server comes back again".
3. "Change type from imap to pop3 with completed msgstore/mildirstore feature" is a possible/simple method
    to keep Offline-Store file as-is, when user replied  "Don't clear my Offline-Store file!!!".
    This is a friend of "-n Offline-Store" or "Suffixed Offline-Store" in comment #3.
    Without such emnhanceents, improvements in Tb around Offline-Store file is very hard. Rather, nearly impossible.

Needless to say, code for such improvements by you is always appreciated very much, if no comments of merely a complaint.

For your case in Bug 1091562.

Main cause of problem in your case is fault in Windows Live, isn't it?
Why your Windows Live so frequently returns "Select Command is not permitted in current state (NotAuthenticatedServerError)"?
Such response usually means "server is broken now", isn't it?

Anyway, please note that I know well about request of bug 1099275.
   Request for "Warning upon Repair Folder, because re-sync of [Gmail]/All Mail usually takes pretty long".
I believe "what is actually needed" is never "Warning". "What is actulaly needed" is improvements in Tb.

Comment 13

[Ulf Zibis]

I think one of the problems here is the wording.
With "Offline-Store" user is expecting a *persistent store*, but not a volatile cache.
Same with user option "Keep messages for this account on this computer".
If you don't want to change the current behaviour or add a warning message, please change description to something like:
"For this account try to volatily cache messages on this computer to enhance read access performance. In case of sync problems with IMAP server (e.g. server crash, moving to another server, moving the Thunderbird profile to other location, etc.) also special info/settings (message tags, custom tags, junk score, visible columns, ...) will be lost too."
Be aware, that the word "cache" may be difficult/ambiguous to translate to other languages, so additional explanations may be needed.

(In reply to renedon from comment #8)
> A good start for fixing this issue is to implement some debugging/error message informing the user of the "need" to delete the IMAP local storage, and the reason why.
+1

(In reply to WADA from comment #12)
>     However, Tb is currently working as designed, as implemented.
>     If it's Severity=Enhancement, .I also think some improvements are needed for Tb users.
Hm. it is not intuitive and additionally there is no hint in user manual, that lost of special tags/settings or "messages kept on this computer" - just because of a sync problem - is "as designed". So to me it's clearly a bug.

> 2. Even if prompted to user, what can Tb currently do when user replied
> "Don't clear my Offline-Store file!!!".
>     To not touch  Offline-Store file in Tb, only possible way is "go Work
> Offline mode forever, until the server comes back again".
Yes, why not? See additional possibilities from comment 0.
Additionally add: Save/Backup current Offline-Store to "Local Folders/Backup/MyAccountName-1"

> 3. "Change type from imap to pop3 with completed msgstore/mildirstore
> feature" is a possible/simple method
May work for some users which don't have too much subfolders, as with that change, user will loose subfolder and message filtering synchronicity with his server.

> Such response usually means "server is broken now", isn't it?
This is another good reason, that TB should provide an option to save from possible dataloss on the server.
Another reason for such problem could be the change of the server URL or the move to another provider. When the original content stays saved locally, it would be easy to recover the server's content.

> I believe "what is actually needed" is never "Warning".
I disagree, at least temporarily, see above.

> "What is actulaly needed" is improvements in Tb.
In any case too.

Comment 14

[Ulf Zibis]

(In reply to Wayne Mery (:wsmwk) from comment #11)

As I'm using TB exclusively (after move from earlier Netscape) since I use email, I don't know much about other clients, and additionally I suspect if they would name a volatile cache as "Offline-Store".

Comment 15

[WADA:World Anti-bad-Duping Agency]

(In reply to Ulf Zibis from comment #13)
> (In reply to WADA from comment #12)
> (snip)
> So to me it's clearly a bug.

I see. I'll never read this bug. I'll work for bug like bug 1099275 only. Bye.

Comment 16

[renedon]

(In reply to Wayne Mery (:wsmwk) from comment #11)
> It would be helpful to cite other mail client(s) that function as requested
> by this bug report, and do it without violating the expected imap
> client<>server framework mentioned by kent.
Other clients that I have never seen needing to *wipe* local storage i.e. redownload the whole mailbox from the server:
- Microsoft Outlook windows client
- Microsoft Live Mail 2013 windows client
- iPhone/iPad mail client
connected with:
- Windows Live/Hotmail mail addresses
- Microsoft Exchange mail server (company)

Now, whilst what they are doing looks and smells 100% like IMAP, I'm not sure whether in all those cases true IMAP is used, or a variation. The end result remains the same: I have never experienced them needing to redownload mailboxes from the server, or unilaterally deciding to *wipe* the local storage, unlike TB as reported here.

Comment 17

[99zx6r]

wow so this HUGE bug is being ignored/wontfix'd.
I unistalled firefox cause it's not locally secure and ignores the user's selections of settings silently and i guess thunderbird is bad too.

every so often even if you select offline mode it will delete every mail from the profile folder - i have been keeping a daily backup of the profile folder so i can copy-paste when it does this.

no software should just wipe a profile cause it can't find a server especially when the user has selected to keep mails for offline use...

yet again terrible coding guys.

Comment 18

[pbhj]

New devastated user reporting:

1) Email host closed account. 

2) I had TB in offline mode.

3) I removed passwords to prevent access to the server, assuming if it couldn't sync that Thunderbird would maintain the status quo.

4) Went online in TB.

5) All email deleted from accounts where the server details were mangled.

Thunderbird assumes the server (that it can no longer connect to) is right and removes 20k emails.

This paradigm of "whatever the server says is right" is fine, but you need to actually be sure you're connected to the server.

FWIW there is a setting "keep messages for this account on this computer" which rather sounds like it's going to keep the messages on the local computer. But instead it should perhaps be "temporarily cache messages on this computer (warning messages will be deleted if server can't be accessed)".

Comment 19

[Ruben]

So is someone working on this now?

If not, maybe we should all donate a bounty on this on https://www.bountysource.com/issues/1415111 to get this done.

We need a secure way of keeping local backup of IMAP emails.

Comment 20

[Ulf Zibis]

Please also note special requirements for Drafts, Templates and OUTBOX folders, see: bug 878764#c24

Comment 21

[Ulf Zibis]

Maybe this will work better for a valid link: bug 878764 comment 24