1422568 - Thunderbird deletes folder on IMAP accounts where the server in non contactable.

Status: RESOLVED DUPLICATE of bug 689067

(Thunderbird :: Folder and Message Lists, enhancement)

Description

[Matt]

Anecdotal evidence in Support forums indicates that Thunderbird is removing folders from IMAP accounts when the mail server is non contactable.  Usually de to firewall or anti virus issues post update.  This is not an optimal situation for those that are paying for data allowances and having whole accounts re-downloaded.

Is there something we can do to back out of the sync process where the server or account is unavailable that does not appear so destructive to data from the user perspective?

Comment 1

[gene smith]

At one point, while working on another bug several months ago, I thought I noticed that when you intentionally unsubscribe a folder that tb removes the locally stored/synchronized email content for that folder, leaving just the *.msf header file. However, I tried that the other day (while working on another bug) and didn't see any file deletion on unsubscribe.

Re bug 1423588: If unsubscribe actually or possibly does remove stored email, then I would hopefully think that tb would only assume a folder is unsubscribed when the imap lsub does not return the previously subscribed folder in the list. Tb should not just assume all folder have unsubscribed because of a simple network error or even when there is an error response to lsub.

I saw something else, again, while working on an unrelated bug. When I tried to drag a gmail folder to another level in the gmail account, it caused *all* my gmail to be re-downloaded.

I will try to determine what is really going on with this.

Comment 2

[Wayne Mery (:wsmwk)]

> Tb should not just assume all folder have unsubscribed because of a simple network error or even when there is an error response to lsub.

exactly. And we get support requests about this as well.

I happened to have a few bugs open and I'll add a few more to make a rat pack https://mzl.la/2BOy3Dr
(I don't claim any or all have a direct connection)

Comment 3

[gene smith]

Ok, I checked again and indeed unsubscribing folder X deletes the corresponding file X containing the local/sync'd emails leaving just headers, X.msf. However, at least for gmail, the unsubscribe and the file removal only takes effect after a tb restart. A restart is also often required when you subscribe a folder for it to be seen.

I forced a network error for gmail by connecting to the wrong port and it did not cause problems at all (other than pop-up message saying could not connect to gmail server). I really don't think it is a network glitch. Instead it is probably the imap server falsely reporting that a folder is unsubscribed, which causes tb to remove the file and then re-download again when server reports that it is subscribed. On the ubuntu forum, the user reporting this found a work-around of un-checking "Show only subscribed folders". This causes tb to not delete files for unsubscribed folders so all folders the server has are shown (that seems to defeat the purpose of un-subscribe). 

Maybe a new selection "Don't remove downloaded emails when unsubscribing" would be another solution? I don't see anything like this in the config editor.

Comment 4

[gene smith]

I checked again and unsubscribe actually removes both files for folder X: X.msf and X.

(Follow-up to gene smith from comment #1)
> 
> I saw something else, again, while working on an unrelated bug. When I tried
> to drag a gmail folder to another level in the gmail account, it caused
> *all* my gmail to be re-downloaded.

I tried this again and see no problem with dragging and dropping gmail folders. Does not cause any folder to re-sync/download.

Also, no problem after starting tb with the network bouncing up/down.

Comment 5

[Wayne Mery (:wsmwk)]

We had a gmail-specific example of this (since fixed) in bug 1374244.

Comment 6

[gene smith]

I did notice something on three non-gmail accounts that I also have running in tb.

On yahoo, (free) outlook and trial office365 outlook, for each I see 11 INBOX.msf files (INBOX-1.msf to INBOX-11.msf) and 1 INBOX file. I think these are being created each time tb starts up since, based on the timestamps, I have restarted tb that many times while doing various tests. 

Only when INBOX for these accounts is clicked on and selected do the emails download and create the INBOX file, it seems.

However, I may have done something "illegal" that causes this. I deleted all the files in the profile under ImapMail/ and the panacea.dat file and restarted tb earlier today. I did this becauses I was seeing other files and directories under ImapMail/ with -1 suffix and I wanted to start "clean" even though they caused no apparent harm. 

I have several other non-gmail accounts running that don't have the INBOX-1..11.msf files, just INBOX.msf.

While doing other experiment with gmail (put a trash folder under INBOX). Restarted tb and now I saw INBOX and All Mail download, each twice. Maybe my profile is totally corrupted now?

  (In reply to Wayne Mery (:wsmwk) from comment #5)
> We had a gmail-specific example of this (since fixed) in bug 1374244.

Restarted tb several times on another machine with uncorrupted (or less corrupted; don't think I changed it manually) profile and see no problem with gmail. So maybe people seeing these problems with gmail re-downloading or other weird folder error (extra folders with -xx suffix) have gone in and tried to "correct" something in their profile and just made it worse. Not pointing fingers but I will try some more things to verify a problem.

Comment 9

[Daniel Vassy]

Re Matt's analysis thay the impact is have to download again. Incorrect! The impact is much more severe that that. If the reason why you cannot contact your emap server is that it has crashed, the uncivilized deletion by Thunderbird of your local email folders means that you can lose ALL YOUR EMAIL HISTORY irretrievable. Lost on the server because it crashed, lost on your computer because Thunderbird zapped the local copy and you cannot restore from it

Comment 10

[Matt]

(In reply to Daniel Vassy from comment #9)
> Re Matt's analysis thay the impact is have to download again. Incorrect! The
> impact is much more severe that that. If the reason why you cannot contact
> your emap server is that it has crashed, the uncivilized deletion by
> Thunderbird of your local email folders means that you can lose ALL YOUR
> EMAIL HISTORY irretrievable. Lost on the server because it crashed, lost on
> your computer because Thunderbird zapped the local copy and you cannot
> restore from it

Not exactly true.  But if you want to have it that way you may.  The support forum is littered with folk that for reasons I fail to understand think a synchronised folder of mail is an archive.  We have been helping them recover their mail for years. It is possible and while it takes a little time is not all that painful.  I suggest you post in the support forum if you want help recovering your mail.  The lesson to take away is that only mail in your "local Folders" is truly local and yours,  but that to is subject to sensible backup regime.

Recovery involves locating the actual files used to store your mail in the profile folder and moving them out of the profile before a compact of folders can occur.  Recovery is not possible after a compact.

Comment 11

[pby_fr]

(In reply to Matt from comment #10)
> Not exactly true.  But if you want to have it that way you may.  The support
> forum is littered with folk that for reasons I fail to understand think a
> synchronised folder of mail is an archive.  We have been helping them
> recover their mail for years. It is possible and while it takes a little
> time is not all that painful.  I suggest you post in the support forum if
> you want help recovering your mail.  The lesson to take away is that only
> mail in your "local Folders" is truly local and yours,  but that to is
> subject to sensible backup regime.
> 
> Recovery involves locating the actual files used to store your mail in the
> profile folder and moving them out of the profile before a compact of
> folders can occur.  Recovery is not possible after a compact.

I'm not sure that the bug I created (bug 1450461) is really a duplicate of this one, as the folder inside ImapMail of the profile suddenly pass from 4.9GB to 1.2MB. No recovery possible (other than replacing it with a backup of this folder, when TB is closed).

You are right to say that no loss of email occurs, as everything remains on the server, but it is very troublesome to have to re download every messages, and that TB cannot be used offline during the time the server don't answer correctly. No really user friendly.

But I still have a loss of data: Thunderbird Tags are also deleted when the problem occurs, and when it is on a server like Yahoo which don't support tags, after re-download, all the previous tags are lost.
If at least TB would store these tags in another file, which wouldn't be deleted on such problem, but as it is, loss of data really occurs.

Comment 12

[Wayne Mery (:wsmwk)]

bug 1377106 may be another example

Comment 13

[Wayne Mery (:wsmwk)]

(In reply to gene smith from comment #6)
> ...> 
>   (In reply to Wayne Mery (:wsmwk) from comment #5)
> > We had a gmail-specific example of this (since fixed) in bug 1374244.
> 
> Restarted tb several times on another machine with uncorrupted (or less
> corrupted; don't think I changed it manually) profile and see no problem
> with gmail. So maybe people seeing these problems with gmail re-downloading
> or other weird folder error (extra folders with -xx suffix) have gone in and
> tried to "correct" something in their profile and just made it worse. Not
> pointing fingers but I will try some more things to verify a problem.

-xx suffix and other issues.

Bug 520437 - Garbage file/directory of {foldername}-{suffix}.msf and {foldername}-{suffix}.sbd (and {foldername}-{suffix} also, if offline-use=ON) is created upon each unsubscribe / subscribe

Bug 698321 - Sudden appearance of mirrored IMAP account Inbox and folders (Inbox-10, Inbox-9, ...)

Bug 1376508 - Unified view no longer supports GMail accounts

Bug 1428102 - First synchronization always fails with error "The current operation on xxxx did not succeed. The mail server .... responded: [NONEXISTENT] Unknown mailbox Inbox^yyy^xxxx (Failure)."

Bug 1287223 - duplicated mbox files Sent and Sent

Comment 15

[Jorg K (CEST = GMT+2)]

Matt, regarding this bug and bug 1423588. Gene has fixed an issues in bug 1453643 and bug 1408610 where authentication hiccups led to re-download of e-mail. Can you please check whether the issue still persists after TB 60 beta 3.

We're not sure how firewall or AV could interfere and cause this problem. Gene is on Linux without AV, but I'm sure he could simulate a firewall blocking the connection.