Thunderbird wiping local copies of IMAP folder because mail server rolls UID validity and other server related errors

VERIFIED DUPLICATE of bug 157429

Status

VERIFIED DUPLICATE of bug 157429
4 years ago
4 years ago

People

(Reporter: renedon, Unassigned)

Tracking

31 Branch
x86_64
Windows 8.1

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [technical discussion: comment 3])

Attachments

(2 attachments)

(Reporter)

Description

4 years ago
User Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:32.0) Gecko/20100101 Firefox/32.0
Build ID: 20140923175406

Steps to reproduce:

Open Thunderbird (Automatically starts syncing IMAP accounts)


Actual results:

Inbox of one IMAP account starts downloading from scratch (1600+ headers, 1600+ messages).

All subfolders are affected as well.


Expected results:

Nothing (just check if there are new messages and download new messages if needed)

Account settings are such that Messages are kept on this computer, All messages are sychronised locally regardless of age and Messages are not deleted.

That is why this should never be allowed to happen. Yet the bug happens regularly (once every two-three weeks), causing some download cap to be lost due to the redownload of the IMAP account, plus the time lost to resync everything.
(Reporter)

Comment 1

4 years ago
This happened again today. I captured an error message while synchronising:

"The current operation on 'Inbox' did not succeed. The mail server for account *******@live.nl responded: Select Command is not permitted in current state (NotAuthenticatedServerError)."

Thunderbird then wipes all local copies of the e-mail account! INBOX file, *.MSF files everything.
(Reporter)

Updated

4 years ago
Summary: Thunderbird losing local copies of IMAP → Thunderbird wiping local copies of IMAP

Comment 2

4 years ago
For me TB shows two consecutive error

Namespace Command is not permitted in current state (NotAuthenticatedServerError)
Select Command is not permitted in current state (NotAuthenticatedServerError)

and deletes everything from the account folder. I have this issue with Microsoft's email as well (@hotmail.com).

I'm trying to reproduce this issue with enabled IMAP log, but still no luck.
> Bug summary : Thunderbird wiping local copies of IMAP

"Data saved by auto-sync in Offline-Store file of IMAP account in Tb" !== Local copy of mail data.
In IMAP, all data is held at server. "Data saved by auto-sync in Offline-Store file of IMAP account in Tb" is a kind of Cache for performance gain.
If you need "Local Copy of IMAP mail data", you need to copy IMAP mails to "Local Folders" which is for holding mail data locally.

If re-organazation of Mbox occurs at server, UIDVALIDITY is usually lost, so Tb fetches all data from scratch again upon next server access.
If server connection is impossible, it's same as "IMAP account doesn't exist any more", "IMAP folder doesn't exist any more", in some situations. So Tb fetches all data from scratch upon next successful server access.
I guess that repeared error of "Select Command is not permitted in current state (NotAuthenticatedServerError)" from your server generates such condition. "SELECT failure" is usually same as "Mbox doesn't exist" for IMAP clients.

You can see "deletion of all folder file for imap" and "fetch all data from scratch" by next test.
  1. Delete panacea.dat(Folder Cache) => Because Folder Cache is deleted, "sync'ed with server" status is lost.
  2. Repeat "Start Tb with Work Offline mode, and termiinate Tb",
      or Pull-Off LAN cable and repeat "restart Tb then terminate Tb".
      => Because sync'ed status is lost due to (1), Tb tries to communicate with server,
            but by "Work Offline mode" or "Pull-Off LAN cable", server access is impossible.
            This is similar to "server is not available any more" if sync'ed status is already lost.
  3. Restart Tb. When imap server access is successful, all data is fetched from scratch.

FYI.
Another cause of "fetch all data from scratch" : "Delete .msf file"
   Even if you didn't delete .msf file, other software such as auto-backup, virus scanner,  can interfere Tb's .msf file open.
   Because "open .msf file" failed, it's same as "data in .msf file=null" in some special situations.
   This kind of issue may produce phenomenon of "fetch all data from scratch".

> Namespace Command is not permitted in current state (NotAuthenticatedServerError)
> Select Command is not permitted in current state (NotAuthenticatedServerError)

This kind of error is usually indicator of "imap server is not well configured".
Why such error can so frequently occur in your server?
Is your "imap account definition in Tb" correct/proper/valid?
(ISP requests SSL, but server software himself supports StartTLS too.  So, connection/imap command via. StartTLS is usually not prohibited, but it's rejected when special case.)
Do you use proxy server(including local proxy of anti-virus software) for your imap connection?

Comment 4

4 years ago
>This kind of error is usually indicator of "imap server is not well configured".
>Why such error can so frequently occur in your server?

It's not my server. I have (had?) this issue with a hotmail address. There was long outage with their system some time ago and TB deleted/redownloaded all emails quite often. As i can see, the reporter have this issue with Microsoft account too (@live.nl).
No proxy server or any fancy AV software.

>"SELECT failure" is usually same as "Mbox doesn't exist" for IMAP clients.
Select failure in normal state does NOT produce errors. Error on SELECT should be a sign for TB DO NOT delete any folder.
(Reporter)

Comment 5

4 years ago
(In reply to WADA from comment #3)
> > Bug summary : Thunderbird wiping local copies of IMAP
> 
> "Data saved by auto-sync in Offline-Store file of IMAP account in Tb" !==
> Local copy of mail data.
> In IMAP, all data is held at server. "Data saved by auto-sync in
> Offline-Store file of IMAP account in Tb" is a kind of Cache for performance
> gain.
> If you need "Local Copy of IMAP mail data", you need to copy IMAP mails to
> "Local Folders" which is for holding mail data locally.
That point is kind of moot, because Microsoft Outlook does not have this problem. Thunderbird is the one that wipes local copies of data, with the user finding out after the fact! I have ticked the tickbox "Keep messages for this account on this computer", so that Thunderbird keeps messages for this account on this computer. Yet Thunderbird wipes those messages without asking for my permission.

> 
> If re-organazation of Mbox occurs at server, UIDVALIDITY is usually lost, so
> Tb fetches all data from scratch again upon next server access.
> If server connection is impossible, it's same as "IMAP account doesn't exist
> any more", "IMAP folder doesn't exist any more", in some situations. So Tb
> fetches all data from scratch upon next successful server access.
> I guess that repeared error of "Select Command is not permitted in current
> state (NotAuthenticatedServerError)" from your server generates such
> condition. "SELECT failure" is usually same as "Mbox doesn't exist" for IMAP
> clients
Are you saying that it is Microsoft that changes the outlook.com/live.nl server somehow at irregular intervals, causing Thunderbird to wipe the local copy of the IMAP mailbox? Again Microsoft Outlook seems to handle this very well.

.
> 
> You can see "deletion of all folder file for imap" and "fetch all data from
> scratch" by next test.
>   1. Delete panacea.dat(Folder Cache) => Because Folder Cache is deleted,
> "sync'ed with server" status is lost.
>   2. Repeat "Start Tb with Work Offline mode, and termiinate Tb",
>       or Pull-Off LAN cable and repeat "restart Tb then terminate Tb".
>       => Because sync'ed status is lost due to (1), Tb tries to communicate
> with server,
>             but by "Work Offline mode" or "Pull-Off LAN cable", server
> access is impossible.
>             This is similar to "server is not available any more" if sync'ed
> status is already lost.
>   3. Restart Tb. When imap server access is successful, all data is fetched
> from scratch.
I don't see how this would be necessary.

> 
> FYI.
> Another cause of "fetch all data from scratch" : "Delete .msf file"
>    Even if you didn't delete .msf file, other software such as auto-backup,
> virus scanner,  can interfere Tb's .msf file open.
>    Because "open .msf file" failed, it's same as "data in .msf file=null" in
> some special situations.
>    This kind of issue may produce phenomenon of "fetch all data from
> scratch".
> 
> > Namespace Command is not permitted in current state (NotAuthenticatedServerError)
> > Select Command is not permitted in current state (NotAuthenticatedServerError)
> 
> This kind of error is usually indicator of "imap server is not well
> configured".
> Why such error can so frequently occur in your server?
> Is your "imap account definition in Tb" correct/proper/valid?
> (ISP requests SSL, but server software himself supports StartTLS too.  So,
> connection/imap command via. StartTLS is usually not prohibited, but it's
> rejected when special case.)
> Do you use proxy server(including local proxy of anti-virus software) for
> your imap connection?
The server is Microsoft's Live server, so I cannot see how there could be issues with the configuration. And if there are, I couldn't change those. If there are, it would perhaps be good for many users if Thunderbird could handle Microsoft's Live server.

In short, I'm not convinced the problem is a server problem, or a user problem. I suspect Thunderbird makes a wrong decision. Or a too rigorous decision.
> Namespace Command is not permitted in current state (NotAuthenticatedServerError)
> Select Command is not permitted in current state (NotAuthenticatedServerError)

Google search for "Namespace Command is not permitted...".
> https://www.google.com/?gws_rd=ssl#q=Namespace+Command+is+not+permitted+in+current+state+%28NotAuthenticatedServerError%29
Google search for "Select Command is not permitted...".
> https://www.google.com/?gws_rd=ssl#q=Select+Command+is+not+permitted+in+current+state+%28NotAuthenticatedServerError%29
>   Some of found one.s
>     http://www.outlook-express-forum.de/windows-live-problem-5/
>     http://forums.mozillazine.org/viewtopic.php?f=39&t=2810075

It looks that Windows Live frequently falls in troubles...
What can IMAP client do when server is in trouble and it's imposible to connect or access to server repeadtedly for several days or for a while?
Such trouble and such error response in some situations is perhaps sometimes similar to "server is gone" for Tb...

Offline-store file/Auto-sync in Tb is mainly for "mail viewing while Work Offline mode" when status is "already sync'ed with imap server". So, if server connection is impossible when going back to Work Online mode, "already sync'ed with imap server" status is lost.
This is reason why I said "it's similar to Disk Cache" and "fetch all data from scratch can happen in such situation".

To improve this kind of "loss of offline-stote file content for user", feature like "per account Work Offline mode" is already requested.
   If server is temporry unavailable, set the account as "Work Offline mode server" and Tb stops server access.
   Because server access will not be attempted by Tb, "synced with server" state is kept, then Offline-store content is readable..

Comment 7

4 years ago
>If server is temporry unavailable, set the account as "Work Offline mode server" and Tb stops server access.
Actually if server is unavailable TB won't delete anything. Just change server address in the account settings to some random string and you'll see.

imo the logic could be very simple
if some folders disappear + INBOX disappears too => do not delete anything and set acc to offline mode
coz if there is no INBOX then it's clearly something goes wrong.
(Reporter)

Comment 8

4 years ago
(In reply to WADA from comment #6)
> What can IMAP client do when server is in trouble and it's imposible to
> connect or access to server repeadtedly for several days or for a while?
Well, perhaps NOT delete all my synced messages?

Better ask the questions: Why does Thunderbird wipe all locally saved messaged if a server is unavailable for a few days?

I have ticked the setting "Keep messages for this account on this computer" !

> Such trouble and such error response in some situations is perhaps sometimes
> similar to "server is gone" for Tb...
I don't know, maybe, but is it the right conclusion by TB?

> 
> Offline-store file/Auto-sync in Tb is mainly for "mail viewing while Work
> Offline mode" when status is "already sync'ed with imap server". So, if
> server connection is impossible when going back to Work Online mode,
> "already sync'ed with imap server" status is lost.
> This is reason why I said "it's similar to Disk Cache" and "fetch all data
> from scratch can happen in such situation".
Causing my 400mb mailbox to be downloaded wholly again... and again... and again... potentially.


> 
> To improve this kind of "loss of offline-stote file content for user",
> feature like "per account Work Offline mode" is already requested.
>    If server is temporry unavailable, set the account as "Work Offline mode
> server" and Tb stops server access.
>    Because server access will not be attempted by Tb, "synced with server"
> state is kept, then Offline-store content is readable..
Are you saying this is already work in progress? That would be great if it was. Can you provide a link to that work? Edit: I just read the above response telling that it won't be a solution to this problem.
(In reply to renedon from comment #8)
See dependency tree for bug 881966, and read bugs listed in the dependency tree.
bug 881966, is a report of "loss of offline-store content". bug 312619 is "feature request: per-account offline status".
(Reporter)

Comment 10

4 years ago
Ok thanks. The first one is perhaps related (881966). 

The second one (312619) is not related as it talks about a user setting. The bug discussed here is about an automated process (syncing).

Updated

4 years ago
Status: UNCONFIRMED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 157429

Comment 12

4 years ago
wow so this HUGE bug is being ignored/wontfix'd.
I unistalled firefox cause it's not locally secure and ignores the user's selections of settings silently and i guess thunderbird is bad too.

every so often even if you select offline mode it will delete every mail from the profile folder - i have been keeping a daily backup of the profile folder so i can copy-paste when it does this.

no software should just wipe a profile cause it can't find a server especially when the user has selected to keep mails for offline use...

yet again terrible coding guys.

Comment 13

4 years ago
Posted file IMAP log for wiping
Reproduced with IMAP log enabled. The log in the attachment.

1. Run TB
2. Two errors 'Namespace Command is not permitted in current state (NotAuthenticatedServerError)' and 'Select Command is not permitted in current state (NotAuthenticatedServerError)'
3. Offline folders deleted (only msgFilterRules.dat, Archives.msf and Templates.msf left in the account folder)

Comment 14

4 years ago
Here is an example of my local folders for my Outlook.com (IMAP) account being wiped upon launch of Thunderbird and connection to the server.

This session was dead afterwards.  I closed and re-launched Thunderbird and all of the account contents re-downloaded.
iirc the original bug report is related to server issues, and a misunderstanding of what "Keep messages for this account on this computer" does. This setting means keep a *cache* of the messages IN SYNC with the server - it is NOT intended to be a permanent copy.  And if the server renumbers the messages, the imap spec requires the mail client to resync such messages. If Outlook isn't behave the same, it's not following the imap spec.

I can't speak to the current state of hotmail nor outlook.com, but they generally seem to be not very stable.
Summary: Thunderbird wiping local copies of IMAP → Thunderbird wiping local copies of IMAP folder because mail server rolls UID validity and other server related errors
Whiteboard: [technical discussion: comment 3]

Comment 16

4 years ago
the best solution is to uninstall thunderbird and use something better

I have and "i'm lovin it!"

Comment 17

4 years ago
Hey 99zx6r, do tell... what are you using now?
(In reply to Serg Iv from comment #13)
> Created attachment 8670199 [details]
> IMAP log for wiping
> 
> Reproduced with IMAP log enabled. The log in the attachment.
> 
> 1. Run TB
> 2. Two errors 'Namespace Command is not permitted in current state
> (NotAuthenticatedServerError)' and 'Select Command is not permitted in
> current state (NotAuthenticatedServerError)'

bug 1206440 comment 1 explains
Status: RESOLVED → VERIFIED

Comment 19

4 years ago
(In reply to Wayne Mery (:wsmwk, use Needinfo for questions) from comment #15)
> ...
> server - it is NOT intended to be a permanent copy.  And if the server
> renumbers the messages, the imap spec requires the mail client to resync
> such messages. If Outlook isn't behave the same, it's not following the imap
> spec.
> ...

How can you say about what server does if here wasn't any server logs until I posted it? have you seen it? Pretty weird stuff in there:

4856[11b12790]: 12585800:imap-mail.outlook.com:NA:CreateNewLineFromSocket: 2 OK Authentication encountered server error. Try again later.

4856[11b12790]: login succeeded
4856[11b12790]: 12585800:imap-mail.outlook.com:A:SendData: 3 namespace

4856[11b12790]: ReadNextLine [stream=125fe650 nb=88 needmore=0]
4856[11b12790]: 12585800:imap-mail.outlook.com:A:CreateNewLineFromSocket: 3 NO Namespace Command is not permitted in current state (NotAuthenticatedServerError)
(Reporter)

Comment 20

4 years ago
(In reply to Wayne Mery (:wsmwk, use Needinfo for questions) from comment #15)
> And if the server
> renumbers the messages, the imap spec requires the mail client to resync
> such messages. If Outlook isn't behave the same, it's not following the imap
> spec.
> 
> I can't speak to the current state of hotmail nor outlook.com, but they
> generally seem to be not very stable.
Correct, Thunderbird behaves differently in that it completely wipes the local copy (1000's of messages, 100's of MiB's), and requires a total re-download of the whole mailbox.

Your last point doesn't exactly offer much since we are discussing Thunderbirds instability, regarding a function that appears to be much more stable in hotmail/outlook IMAP clients. I have *never* experienced hotmail/outlook clients wiping my local IMAP copy, but I've just had another flurry of episodes there Thunderbird wiped my local IMAP copy. I'm currently forced to re-download my e-mail account for the second time in three days (1000's of messages, 100's of MiB's).
The standards of other mail clients and how they decide to behave is there business, even it is to ignore protocol standards it is their perrogative. 

So let's put this differently, and sorry if it sounds blunt, but these are your choices:
* stay with Thunderbird and live with what you are seeing when your server continues to behave badly
* change your mail client to something other than Thunderbird
* change your server/ISP to one that behaves better
(Reporter)

Comment 22

4 years ago
AFAIK you're jumping to conclusions. I haven't seen proof of "server behaving badly", or "ignoring protocol standards" relating to this particular bug report.

Have you? All I see is my IMAP account being wiped, which doesn't appear to happen ever with other e-mail clients.

It would be good if we could see some more logging/debug information around this particular error, which provides more detail than the current transient flash of an error message which is all we see at the moment. Or even the option NOT to delete/resync the current mailbox cache, with the consequence that the user will start working offline. That would kill transient glitches and save the mailbox from being wiped unneccesarily.
You need to log in before you can comment on or make changes to this bug.