If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

intermittent "[SERVERBUG] UID FETCH Server error" on imap.mail.yahoo.com causes the deletion and redownload of all inbox messages

RESOLVED FIXED in Thunderbird 7.0

Status

MailNews Core
Networking: IMAP
RESOLVED FIXED
7 years ago
6 years ago

People

(Reporter: al_9x, Assigned: Bienvenu)

Tracking

unspecified
Thunderbird 7.0
x86
Windows XP

Thunderbird Tracking Flags

(thunderbird6 fixed)

Details

Attachments

(1 attachment)

(Reporter)

Description

7 years ago
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.17) Gecko/20110420 Firefox/3.6.17
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.17) Gecko/20110414 Lightning/1.0b2 Thunderbird/3.1.10

sometimes when checking for new messages, an error is logged:

"The current operation on 'Inbox' did not succeed. The mail server for account: <account> responded: [SERVERBUG] UID FETCH Server error - Please try again later."

all inbox messaged are deleted:

"Deleted <all> messages from Inbox"

and subsequently refetched

Reproducible: Sometimes
(Reporter)

Comment 1

7 years ago
Why should a server error cause data loss on the client?  The only time messages should be deleted is if the server reports without any error that they are gone, and even then there should be safeguards/confirmations for bulk deletes.  But if there is an error, definitely nothing should be deleted.
(Reporter)

Comment 2

7 years ago
In general, I've noticed that Thunderbird is way too eager to blow away local messages/folders on the slightest server hiccup.

I seem to remember situations when an IMAP server would somehow misconfigure the "IMAP server directory" or some namespace, which would cause a problem with folder enum, and Thunderbird just deleted entire folders (from disk).
(Reporter)

Comment 3

7 years ago
Just noticed it again, about two hours after the last one.
(Assignee)

Comment 4

7 years ago
An imap protocol log of a session where this happens would be helpful. It's possible that the IMAP server is changing UID validity when it has the server error as well, which requires the IMAP client to blow away cached data. Calling this data loss cheapens the term, imo - the data is still there on your server, if I'm understanding you correctly.
(Reporter)

Comment 5

7 years ago
(In reply to comment #4)
> Calling this data loss cheapens the term, imo - the data is still
> there on your server

The client message store is a backup of the server, not just a perf cache, so "blowing away cached data" is a euphemism for "deleting my backup"

That you view this merely as "cached data" of course explains why TB is frequently so careless with it.  As I mentioned, over the years of using TB, I've encountered many scenarios where the all messages and/or entire folders were blown away, due to temporary server glitches.

TB should treat the message store as valuable data and go to some lengths to protect it.  For example, even when a server wants to perform a legitimate bulk delete or drop folders, the user should be warned and have an opportunity to cancel and create a more permanent backup of the current message store.
(Assignee)

Comment 6

7 years ago
Please see http://tools.ietf.org/html/rfc4549 as to why Thunderbird is REQUIRED to delete cached data when UID VALIDITY changes. Not doing that would lead directly to data loss on the server.

If UID VALIDITY is not changing, then I agree Thunderbird should not delete the cached data. That's why I asked for an IMAP Protocol log, so I can see what's going on.
(Reporter)

Comment 7

7 years ago
It doesn't look like uid validity is changing
_______________________________________________________________________________
2764[5c66dc0]: ImapThreadMainLoop entering [this=2f5f000]
0[1931140]: 2f5f000:yahoo.com:NA:SetupWithUrl: clearing IMAP_CONNECTION_IS_OPEN
2764[5c66dc0]: 2f5f000:yahoo.com:NA:ProcessCurrentURL: entering
2764[5c66dc0]: 2f5f000:yahoo.com:NA:ProcessCurrentURL:imap://<account>/select%3E/INBOX:  = currentUrl
2764[5c66dc0]: 2f5f000:yahoo.com:NA:CreateNewLineFromSocket: * OK [CAPABILITY IMAP4rev1 ID NAMESPACE X-ID-ACLID UIDPLUS LITERAL+ CHILDREN XAPPLEPUSHSERVICE XYMHIGHESTMODSEQ AUTH=PLAIN AUTH=LOGIN AUTH=XYMCOOKIE AUTH=XYMECOOKIE AUTH=XYMCOOKIEB64 AUTH=XYMPKI] IMAP4rev1 imapgate-0.7.68_5.309535 imap424.mail.ne1.yahoo.com
2764[5c66dc0]: 2f5f000:yahoo.com:NA:SendData: 1 authenticate plain
2764[5c66dc0]: 2f5f000:yahoo.com:NA:CreateNewLineFromSocket: + 
2764[5c66dc0]: 2f5f000:yahoo.com:NA:SendData: Logging suppressed for this command (it probably contained authentication information)
2764[5c66dc0]: 2f5f000:yahoo.com:NA:CreateNewLineFromSocket: 1 OK AUTHENTICATE completed - Mailbox size in bytes is 16641614
2764[5c66dc0]: 2f5f000:yahoo.com:A:SendData: 2 select "INBOX"
2764[5c66dc0]: 2f5f000:yahoo.com:A:CreateNewLineFromSocket: * 80 EXISTS
2764[5c66dc0]: 2f5f000:yahoo.com:A:CreateNewLineFromSocket: * 0 RECENT
2764[5c66dc0]: 2f5f000:yahoo.com:A:CreateNewLineFromSocket: * OK [UIDVALIDITY 1] UIDs valid
2764[5c66dc0]: 2f5f000:yahoo.com:A:CreateNewLineFromSocket: * OK [UIDNEXT 11501] Predicted next UID
2764[5c66dc0]: 2f5f000:yahoo.com:A:CreateNewLineFromSocket: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
2764[5c66dc0]: 2f5f000:yahoo.com:A:CreateNewLineFromSocket: * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft)] Permanent flags
2764[5c66dc0]: 2f5f000:yahoo.com:A:CreateNewLineFromSocket: * OK [HIGHESTMODSEQ 5614042411583794968]
2764[5c66dc0]: 2f5f000:yahoo.com:A:CreateNewLineFromSocket: 2 OK [READ-WRITE] SELECT completed; now in selected state
2764[5c66dc0]: 2f5f000:yahoo.com:S-INBOX:SendData: 3 UID fetch 1:* (FLAGS)
2764[5c66dc0]: 2f5f000:yahoo.com:S-INBOX:CreateNewLineFromSocket: 3 BAD [SERVERBUG] UID FETCH Server error - Please try again later
2764[5c66dc0]: 2f5f000:yahoo.com:S-INBOX:ProcessCurrentURL: entering
2764[5c66dc0]: 2f5f000:yahoo.com:S-INBOX:ProcessCurrentURL:imap://<accout>/folderstatus%3E/INBOX:  = currentUrl
2764[5c66dc0]: 2f5f000:yahoo.com:S-INBOX:SendData: 4 noop
2764[5c66dc0]: 2f5f000:yahoo.com:S-INBOX:CreateNewLineFromSocket: 4 OK NOOP completed
_______________________________________________________________________________
(Assignee)

Comment 8

7 years ago
OK, thx, the server is giving us an error when we're trying to determine which messages exist on the server and we're treating that as having no messages in the folder. This will prevent us from getting new headers, etc. We may have to log out of the connection since it's not useful to us, and our state is stale (e.g., we won't know about messages deleted from a different machine/client). But we definitely don't have to delete our cached data, and I'll try to come up with a patch.

Comment 9

7 years ago
> The client message store is a backup of the server, not just a perf cache

FYI, no, that's not true. The client-side store of IMAP messages is really just a cache, and *not* a backup. It is not complete, and when messages at the server are gone (e.g. due to all sorts of server errors), they are automatically deleted locally, too. This is as designed. So, if you need a backup, please make a dedicated and separate backup.
(Reporter)

Comment 10

7 years ago
(In reply to comment #9)
> > The client message store is a backup of the server, not just a perf cache
> 
> FYI, no, that's not true.

We are not dealing with natural laws here, it's not true because it hasn't been a priority, there is no reason TB couldn't be far more careful about deleting messages and folders.  This bug is an example of a unnecessary bulk delete, but even valid, non error related auto deletes, could be done more safely: warning the user, allowing the user to cancel the operation and go offline, using some sort of recycle bin for auto deleted folders and messages.
(Reporter)

Comment 11

6 years ago
(In reply to comment #8)
> and I'll try to come up with a patch.

Please backport it into the release, the problem has remained fairly consistent, and the constant redownloading of messages can only exacerbate whatever issues the server has.  This bug is fairly serious as it creates a positive feedback loop: the more errors a server has, the more load TB subjects it to, which is likely to further increase the errors ...
(Reporter)

Comment 12

6 years ago
Noticed another type of server error with the same effect (total deletion):

[INUSE] UID FETCH Mailbox in use. Please try again later.
(Assignee)

Comment 13

6 years ago
Created attachment 541859 [details] [diff] [review]
proposed fix

GetServerStateParser().LastCommandSuccessful() checks DeathSignalReceived() already so this change essentially is adding a check that !CommandFailed() && fParserState == stateOK (i.e., no syntax error).

We could probably do some earlier returns out of this method, but I'd rather just fix the issue in this bug and deal with more potential cleanup of this function elsewhere.
Assignee: nobody → dbienvenu
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Attachment #541859 - Flags: review?(neil)
(Reporter)

Comment 14

6 years ago
Can you backport it?  This problem is still fairly regular with yahoo, would be nice to have a 3.1 release that takes care of it.

How general is the solution, are there still situations where some error would cause a needless folder reset?
(Assignee)

Comment 15

6 years ago
If it's backported to anything, it would be 5.0, which comes out next week. More likely, it will just be in 6.0, which comes out roughly 6 weeks after 5.0. The fix handles errors fetching the flags causing us to think there are no messages in the folder, which is this bug.
(Reporter)

Comment 16

6 years ago
(In reply to comment #15)
> If it's backported to anything, it would be 5.0, which comes out next week.
> More likely, it will just be in 6.0, which comes out roughly 6 weeks after
> 5.0.

Will you try to get it into 5.0?
(Assignee)

Comment 17

6 years ago
5.0 is coming out Tuesday, and this hasn't even been reviewed or landed on the trunk yet, so, no.
Comment on attachment 541859 [details] [diff] [review]
proposed fix

>   if (!DeathSignalReceived() && GetServerStateParser().NumberOfMessages())
Was it intentional to leave some calls to DeathSignalReceived?
(Assignee)

Comment 19

6 years ago
(In reply to comment #18)
> Comment on attachment 541859 [details] [diff] [review] [review]
> proposed fix
> 
> >   if (!DeathSignalReceived() && GetServerStateParser().NumberOfMessages())
> Was it intentional to leave some calls to DeathSignalReceived?

Yes, I didn't want failures getting the quota info to prevent you from using an imap folder.
(Reporter)

Comment 20

6 years ago
(In reply to comment #17)
> 5.0 is coming out Tuesday, and this hasn't even been reviewed or landed on
> the trunk yet, so, no.

Are there going to be interim releases you can get this it into?

If one's inbox is sufficiently large it could result in nearly continuous syncing.  By the time one lengthy sync completes another UID FETCH error resets the inbox and starts another one.  With enough users, this becomes an inadvertent DOS.  Is it any wonder that yahoo has occasional difficulties fetching 17 messages (Bug 668143)?  You should release this fix as soon as possible.
(Assignee)

Comment 21

6 years ago
we're releasing every 6 weeks. We're not doing interim releases between those releases because we have very limited resources. And again, this patch hasn't even been reviewed.
(Assignee)

Comment 22

6 years ago
pinging for review.

Updated

6 years ago
Attachment #541859 - Flags: review?(neil) → review+
(Assignee)

Comment 23

6 years ago
fixed on trunk - http://hg.mozilla.org/comm-central/rev/e307f0c7a8f1
Status: ASSIGNED → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
(Assignee)

Updated

6 years ago
Attachment #541859 - Flags: approval-comm-aurora?
Attachment #541859 - Flags: approval-comm-aurora? → approval-comm-aurora+
Checked into aurora: http://hg.mozilla.org/releases/comm-aurora/rev/167637e28eff
status-thunderbird6: --- → fixed
Target Milestone: --- → Thunderbird 7.0
You need to log in before you can comment on or make changes to this bug.