476516 - IMAP repeated re-download of entire Inbox headers with exchange

Status: RESOLVED INCOMPLETE

(MailNews Core :: Networking: IMAP, defect)

Description

[frans]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.5) Gecko/2008121622 Fedora/3.0.5-1.fc10 Firefox/3.0.5
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b3pre) Gecko/20081204 Thunderbird/3.0b1

I am accessing an Exchange 2007 server using IMAP+SSL on a Fedora Core 10.  Several times a day I will see my Inbox suddenly go blank followed by a download of all 5000+ headers.  This appears to be an Exchange-specific bug per the following thread:
http://forums.mozillazine.org/viewtopic.php?f=39&t=893585

I have the same thunderbird running on Vista with no problem. 

This problem was present on Fedora 8 and is also present in Shredder (thunderbird 3.0b1).

Reproducible: Sometimes

Steps to Reproduce:
1.
2.
3.

Comment 1

[Ludovic Hirlimann [:Usul]]

Frans could you follow the instructions at https://wiki.mozilla.org/MailNews:Logging and attach the logs to this bug ?

Comment 2

[frans]

Ludovic,
I can't attach the full logs until I can find a way to scrub personal data but I have attached the sections leading up to what appears to be the fetch of the entire inbox all over again.  This is on Thunderbird 2.0.0.19.  The first column is line numbers I added to keep track of where in the log file I was looking.

Please let me know if there is other data and/or sections I can provide.  Unfortunately this is a IMAP+SSL so packet captures won't do any good.

Comment 3

[frans]

Attachment: Snips of log file showing, potentially, where things go wrong before an entire fetch of inbox begins.

Comment 4

[Ludovic Hirlimann [:Usul]]

(In reply to comment #2)
> Ludovic,
> I can't attach the full logs until I can find a way to scrub personal data but
> I have attached the sections leading up to what appears to be the fetch of the
> entire inbox all over again.  This is on Thunderbird 2.0.0.19.  The first
> column is line numbers I added to keep track of where in the log file I was
> looking.
> 
> Please let me know if there is other data and/or sections I can provide. 
> Unfortunately this is a IMAP+SSL so packet captures won't do any good.

This looks good would even be better if you could provide the logs with a recent version of shredder b1 or a nightly will do.

Comment 5

[frans]

I will switch to shredder b1.  Please note that I do not wipe my on-disk profile when switching between versions.  Let me know if I should.

Comment 6

[Felix]

hey there, I am also on that forum thread and have had this problem for a while.

I am on win xp SP2 TB version 2.0.0.19 (20081209), Microsoft Exchange Server 2003 IMAP4rev1 server version 6.5.7638.1. I also have an enterprise blackberry setup which has it's own issues (seems to dis-reappear brand new messages to do something BB with them), AND I have a message filter that copies all incoming message to my local GMAIL IMAP instance as back-up, online access, which might also be interesting. 

I did not have a BB for a period of 4 months and had the disappearing messages problem then too. for me not all messages disappear, sometime most of ~800 (keep inbox small I know) and sometime only a few. rarely the newest messages, usually from the middle or even bottom somewhere. 

I have been logging on full blast for ever and have tons of logs will try to add something soon that shows just before messages disappearing (was staring at the screen when it happened), closed it down, restarted with new log and can show that too. 

I'll handscrape I guess, any pointer to making that process easier would help.

Comment 7

[frans]

That's interesting..  

- I believe I have also seen the partial re-download.  But less often, perhaps because it is less noticeable, than the full white-out of Inbox.
- I also have Blackberry hooked to my account.  Not sure if this should matter.. someone with better knowledge of BES/EXCHANGE/IMAP might have comment.  For what it's worth I do not have this problem with Outlook (IMAP) or Evolution mail clients.
- This problem has not bothered me on XP or Vista with Thunderbird 2.0.

As to parsing the log files, I would suggest using Unix "nl" command to line-number so you know where you are (maybe Cygwin has an equivalent if you don't have access).  The logger will output a line for each message issued a FETCH so scan through the log file looking for thousands (however many in your inbox or other folders) of FETCH commands and then scroll back.

Do you, by chance, have multiple IMAP clients open to your account at a time?

Comment 8

[Felix]

(In reply to comment #7)
> That's interesting..  
> 
> - I believe I have also seen the partial re-download.  But less often, perhaps
> because it is less noticeable, than the full white-out of Inbox.
> - I also have Blackberry hooked to my account.  Not sure if this should
> matter.. someone with better knowledge of BES/EXCHANGE/IMAP might have comment.
>  For what it's worth I do not have this problem with Outlook (IMAP) or
> Evolution mail clients.
> - This problem has not bothered me on XP or Vista with Thunderbird 2.0.
> 
> As to parsing the log files, I would suggest using Unix "nl" command to
> line-number so you know where you are (maybe Cygwin has an equivalent if you
> don't have access).  The logger will output a line for each message issued a
> FETCH so scan through the log file looking for thousands (however many in your
> inbox or other folders) of FETCH commands and then scroll back.

I'll have a look at that

> Do you, by chance, have multiple IMAP clients open to your account at a time?

I have outlook running too for calendar and have recently switched it to download emails again too (but it was kinds doing that anyway as it does not do what it's told) but had the problem since before that. 

also, the "disappearance" just happened again. when the email notification from this thread showed up, in the same instance a whole bunch of messages disappeared.  

I will process those logs when I find the time. 

I have also switched off the GMAIL IMAP copy as the fact that the messages disappear just when a new one comes in makes me kinda suspicious. we'll say whether that changes anything.

Comment 9

[Felix]

just got back to work, restarted TB just in case (inbox looked suspiciously small) and old emaisl came in after restart. so seems like it was not the GMAIL copy. at leat not only that if there was a relation. will dig into the logs now

Comment 10

[Felix]

Attachment: Log file of just before messages disappeared (I was watching)

I had just sent a message. a new message came in (reply notification fromthis bug). as that message appeared the other messages vanished. 

I shut down when all transfers were completed. startup log to follow.

Comment 11

[Felix]

Attachment: after TB restart the messages reappear

restarted TB and the messages that had just vanished re-appear. I shut down again as soon as all transfers are completed. 

folder Inbox is set to download full messages.

Comment 12

[frans]

Here are the log snips for shredder.  Please advise on whether there is anything else I can submit as I am going to have to switch to Evolution soon as the user experience is simply too problematic.

Comment 13

[frans]

Attachment: log snips for shredder

Comment 14

[David :Bienvenu]

the server seems a bit confused - it claims to support the QUOTA extension but complains whenever we ask for QUOTAROOT on a mailbox, and it also seems to like to drop the connection:

61962  -1318061168[b44051c0]: b3b0f000:secure.emailsrvr.com:S-INBOX:SendData: 15 UID fetch 38670:* (FLAGS)^M
 61963  -1318061168[b44051c0]: ReadNextLine [stream=b3b31028 nb=29 needmore=0]
 61964  -1318061168[b44051c0]: b3b0f000:secure.emailsrvr.com:S-INBOX:CreateNewLineFromSocket: * BYE Connection closed. 14^M
 
Maybe we get confused when the server drops the connection immediately after we ask for FLAGS; I'll have to look at the code.

Comment 15

[frans]

I am seeing this behavior on my Windows Vista system as well (Thunderbird 2.0.0.19) so my earlier observation about this being a Linux-specific bug is not accurate.  Please let me know if more debug data will be useful, this is a most aggravating problem.

Comment 16

[Felix]

Attachment: first thing in the morning, after hitting F5 for refresh all messages disappear

please see mark-up in file: essentially, come to desk, hit F5, whole chunk messages gone straight away, shutdown TB

Comment 17

[Felix]

Attachment: what happend on restart (after F5 incident)

restarted TB, messages come back, shut down again to save logs

Comment 18

[Felix]

seconding [e-mail]'s comments. On XP here and have this issue. new logs above.

Comment 19

[Felix]

hey, am losing my mind here with this issue... is this getting any traction? I have more logs I can post, if required. let us know how we can help. my worklife suffers a lot right now ;-) thanks!

Comment 20

[David :Bienvenu]

f5 does a check for new mail, which gives the server a chance to tell us about messages that have been removed, perhaps by a different client. That's what these expunges are, afaict.

468[58356a0]: 44bec78:ukdcdx01.SERVER:S-INBOX:CreateNewLineFromSocket: * 1212 EXPUNGE
1468[58356a0]: ReadNextLine [stream=2207db0 nb=16 needmore=0]
1468[58356a0]: 44bec78:ukdcdx01.SERVER:S-INBOX:CreateNewLineFromSocket: * 1212 EXPUNGE
1468[58356a0]: ReadNextLine [stream=2207db0 nb=16 needmore=0]
1468[58356a0]: 44bec78:ukdcdx01.SERVER:S-INBOX:CreateNewLineFromSocket: * 1212 EXPUNGE
1468[58356a0]: ReadNextLine [stream=2207db0 nb=16 needmore=0]
1468[58356a0]: 44bec78:ukdcdx01.SERVER:S-INBOX:CreateNewLineFromSocket: * 1212 EXPUNGE
1468[58356a0]: ReadNextLine [stream=2207db0 nb=16 needmore=0]
1468[58356a0]: 44bec78:ukdcdx01.SERVER:S-INBOX:CreateNewLineFromSocket: * 1212 EXPUNGE
1468[58356a0]: ReadNextLine [stream=2207db0 nb=23 needmore=0]

Comment 21

[Felix]

Attachment: another re-download (only 1 message?)

Comment 22

[Felix]

the file I just upload has more of the expunging going on. but this is not for real, i.e. not from a real other client interaction. as all those messages come back. 

the great thing is they come back to much as "NEW" that the message filter on all new messages that I have copies them to GMAIL again, like new messages. 

so no messages are being removed. they remove "themselves". again blackberry is doing funny stuff but I have this issue with or without blackberry. right now it is ridiculous. I just redownloaded 600 messages and copied them all to GMAIL again ;-)

regarding the log above, let me know what I should look for, as with this one I forgot how many messages were re-downloaded.

Comment 23

[Felix]

Attachment: the actual download (a lot of message re-downloaded)

correction: 215 MB worth of log file was re-downloaded after restarting TB. this log is the startup until the first header download.

Comment 24

[frans]

Not that anyone is doing anything with this bug, but I noticed something interesting.  In trying to figure out where my disk space was going I stumbled upon the local Thunderbird IMAP file for my Exchange server connection.  It was over 10G in size.  My actual mail store is under 300mb.  It seems, maybe, everytime thunderbird re-downloads everything it appends to it to the file.  

I've subsequently given up on using T-bird in favor of Evolution (which doesn't have this problem) but in case someone picks this bug up down the road.

Comment 25

[Felix]

interesting, was always planning to track that. but have since switched off message downloads so never had a chance. this problem is still driving me nuts although I use a different laptop a lot recently which has the "same" setup and is not as much affected for some reason. 

have also tried to tell outlook not to download any messages although I think it ignores that command. when I am back at my office machine more can report whether there was any improvement, but yeah to me this bug is a major annoyance. 

still think it's got something to do with exchange and blackberry enterprise setup, you have a BB setup too right?

Comment 26

[frans]

Yes, I have BB installed as well.  I don't know enough about the IMAP protocol to hypothesize how BB could cause these glitches though.  In theory, I believe, I should be able to rearrange the message store however I want and IMAP will simply sync up upon request.

Any developers out there following this?

Comment 27

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #24)
> It seems, maybe, everytime thunderbird re-downloads everything it appends to it to the file.  

If your "re-download every thing is caused by currupted .msf or outdated .msf", your "it appends to it to the file" can occur by "internal rebuild index". See Bug 487992.
Issue of "compact folder doesn't reduce offline-store file size" may be relevant. But Im' not sure, because Bug 466730 was already fixed(see Bug 466730 Comment #22, #23, #25).

And, if external program such as anti-virus software changes timestamp or file size of Tb's offline-store file, the "outdated .msf" condition may occur.

Comment 28

[Ludovic Hirlimann [:Usul]]

One other issue related with blackberries is : bug 482472

Comment 29

[Felix]

Hey WADA, 

don't necessarily think the inbox file growth is necessarily the issue. I have had fun for years with allt he above, and learned not to have virus protection on TB folders ;-)

what the real issue is is that messages disappear and get re-downloaded without clear pattern, see above. made [e-mail] switch client, and I am suffering ;-)

I know about "bug 482472" but I thought it's a feature. as BES or exchange or someone actually changes the message format. it comes in as plain-text, the BES magic happens and it cames back as HTML mail instead. wow!

this behaviour might have something to do with this though.

Comment 30

[Wayne Mery (:wsmwk)]

better with ftp://ftp.mozilla.org/pub/thunderbird/nightly/latest-comm-1.9.1/ (almost version 3)?

do you consider this to be a regression from version 2?  or from alpha 1?   (sorry if it's already been answered)

Comment 31

[Wayne Mery (:wsmwk)]

bienvenu, wada
probable duplicate?
xref  Bug 450128 -  Imap connection "spam" makes client "unresponsive", executing same command over and over

Felix, version 3 has fixes for exchange issues

also, frans in comment #24
> IMAP file for my Exchange server connection.  It was over 10G in size.

Comment 32

[Wayne Mery (:wsmwk)]

(In reply to comment #31)
> bienvenu, wada
> probable duplicate?
> xref  Bug 450128 -  Imap connection "spam" makes client "unresponsive",
> executing same command over and over
> 
> Felix, version 3 has fixes for exchange issues
> 
> also, frans in comment #24
> > IMAP file for my Exchange server connection.  It was over 10G in size.

perhaps bug 534835 which is "redownloads"

Comment 33

[Wayne Mery (:wsmwk)]

frans' address bounces.
felix, not sure.  iirc his address setup is weird

Comment 34

[Joshua Cranmer [:jcranmer]]

RESOLVED INCOMPLETE due to lack of response to previous comment. If you feel this change was made in error, please respond to this bug with your reasons why.