518702 - Synchronization and indexing done continuously, repeatedly (imap message fetch, MS Exchange server returns wrong rfc822.size)

Status: RESOLVED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[Mahesh Asolkar]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.4pre) Gecko/20090924 Lightning/1.0pre Shredder/3.0pre

On my MS Exchange server connected via IMAP, I have a few folders that have a couple thousand messages each. I am seeing that these folders are continuously and repeatedly being synchronized and indexed.

Reproducible: Always

Steps to Reproduce:
1. Find an MS Exchange account with folders containing large number of messages
2. Connect with Shredder version mentioned in the bug
3. Subscribe to the folders
Actual Results:  
Everything works as expected, except that the folders are continuously and repeatedly synchronized and indexed in their entirety (if a folder has 2500 messages, all 2500 messages seem to be downloaded again - at least that is what appears looking at the Activity Manager) 

Expected Results:  
Messages once downloaded should be skipped in subsequent synchronization attempts. The first synchronization should take long and following ones should be quick.

This constant activity keeps my network and CPU very busy. Which is very unsettling!

This happens especially when I navigate into these folders.

Comment 1

[David :Bienvenu]

dup, fixed in today's nightly build, if you want to try that - http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-comm-1.9.1/

Comment 2

[David :Bienvenu]

Exchange has its own issues - un-duping.

Comment 3

[David :Bienvenu]

See https://bugzilla.mozilla.org/show_bug.cgi?id=517466#c33 for more info.

There are several ways to work around the Exchange problem - one is to make sure that the offline download succeeds the second time, once Exchange has figured out how big the message is. An other is to detect Exchange server and just turn off the size comparison in the offline download code. A third approach would be to set the message size based on the size of the string literal the server gives us when we ask for the whole message.

Comment 4

[David :Bienvenu]

Attachment: proposed fix

This patch works around the Exchange bug as follows:

When we do a body fetch for offline use, pass the rfc822.size the server gave us when we did the fetch. Set the msg size on the msg hdr to this size.

This works because Exchange figures out the actual size as a side effect of fetching the whole message, so the second time we try to fetch the message, it gives us the right size, and we use it.

I also tweaked the offline download code to throw away the part of the offline store that just had a partial downloaded message in it, so the offline store doesn't grow unnecessarily. 

I wasn't able to test this in the wild because our Exchange server seems to always get the size wrong in the direction we're forgiving of - it always seems to think the message is smaller than it actually is, and we handle getting more data than we expected. But I was able to poke at things in the debugger and verify that this code is run and works.

Comment 5

[David :Bienvenu]

I can generate a try server build if anyone is interested in testing this patch.

Comment 6

[David :Bienvenu]

Try server builds I requested should show up here this afternoon - http://tinderbox.mozilla.org/showbuilds.cgi?tree=ThunderbirdTry

Comment 7

[David :Bienvenu]

Also, if you have any control over your Exchange server, you can fix this on the server side - http://support.microsoft.com/default.aspx/kb/191504

Comment 8

[Mahesh Asolkar]

(In reply to comment #6)
> Try server builds I requested should show up here this afternoon -
> http://tinderbox.mozilla.org/showbuilds.cgi?tree=ThunderbirdTry

I tested this build ([e-mail]2). I wonder if this is the right build, because I did not see any change in behavior. Messages in many folders were still being downloaded repeatedly.

Is there any log/transcript I can provide to help debug?

Comment 9

[David :Bienvenu]

my try server build will have my name (bienvenu) on it. It should be out around 1PM PST.

Comment 10

[Mahesh Asolkar]

I was hoping so. I'll wait until later today and test the correct one.

Comment 11

[David :Bienvenu]

linux try server build is available now - 
http://s3.mozillamessaging.com/build/try-server/2009-09-29_11:[e-mail]-1254249347/[e-mail]2

Comment 12

[David :Bienvenu]

and windows build here - http://s3.mozillamessaging.com/build/try-server/2009-09-29_11:[e-mail]-1254249347/[e-mail]

mac build failed - let me know if anyone wants a mac build to try.

Comment 13

[Phil Johnson]

I just tried out the Windows build.  After re-downloading all my IMAP folders it seemed to work, although I was still seeing 9 messages (in a folder with 1034 messages) downloaded every time I visited it.  It flashed up so quickly that I can barely read it, but it happened every time.

So I quit Shredder, moved aside the local folder copy and the .msf file and restarted.  After it rebuilt the file, the first two or three times I revisited that folder I still got the "Downloading..." behavior, but after that it settled down and seems to be working fine.  Now I seem to be able to visit all my IMAP folders without any problem.

Thanks for the quick turnaround on this bug fix!

Comment 14

[Mahesh Asolkar]

(In reply to comment #12)
> and windows build here -
> http://s3.mozillamessaging.com/build/try-server/2009-09-29_11:[e-mail]-1254249347/[e-mail]
> 
> mac build failed - let me know if anyone wants a mac build to try.

I've been using this build for about an hour now. I can confirm that the excessive network/CPU activity due to synchronizing/indexing is not there anymore. I'll continue to use this build for the next day or so and report back if I find anything interesting.

Comment 15

[Mahesh Asolkar]

(In reply to comment #14)
> (In reply to comment #12)
> > and windows build here -
> > http://s3.mozillamessaging.com/build/try-server/2009-09-29_11:[e-mail]-1254249347/[e-mail]
> > 
> > mac build failed - let me know if anyone wants a mac build to try.
> 
> I've been using this build for about an hour now. I can confirm that the
> excessive network/CPU activity due to synchronizing/indexing is not there
> anymore. I'll continue to use this build for the next day or so and report back
> if I find anything interesting.

Sorry, I used the Linux build. I thought I replied to the Linux comment.

Comment 16

[Mark Banner (:standard8)]

Comment on attachment 403517 [details] [diff] [review]
proposed fix

This seems to do the right thing - we've got positive comments from those seeing this issue and it doesn't regress anything on my server at least.

Lets get this in and see if it helps a bit more.

Comment 17

[David :Bienvenu]

fix checked in.

Comment 18

[David :Bienvenu]

thx to those who tried the try server builds.

Comment 22

[dollarbang]

I don't have anything to add, except that I am having this problem with my GMail account using IMAP protocol. Currently, I have unchecked download every xx minutes, and only download upon startup or manual download.

I'm running Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.8) Gecko/20100227 Lightning/1.0b1 Thunderbird/3.0.3.

Comment 23

[George M]

This bug is not fixed in 3.0.3 I'm getting continuous re-indexing with dovecot IMAP on windows. There are rather big mailboxes (180K, 65K, 3K messages) and it seems to be occurring for all those folders.

Comment 24

[Mark Banner (:standard8)]

(In reply to comment #23)
> This bug is not fixed in 3.0.3 I'm getting continuous re-indexing with dovecot
> IMAP on windows. There are rather big mailboxes (180K, 65K, 3K messages) and it
> seems to be occurring for all those folders.

Well given this bug mentions exchange quite a bit, I would suggest that you file a new bug with full details of what you're seeing.