Synchronization and indexing done continuously, repeatedly (imap message fetch, MS Exchange server returns wrong rfc822.size)

RESOLVED FIXED in Thunderbird 3.0rc1

Status

defect
--
major
RESOLVED FIXED
10 years ago
9 years ago

People

(Reporter: asolkar, Unassigned)

Tracking

({perf, regression})

unspecified
Thunderbird 3.0rc1
Bug Flags:
blocking-thunderbird3 +

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [no l10n impact])

Attachments

(1 attachment)

Reporter

Description

10 years ago
User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.4pre) Gecko/20090924 Lightning/1.0pre Shredder/3.0pre

On my MS Exchange server connected via IMAP, I have a few folders that have a couple thousand messages each. I am seeing that these folders are continuously and repeatedly being synchronized and indexed.

Reproducible: Always

Steps to Reproduce:
1. Find an MS Exchange account with folders containing large number of messages
2. Connect with Shredder version mentioned in the bug
3. Subscribe to the folders
Actual Results:  
Everything works as expected, except that the folders are continuously and repeatedly synchronized and indexed in their entirety (if a folder has 2500 messages, all 2500 messages seem to be downloaded again - at least that is what appears looking at the Activity Manager) 

Expected Results:  
Messages once downloaded should be skipped in subsequent synchronization attempts. The first synchronization should take long and following ones should be quick.

This constant activity keeps my network and CPU very busy. Which is very unsettling!

This happens especially when I navigate into these folders.

Comment 1

10 years ago
dup, fixed in today's nightly build, if you want to try that - http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-comm-1.9.1/
Status: UNCONFIRMED → RESOLVED
Last Resolved: 10 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 517466

Comment 2

10 years ago
Exchange has its own issues - un-duping.
Status: RESOLVED → UNCONFIRMED
Resolution: DUPLICATE → ---

Comment 3

10 years ago
See https://bugzilla.mozilla.org/show_bug.cgi?id=517466#c33 for more info.

There are several ways to work around the Exchange problem - one is to make sure that the offline download succeeds the second time, once Exchange has figured out how big the message is. An other is to detect Exchange server and just turn off the size comparison in the offline download code. A third approach would be to set the message size based on the size of the string literal the server gives us when we ask for the whole message.
Assignee: nobody → bienvenu
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Flags: blocking-thunderbird3+
Target Milestone: --- → Thunderbird 3.0rc1
Whiteboard: [no l10n impact]

Comment 4

10 years ago
Posted patch proposed fixSplinter Review
This patch works around the Exchange bug as follows:

When we do a body fetch for offline use, pass the rfc822.size the server gave us when we did the fetch. Set the msg size on the msg hdr to this size.

This works because Exchange figures out the actual size as a side effect of fetching the whole message, so the second time we try to fetch the message, it gives us the right size, and we use it.

I also tweaked the offline download code to throw away the part of the offline store that just had a partial downloaded message in it, so the offline store doesn't grow unnecessarily. 

I wasn't able to test this in the wild because our Exchange server seems to always get the size wrong in the direction we're forgiving of - it always seems to think the message is smaller than it actually is, and we handle getting more data than we expected. But I was able to poke at things in the debugger and verify that this code is run and works.
Attachment #403517 - Flags: superreview?(bugzilla)
Attachment #403517 - Flags: review?(bugzilla)

Comment 5

10 years ago
I can generate a try server build if anyone is interested in testing this patch.
Whiteboard: [no l10n impact] → [no l10n impact][has patch for review]

Comment 6

10 years ago
Try server builds I requested should show up here this afternoon - http://tinderbox.mozilla.org/showbuilds.cgi?tree=ThunderbirdTry

Comment 7

10 years ago
Also, if you have any control over your Exchange server, you can fix this on the server side - http://support.microsoft.com/default.aspx/kb/191504
Reporter

Comment 8

10 years ago
(In reply to comment #6)
> Try server builds I requested should show up here this afternoon -
> http://tinderbox.mozilla.org/showbuilds.cgi?tree=ThunderbirdTry

I tested this build (dascher@mozillamessaging.com-472764v2-mail-try-linux.tar.bz2). I wonder if this is the right build, because I did not see any change in behavior. Messages in many folders were still being downloaded repeatedly.

Is there any log/transcript I can provide to help debug?

Comment 9

10 years ago
my try server build will have my name (bienvenu) on it. It should be out around 1PM PST.
Reporter

Comment 10

10 years ago
I was hoping so. I'll wait until later today and test the correct one.

Comment 13

10 years ago
I just tried out the Windows build.  After re-downloading all my IMAP folders it seemed to work, although I was still seeing 9 messages (in a folder with 1034 messages) downloaded every time I visited it.  It flashed up so quickly that I can barely read it, but it happened every time.

So I quit Shredder, moved aside the local folder copy and the .msf file and restarted.  After it rebuilt the file, the first two or three times I revisited that folder I still got the "Downloading..." behavior, but after that it settled down and seems to be working fine.  Now I seem to be able to visit all my IMAP folders without any problem.

Thanks for the quick turnaround on this bug fix!
Reporter

Comment 14

10 years ago
(In reply to comment #12)
> and windows build here -
> http://s3.mozillamessaging.com/build/try-server/2009-09-29_11:40-bienvenu@nventure.com-1254249347/bienvenu@nventure.com-1254249347-mail-try-win32.installer.exe
> 
> mac build failed - let me know if anyone wants a mac build to try.

I've been using this build for about an hour now. I can confirm that the excessive network/CPU activity due to synchronizing/indexing is not there anymore. I'll continue to use this build for the next day or so and report back if I find anything interesting.
Reporter

Comment 15

10 years ago
(In reply to comment #14)
> (In reply to comment #12)
> > and windows build here -
> > http://s3.mozillamessaging.com/build/try-server/2009-09-29_11:40-bienvenu@nventure.com-1254249347/bienvenu@nventure.com-1254249347-mail-try-win32.installer.exe
> > 
> > mac build failed - let me know if anyone wants a mac build to try.
> 
> I've been using this build for about an hour now. I can confirm that the
> excessive network/CPU activity due to synchronizing/indexing is not there
> anymore. I'll continue to use this build for the next day or so and report back
> if I find anything interesting.

Sorry, I used the Linux build. I thought I replied to the Linux comment.
Comment on attachment 403517 [details] [diff] [review]
proposed fix

This seems to do the right thing - we've got positive comments from those seeing this issue and it doesn't regress anything on my server at least.

Lets get this in and see if it helps a bit more.
Attachment #403517 - Flags: superreview?(bugzilla)
Attachment #403517 - Flags: superreview+
Attachment #403517 - Flags: review?(bugzilla)
Attachment #403517 - Flags: review+
OS: Linux → All
Hardware: x86 → All
Whiteboard: [no l10n impact][has patch for review] → [no l10n impact][ready to land]

Comment 17

10 years ago
fix checked in.
Status: ASSIGNED → RESOLVED
Last Resolved: 10 years ago10 years ago
Resolution: --- → FIXED

Comment 18

10 years ago
thx to those who tried the try server builds.
Whiteboard: [no l10n impact][ready to land] → [no l10n impact]

Updated

10 years ago
Duplicate of this bug: 519749

Updated

10 years ago
Duplicate of this bug: 522889

Updated

10 years ago
Assignee: bienvenu → nobody
Severity: normal → major
Component: General → Networking: IMAP
Keywords: perf
Product: Thunderbird → MailNews Core
QA Contact: general → networking.imap
Summary: Synchronization and indexing done continuously, repeatedly → Synchronization and indexing done continuously, repeatedly (imap message fetch)

Updated

10 years ago
Keywords: regression
Duplicate of this bug: 528164

Updated

10 years ago
Summary: Synchronization and indexing done continuously, repeatedly (imap message fetch) → Synchronization and indexing done continuously, repeatedly (imap message fetch), affects exchange

Comment 22

9 years ago
I don't have anything to add, except that I am having this problem with my GMail account using IMAP protocol. Currently, I have unchecked download every xx minutes, and only download upon startup or manual download.

I'm running Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.8) Gecko/20100227 Lightning/1.0b1 Thunderbird/3.0.3.

Comment 23

9 years ago
This bug is not fixed in 3.0.3 I'm getting continuous re-indexing with dovecot IMAP on windows. There are rather big mailboxes (180K, 65K, 3K messages) and it seems to be occurring for all those folders.
(In reply to comment #23)
> This bug is not fixed in 3.0.3 I'm getting continuous re-indexing with dovecot
> IMAP on windows. There are rather big mailboxes (180K, 65K, 3K messages) and it
> seems to be occurring for all those folders.

Well given this bug mentions exchange quite a bit, I would suggest that you file a new bug with full details of what you're seeing.
Summary: Synchronization and indexing done continuously, repeatedly (imap message fetch), affects exchange → Synchronization and indexing done continuously, repeatedly (imap message fetch, MS Exchange server returns wrong rfc822.size)
You need to log in before you can comment on or make changes to this bug.