Closed Bug 534835 Opened 13 years ago Closed 13 years ago
offline folder size constantly increases using Exchange IMAP (Exchange returns wrong rfc822 size) - messages redownload
1.08 MB, application/x-zip-compressed
756.93 KB, application/x-zip-compressed
4.60 KB, patch
|Details | Diff | Splinter Review|
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; de-DE; rv:18.104.22.168) Gecko/20091102 Firefox/3.5.5 Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:22.214.171.124) Gecko/20090605 Thunderbird/126.96.36.199 Mnenhy/0.7.6.666 Using the Exchange Inbox (via IMAP) as Offline folder configured and Global search enabled the size of the local Offline file increases constantly On my account the original size of the mailbox is ~100 MB The offline file is now at 250 MB and increases after each refresh interval Another IMAP account on a Cyrus imap server is behaving normal. Reproducible: Always Steps to Reproduce: 1.Configure IMAP account on exchange server 2.Select Inbox as Offline available 3.Enable Global search Actual Results: Offline file size increases Expected Results: Offline files size increases
Frank global search should only touch the global-messages-db.sqlite file. It should not interfere with the local folders. If you disable the global indexer (see https://wiki.mozilla.org/Thunderbird:Using_Gloda) - how do your local folder behave , do they still grow ? While your at playing with prefs you might want to clean up your version string as described in https://bugzilla.mozilla.org/show_bug.cgi?id=533651#c4 . Could you provide us with an imap log as described at https://wiki.mozilla.org/MailNews:Logging#Windows so we can try to analyse what is going on. We would need Imap:5 and ImapAutoSync:5 log files. You can attach the logs using the add attachment link above.
See Also: → 532134
(In reply to comment #0) > Gecko/20090605 Thunderbird/188.8.131.52 Mnenhy/0.7.6.666 > Steps to Reproduce: >(snip) > 3.Enable Global search Tb 2 doesn't have "Global Search and Indexer" capability(it's Tb3's feature). "Global search" by what? > Actual Results: > Offline file size increases If Tb2, and if rebuild-index is executed repeatedly, file size of offline store increases upon each rebuild-index. (bug 487992. fixed bt Tb3.0rc1) Problem of bug 487992?
If i disable the global indexer and compress afterward the Inbox (Exchange) the file size of the offline file stays stable and does not increase any longer! I will attach the requested logfiles as soon as possible
Correction to my former entry : The Inbox file size is increasing without the global indexer enabled ! I see that TB is always refreshing around 100 entries of the IMAP folder (of around 3000) I will attach the logfiles
imap:5 logfile there shall be inside 2 sequences of manual refreshing the Inbox
Sorry I forgot could you log imap:5,imapAutoSync:5 both at the same time ?
Component: General → Networking: IMAP
Product: Thunderbird → MailNews Core
QA Contact: general → networking.imap
as requested with ImapAutoSync
Sigh, in this case, Exchange is not correcting the rfc822 size when we fetch the whole message. I've seen Exchange do correct it; I don't know if was a newer version of Exchange or not. If you have any control over your Exchange server, you can configure it to report the correct rfc822 size - search the web for "exchange imap rfc822.size wrong" or look here - http://social.technet.microsoft.com/Forums/en-US/exchangesvrgeneral/thread/21adbe96-e21b-458a-8242-2c3894b9d7cf http://support.microsoft.com/kb/191504 has info as well, but it seems to be down right now. I guess we'll need to add some more hacks to TB to deal with Exchange's mendacity.
Assignee: nobody → bienvenu
Status: NEW → ASSIGNED
Ok thanks but i didn't had the problem with TB 2 ;-)
yes, tb 2 didn't sanity check the message size in the offline store. Tb 3 does. Not that I'm saying Exchange is insane or anything :-)
So in fact for TB3 needs to be a workaround/hack for Exchange to behave like TB2 ? Thanks Bill ! Will this be someday go into TB3 ? Frank
(In reply to comment #12) > *** Bug 535005 has been marked as a duplicate of this bug. *** Bienvenu I dupped but you miaght want to have a peek at the log posted in comment #0
Yes, I'm going to try to get a fix for 3.01 or 3.02. Ludo, the log in bug 535005 doesn't really show the problem in action, but it does show them running a beta version of Exchange (this bug's log doesn't show the same Exchange version).
Thanks for providing a fix, so i don't need to go back to TB2
FWIW in TB2 this was resolved by setting the following: mail.imap.fetch_by_chunks = false mail.server.default.fetch_by_chunks = false
Hi i tried this settings which had worked in TB2 with TB3 but they did not help. As soon as i start to get get Inbox of the Exchange account offline the offline file size increases constantly ;-) So this workaround does not help and of course i restarted TB after all changes
This patch makes us remember the literal size on a full message fetch, and adds an xpcshell test to check that we handle a bad rfc822 size, which also involved extending the fake server to return an incorrect rfc822 size if asked.
(In reply to comment #4) > Correction to my former entry : > The Inbox file size is increasing without the global indexer enabled ! Removing "global search" from bug summary to avoid confusion, and adding "wrong rfc822 size" for ease of search.
Summary: offline folder size constantly increases using Exchange IMAP with global search enabled → offline folder size constantly increases using Exchange IMAP (Exchange returns wrong rfc822 size)
The actual code change is two lines, but the test changes are a bit more extensive. I added the ability to override the message size on the imap fakeserver, so that it would lie about the message size, and I changed the imap download test to add a case where it lies, and make sure we handle that.
this is something I feel we need for 3.01, given the failure mode, and the number of people that have run into it.
blocking-thunderbird3.0: --- → .1+
Whiteboard: [has protocol logs] → [has patch for review]
I feel the same too ;-) because i would like (better i must unfortunately) use the Exchange server
Related to https://bugzilla.mozilla.org/show_bug.cgi?id=532134 ?
quite possibly, yes, since you're using Exchange.
I'm just trying the test on trunk and I'm seeing: ###!!! ASSERTION: offline message doesn't start with From : 'VerifyOfflineMessage(m_offlineHeader, inputStream)', file /Users/moztest/comm/main/src/mailnews/base/util/nsMsgDBFolder.cpp, line 1698 Unfortunately it doesn't apply to branch due to bitrot
Do you have a more complete stack trace? That's common code, used by news and imap, and I've seen it with news, with and w/o this patch...
Stack trace of assertion (note this is on Mac): #6 0x005c773a in NS_DebugBreak_P (aSeverity=1, aStr=0x1c437ac "offline message doesn't start with From ", aExpr=0x1c43778 "VerifyOfflineMessage(m_offlineHeader, inputStream)", aFile=0x1c425d4 "/Users/moztest/comm/main/src/mailnews/base/util/nsMsgDBFolder.cpp", aLine=1698) at /Users/moztest/comm/main/src/mozilla/xpcom/base/nsDebugImpl.cpp:356 #7 0x0182477e in nsMsgDBFolder::EndNewOfflineMessage (this=0x112a400) at /Users/moztest/comm/main/src/mailnews/base/util/nsMsgDBFolder.cpp:1697 #8 0x01a9fe5a in nsImapMailFolder::NormalEndMsgWriteStream (this=0x112a400, uidOfMessage=2, markRead=0, imapUrl=0x56558d0) at /Users/moztest/comm/main/src/mailnews/imap/src/nsImapMailFolder.cpp:4480 #9 0x005d53b9 in NS_InvokeByIndex_P (that=0x112a538, methodIndex=5, paramCount=3, params=0x8250b0) at /Users/moztest/comm/main/src/mozilla/xpcom/reflect/xptcall/src/md/unix/xptcinvoke_unixish_x86.cpp:179 #10 0x005c1bc2 in nsProxyObjectCallInfo::Run (this=0x8250e0) at /Users/moztest/comm/main/src/mozilla/xpcom/proxy/src/nsProxyEvent.cpp:181 #11 0x005b66da in nsThread::ProcessNextEvent (this=0x80f490, mayWait=1, result=0xbfffd0a4) at /Users/moztest/comm/main/src/mozilla/xpcom/threads/nsThread.cpp:527 #12 0x005d53b9 in NS_InvokeByIndex_P (that=0x80f490, methodIndex=8, paramCount=2, params=0xbfffd094) at /Users/moztest/comm/main/src/mozilla/xpcom/reflect/xptcall/src/md/unix/xptcinvoke_unixish_x86.cpp:179 #13 0x009ee175 in XPCWrappedNative::CallMethod (ccx=@0xbfffd2f0, mode=XPCWrappedNative::CALL_METHOD) at /Users/moztest/comm/main/src/mozilla/js/src/xpconnect/src/xpcwrappednative.cpp:2727 #14 0x009fae99 in XPC_WN_CallMethod (cx=0x11f5000, obj=0x55d8e60, argc=1, argv=0xbfffd3d0, vp=0xbfffd3d4) at /Users/moztest/comm/main/src/mozilla/js/src/xpconnect/src/xpcwrappednativejsops.cpp:1756 #15 0x0570d3f3 in ?? () #16 0x003caa2b in ExecuteTrace (cx=0x11f5000, f=0x12d68d4, state=@0xbfffd464) at /Users/moztest/comm/main/src/mozilla/js/src/jstracer.cpp:6524 #17 0x003fdef3 in ExecuteTree (cx=0x11f5000, f=0x12d68d4, inlineCallCount=@0xbfffd8b0, innermostNestedGuardp=0xbfffd548) at /Users/moztest/comm/main/src/mozilla/js/src/jstracer.cpp:6620 #18 0x003ff294 in js_MonitorLoopEdge (cx=0x11f5000, inlineCallCount=@0xbfffd8b0, reason=Record_Branch) at /Users/moztest/comm/main/src/mozilla/js/src/jstracer.cpp:7113 #19 0x00305f17 in js_Interpret (cx=0x11f5000) at jsops.cpp:360 #20 0x0032d7cd in js_Execute (cx=0x11f5000, chain=0x55c3a40, script=0x4def790, down=0x0, flags=0, result=0xbfffed08) at jsinterp.cpp:1618 #21 0x0029b379 in JS_ExecuteScript (cx=0x11f5000, obj=0x55c3a40, script=0x4def790, rval=0xbfffed08) at /Users/moztest/comm/main/src/mozilla/js/src/jsapi.cpp:4956 #22 0x00004f36 in ProcessFile (cx=0x11f5000, obj=0x55c3a40, filename=0x0, file=0xa02455e0, forceTTY=1) at /Users/moztest/comm/main/src/mozilla/js/src/xpconnect/shell/xpcshell.cpp:1010 #23 0x000050e7 in Process (cx=0x11f5000, obj=0x55c3a40, filename=0x0, forceTTY=1) at /Users/moztest/comm/main/src/mozilla/js/src/xpconnect/shell/xpcshell.cpp:1049 #24 0x000057b1 in ProcessArgs (cx=0x11f5000, obj=0x55c3a40, argv=0xbfffef2c, argc=15) at /Users/moztest/comm/main/src/mozilla/js/src/xpconnect/shell/xpcshell.cpp:1216 #25 0x000062a4 in main (argc=15, argv=0xbfffef2c, envp=0xbfffef6c) at /Users/moztest/comm/main/src/mozilla/js/src/xpconnect/shell/xpcshell.cpp:1879
this version should work on the mac - it's against 3.01
Comment on attachment 420981 [details] [diff] [review] fix test code that broke on mac... r/sr/a=Standard8
fixed on trunk and 3.01
I'm running 3.01 and thunderbird is still redownloading my entire mailboxes. Linux/x86_64, IMAP/s server is Dovecot.
Summary: offline folder size constantly increases using Exchange IMAP (Exchange returns wrong rfc822 size) → offline folder size constantly increases using Exchange IMAP (Exchange returns wrong rfc822 size) - messages redownload
(In reply to comment #8) > Sigh, in this case, Exchange is not correcting the rfc822 size when we fetch > the whole message. I've seen Exchange do correct it; I don't know if was a > newer version of Exchange or not. I saw report of next phenomenon in other bug. Fetched message headers is different from initial fetch just after new mail arrival (fetch via IDLE in that case). Header order, headers in message header part, are altered by Exchange. In that case, added headers by spam filter were seen. I thought different UID is assinged in that case, so I didn't request detailed IMAP log, but Exchange may replace mail data and use same UID. If problem with newer Exchange, Exchange's behaviour like above may cause of prorblem.
Following is Web page found by Google search for "ms exchange rfc822.size". > http://lumbgaps.blogspot.com/2009/04/exchange-2007-imap-rfc822size-and.html Wrong RFC822.SIZE after Exchange server 2007 seems intensional RFC violation for performance reason. MS Exchange server's option(2007 or later) of "Violate RFC 3502 or not" sounds -ImapSettings -EnableExactRFC822Size:$false/$true. So, it looks that problem occurs or not depends on option setting of MS Exchange server if 2007 or later. > Build numbers and release dates for Exchange Server > http://support.microsoft.com/kb/158530
You need to log in before you can comment on or make changes to this bug.