Last Comment Bug 534835 - offline folder size constantly increases using Exchange IMAP (Exchange returns wrong rfc822 size) - messages redownload
: offline folder size constantly increases using Exchange IMAP (Exchange return...
Status: RESOLVED FIXED
[gs]
: fixed-seamonkey2.0.3, perf
Product: MailNews Core
Classification: Components
Component: Networking: IMAP (show other bugs)
: unspecified
: x86 Windows XP
: -- major (vote)
: Thunderbird 3.1a1
Assigned To: David :Bienvenu
:
:
Mentors:
http://gsfn.us/t/ogo5
: 532134 534297 535005 542816 (view as bug list)
Depends on: 544396
Blocks:
  Show dependency treegraph
 
Reported: 2009-12-15 01:26 PST by Frank Zimmer
Modified: 2010-05-27 03:45 PDT (History)
12 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
.1+
.1-fixed


Attachments
imap:5 logfile (1.08 MB, application/x-zip-compressed)
2009-12-15 03:31 PST, Frank Zimmer
no flags Details
imap:5 + ImapAutoSync:5 (756.93 KB, application/x-zip-compressed)
2009-12-15 04:50 PST, Frank Zimmer
no flags Details
wip (3.68 KB, patch)
2009-12-30 17:21 PST, David :Bienvenu
no flags Details | Diff | Splinter Review
proposed fix (6.06 KB, patch)
2010-01-04 11:28 PST, David :Bienvenu
no flags Details | Diff | Splinter Review
fix test code that broke on mac... (4.60 KB, patch)
2010-01-10 14:08 PST, David :Bienvenu
standard8: review+
standard8: superreview+
standard8: approval‑thunderbird3.0.1+
Details | Diff | Splinter Review

Description Frank Zimmer 2009-12-15 01:26:19 PST
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; de-DE; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.22) Gecko/20090605 Thunderbird/2.0.0.22 Mnenhy/0.7.6.666

Using the Exchange Inbox (via IMAP) as Offline folder configured and Global search enabled the size of the local Offline file increases constantly
On my account the original size of the mailbox is ~100 MB 
The offline file is now at 250 MB and increases after each refresh interval
Another IMAP account on a Cyrus imap server is behaving normal.

Reproducible: Always

Steps to Reproduce:
1.Configure IMAP account on exchange server
2.Select Inbox as Offline available
3.Enable Global search
Actual Results:  
Offline file size increases

Expected Results:  
Offline files size increases
Comment 1 Ludovic Hirlimann [:Usul] 2009-12-15 01:38:21 PST
Frank global search should only touch the global-messages-db.sqlite file. It should not interfere with the local folders.

If you disable the global indexer (see https://wiki.mozilla.org/Thunderbird:Using_Gloda) - how do your local folder behave , do they still grow ?

While your at playing with prefs you might want to clean up your version string as described in https://bugzilla.mozilla.org/show_bug.cgi?id=533651#c4 .

Could you provide us with an imap log as described at https://wiki.mozilla.org/MailNews:Logging#Windows so we can try to analyse what is going on. We would need Imap:5 and ImapAutoSync:5 log files. You can attach the logs using the add attachment link above.
Comment 2 WADA 2009-12-15 02:28:00 PST
(In reply to comment #0)
> Gecko/20090605 Thunderbird/2.0.0.22 Mnenhy/0.7.6.666

> Steps to Reproduce:
>(snip)
> 3.Enable Global search

Tb 2 doesn't have "Global Search and Indexer" capability(it's Tb3's feature).
"Global search" by what?
 
> Actual Results:  
> Offline file size increases

If Tb2, and if rebuild-index is executed repeatedly, file size of offline store increases upon each rebuild-index. (bug 487992. fixed bt Tb3.0rc1)
Problem of bug 487992?
Comment 3 Frank Zimmer 2009-12-15 03:13:36 PST
If i disable the global indexer and compress afterward the Inbox (Exchange)
the file size of the offline file stays stable and does not increase any longer!

I will attach the requested logfiles as soon as possible
Comment 4 Frank Zimmer 2009-12-15 03:23:35 PST
Correction to my former entry :

The Inbox file size is increasing without the global indexer enabled !
I see that TB is always refreshing around 100 entries of the IMAP folder (of around 3000) 

I will attach the logfiles
Comment 5 Frank Zimmer 2009-12-15 03:31:46 PST
Created attachment 417665 [details]
imap:5 logfile

imap:5 logfile
there shall be inside 2 sequences of manual refreshing the Inbox
Comment 6 Ludovic Hirlimann [:Usul] 2009-12-15 04:01:20 PST
Sorry I forgot could you log imap:5,imapAutoSync:5 both at the same time ?
Comment 7 Frank Zimmer 2009-12-15 04:50:30 PST
Created attachment 417671 [details]
imap:5 + ImapAutoSync:5

as requested with ImapAutoSync
Comment 8 David :Bienvenu 2009-12-15 07:16:42 PST
Sigh, in this case, Exchange is not correcting the rfc822 size when we fetch the whole message. I've seen Exchange do correct it; I don't know if was a newer version of Exchange or not. If you have any control over your Exchange server, you can configure it to report the correct rfc822 size - search the web for "exchange imap rfc822.size wrong"  or look here - http://social.technet.microsoft.com/Forums/en-US/exchangesvrgeneral/thread/21adbe96-e21b-458a-8242-2c3894b9d7cf

http://support.microsoft.com/kb/191504 has info as well, but it seems to be down right now.

I guess we'll need to add some more hacks to TB to deal with Exchange's mendacity.
Comment 9 Frank Zimmer 2009-12-15 07:58:03 PST
Ok thanks
but i didn't had the problem with TB 2 ;-)
Comment 10 David :Bienvenu 2009-12-15 08:34:09 PST
yes, tb 2 didn't sanity check the message size in the offline store. Tb 3 does. Not that I'm saying Exchange is insane or anything :-)
Comment 11 Frank Zimmer 2009-12-16 00:06:07 PST
So in fact for TB3 needs to be a workaround/hack for Exchange to behave like TB2 ?
Thanks Bill !

Will this be someday go into TB3 ?

Frank
Comment 12 Ludovic Hirlimann [:Usul] 2009-12-16 05:51:10 PST
*** Bug 535005 has been marked as a duplicate of this bug. ***
Comment 13 Ludovic Hirlimann [:Usul] 2009-12-16 05:51:48 PST
(In reply to comment #12)
> *** Bug 535005 has been marked as a duplicate of this bug. ***

Bienvenu I dupped but you miaght want to have a peek at the log posted in comment #0
Comment 14 David :Bienvenu 2009-12-16 06:47:45 PST
Yes, I'm going to try to get a fix for 3.01 or 3.02. Ludo, the log in bug 535005 doesn't really show the problem in action, but it does show them running a beta version of Exchange (this bug's log doesn't show the same Exchange version).
Comment 15 Frank Zimmer 2009-12-16 07:32:07 PST
Thanks for providing a fix, so i don't need to go back to TB2
Comment 16 virgo.stellar.stream 2009-12-17 15:20:13 PST
FWIW in TB2 this was resolved by setting the following:
mail.imap.fetch_by_chunks = false
mail.server.default.fetch_by_chunks = false
Comment 17 Frank Zimmer 2009-12-18 03:06:56 PST
Hi
i tried this settings which had worked in TB2 with TB3 but they did not
help.
As soon as i start to get get Inbox of the Exchange account offline the
offline file size increases constantly ;-)
So this workaround does not help
and

of course i restarted TB after all changes
Comment 18 David :Bienvenu 2009-12-30 17:21:45 PST
Created attachment 419629 [details] [diff] [review]
wip

This patch makes us remember the literal size on a full message fetch, and adds an xpcshell test to check that we handle a bad rfc822 size, which also involved extending the fake server to return an incorrect rfc822 size if asked.
Comment 19 WADA 2009-12-31 03:02:37 PST
(In reply to comment #4)
> Correction to my former entry :
> The Inbox file size is increasing without the global indexer enabled !

Removing "global search" from bug summary to avoid confusion, and adding "wrong rfc822 size" for ease of search.
Comment 20 David :Bienvenu 2010-01-04 11:28:01 PST
Created attachment 419940 [details] [diff] [review]
proposed fix

The actual code change is two lines, but the test changes are a bit more extensive. I added the ability to override the message size on the imap fakeserver, so that it would lie about the message size, and I changed the imap download test to add a case where it lies, and make sure we handle that.
Comment 21 David :Bienvenu 2010-01-04 11:29:00 PST
this is something I feel we need for 3.01, given the failure mode, and the number of people that have run into it.
Comment 22 Frank Zimmer 2010-01-05 00:47:08 PST
I feel the same too ;-)
because i would like (better i must unfortunately) use the Exchange server
Comment 23 Jasper Knockaert 2010-01-05 13:56:22 PST
Related to https://bugzilla.mozilla.org/show_bug.cgi?id=532134 ?
Comment 24 David :Bienvenu 2010-01-05 14:00:39 PST
quite possibly, yes, since you're using Exchange.
Comment 25 David :Bienvenu 2010-01-05 14:00:55 PST
*** Bug 532134 has been marked as a duplicate of this bug. ***
Comment 26 Mark Banner (:standard8, limited time in Dec) 2010-01-10 08:01:53 PST
I'm just trying the test on trunk and I'm seeing:

###!!! ASSERTION: offline message doesn't start with From : 'VerifyOfflineMessage(m_offlineHeader, inputStream)', file /Users/moztest/comm/main/src/mailnews/base/util/nsMsgDBFolder.cpp, line 1698

Unfortunately it doesn't apply to branch due to bitrot
Comment 27 David :Bienvenu 2010-01-10 11:30:44 PST
Do you have a more complete stack trace? That's common code, used by news and imap, and I've seen it with news, with and w/o this patch...
Comment 28 Mark Banner (:standard8, limited time in Dec) 2010-01-10 12:07:11 PST
Stack trace of assertion (note this is on Mac):

#6  0x005c773a in NS_DebugBreak_P (aSeverity=1, aStr=0x1c437ac "offline message doesn't start with From ", aExpr=0x1c43778 "VerifyOfflineMessage(m_offlineHeader, inputStream)", aFile=0x1c425d4 "/Users/moztest/comm/main/src/mailnews/base/util/nsMsgDBFolder.cpp", aLine=1698) at /Users/moztest/comm/main/src/mozilla/xpcom/base/nsDebugImpl.cpp:356
#7  0x0182477e in nsMsgDBFolder::EndNewOfflineMessage (this=0x112a400) at /Users/moztest/comm/main/src/mailnews/base/util/nsMsgDBFolder.cpp:1697
#8  0x01a9fe5a in nsImapMailFolder::NormalEndMsgWriteStream (this=0x112a400, uidOfMessage=2, markRead=0, imapUrl=0x56558d0) at /Users/moztest/comm/main/src/mailnews/imap/src/nsImapMailFolder.cpp:4480
#9  0x005d53b9 in NS_InvokeByIndex_P (that=0x112a538, methodIndex=5, paramCount=3, params=0x8250b0) at /Users/moztest/comm/main/src/mozilla/xpcom/reflect/xptcall/src/md/unix/xptcinvoke_unixish_x86.cpp:179
#10 0x005c1bc2 in nsProxyObjectCallInfo::Run (this=0x8250e0) at /Users/moztest/comm/main/src/mozilla/xpcom/proxy/src/nsProxyEvent.cpp:181
#11 0x005b66da in nsThread::ProcessNextEvent (this=0x80f490, mayWait=1, result=0xbfffd0a4) at /Users/moztest/comm/main/src/mozilla/xpcom/threads/nsThread.cpp:527
#12 0x005d53b9 in NS_InvokeByIndex_P (that=0x80f490, methodIndex=8, paramCount=2, params=0xbfffd094) at /Users/moztest/comm/main/src/mozilla/xpcom/reflect/xptcall/src/md/unix/xptcinvoke_unixish_x86.cpp:179
#13 0x009ee175 in XPCWrappedNative::CallMethod (ccx=@0xbfffd2f0, mode=XPCWrappedNative::CALL_METHOD) at /Users/moztest/comm/main/src/mozilla/js/src/xpconnect/src/xpcwrappednative.cpp:2727
#14 0x009fae99 in XPC_WN_CallMethod (cx=0x11f5000, obj=0x55d8e60, argc=1, argv=0xbfffd3d0, vp=0xbfffd3d4) at /Users/moztest/comm/main/src/mozilla/js/src/xpconnect/src/xpcwrappednativejsops.cpp:1756
#15 0x0570d3f3 in ?? ()
#16 0x003caa2b in ExecuteTrace (cx=0x11f5000, f=0x12d68d4, state=@0xbfffd464) at /Users/moztest/comm/main/src/mozilla/js/src/jstracer.cpp:6524
#17 0x003fdef3 in ExecuteTree (cx=0x11f5000, f=0x12d68d4, inlineCallCount=@0xbfffd8b0, innermostNestedGuardp=0xbfffd548) at /Users/moztest/comm/main/src/mozilla/js/src/jstracer.cpp:6620
#18 0x003ff294 in js_MonitorLoopEdge (cx=0x11f5000, inlineCallCount=@0xbfffd8b0, reason=Record_Branch) at /Users/moztest/comm/main/src/mozilla/js/src/jstracer.cpp:7113
#19 0x00305f17 in js_Interpret (cx=0x11f5000) at jsops.cpp:360
#20 0x0032d7cd in js_Execute (cx=0x11f5000, chain=0x55c3a40, script=0x4def790, down=0x0, flags=0, result=0xbfffed08) at jsinterp.cpp:1618
#21 0x0029b379 in JS_ExecuteScript (cx=0x11f5000, obj=0x55c3a40, script=0x4def790, rval=0xbfffed08) at /Users/moztest/comm/main/src/mozilla/js/src/jsapi.cpp:4956
#22 0x00004f36 in ProcessFile (cx=0x11f5000, obj=0x55c3a40, filename=0x0, file=0xa02455e0, forceTTY=1) at /Users/moztest/comm/main/src/mozilla/js/src/xpconnect/shell/xpcshell.cpp:1010
#23 0x000050e7 in Process (cx=0x11f5000, obj=0x55c3a40, filename=0x0, forceTTY=1) at /Users/moztest/comm/main/src/mozilla/js/src/xpconnect/shell/xpcshell.cpp:1049
#24 0x000057b1 in ProcessArgs (cx=0x11f5000, obj=0x55c3a40, argv=0xbfffef2c, argc=15) at /Users/moztest/comm/main/src/mozilla/js/src/xpconnect/shell/xpcshell.cpp:1216
#25 0x000062a4 in main (argc=15, argv=0xbfffef2c, envp=0xbfffef6c) at /Users/moztest/comm/main/src/mozilla/js/src/xpconnect/shell/xpcshell.cpp:1879
Comment 29 David :Bienvenu 2010-01-10 14:08:40 PST
Created attachment 420981 [details] [diff] [review]
fix test code that broke on mac...

this version should work on the mac - it's against 3.01
Comment 30 Mark Banner (:standard8, limited time in Dec) 2010-01-10 14:21:22 PST
Comment on attachment 420981 [details] [diff] [review]
fix test code that broke on mac...

r/sr/a=Standard8
Comment 31 David :Bienvenu 2010-01-10 15:48:41 PST
fixed on trunk and 3.01
Comment 32 David :Bienvenu 2010-01-15 09:25:50 PST
*** Bug 534297 has been marked as a duplicate of this bug. ***
Comment 33 Blu3 2010-01-24 12:13:07 PST
I'm running 3.01 and thunderbird is still redownloading my entire mailboxes.

Linux/x86_64, IMAP/s server is Dovecot.
Comment 34 Ludovic Hirlimann [:Usul] 2010-01-31 03:05:10 PST
*** Bug 542816 has been marked as a duplicate of this bug. ***
Comment 35 WADA 2010-05-26 23:23:43 PDT
(In reply to comment #8)
> Sigh, in this case, Exchange is not correcting the rfc822 size when we fetch
> the whole message. I've seen Exchange do correct it; I don't know if was a
> newer version of Exchange or not.

I saw report of next phenomenon in other bug.
  Fetched message headers is different from initial fetch just after new mail
  arrival (fetch via IDLE in that case).
  Header order, headers in message header part, are altered by Exchange.
  In that case, added headers by spam filter were seen.
  I thought different UID is assinged in that case, so I didn't request
  detailed IMAP log, but Exchange may replace mail data and use same UID.
If problem with newer Exchange, Exchange's behaviour like above may cause of prorblem.
Comment 36 WADA 2010-05-27 03:45:20 PDT
Following is Web page found by Google search for "ms exchange rfc822.size".
> http://lumbgaps.blogspot.com/2009/04/exchange-2007-imap-rfc822size-and.html
Wrong RFC822.SIZE after Exchange server 2007 seems intensional RFC violation for performance reason. MS Exchange server's option(2007 or later) of "Violate RFC 3502 or not" sounds -ImapSettings -EnableExactRFC822Size:$false/$true. So, it looks that problem occurs or not depends on option setting of MS Exchange server if 2007 or later.
> Build numbers and release dates for Exchange Server
> http://support.microsoft.com/kb/158530

Note You need to log in before you can comment on or make changes to this bug.