534835 - offline folder size constantly increases using Exchange IMAP (Exchange returns wrong rfc822 size) - messages redownload

Status: RESOLVED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[Frank Zimmer]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; de-DE; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.22) Gecko/20090605 Thunderbird/2.0.0.22 Mnenhy/0.7.6.666

Using the Exchange Inbox (via IMAP) as Offline folder configured and Global search enabled the size of the local Offline file increases constantly
On my account the original size of the mailbox is ~100 MB 
The offline file is now at 250 MB and increases after each refresh interval
Another IMAP account on a Cyrus imap server is behaving normal.

Reproducible: Always

Steps to Reproduce:
1.Configure IMAP account on exchange server
2.Select Inbox as Offline available
3.Enable Global search
Actual Results:  
Offline file size increases

Expected Results:  
Offline files size increases

Comment 1

[Ludovic Hirlimann [:Usul]]

Frank global search should only touch the global-messages-db.sqlite file. It should not interfere with the local folders.

If you disable the global indexer (see https://wiki.mozilla.org/Thunderbird:Using_Gloda) - how do your local folder behave , do they still grow ?

While your at playing with prefs you might want to clean up your version string as described in https://bugzilla.mozilla.org/show_bug.cgi?id=533651#c4 .

Could you provide us with an imap log as described at https://wiki.mozilla.org/MailNews:Logging#Windows so we can try to analyse what is going on. We would need Imap:5 and ImapAutoSync:5 log files. You can attach the logs using the add attachment link above.

Comment 2

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #0)
> Gecko/20090605 Thunderbird/2.0.0.22 Mnenhy/0.7.6.666

> Steps to Reproduce:
>(snip)
> 3.Enable Global search

Tb 2 doesn't have "Global Search and Indexer" capability(it's Tb3's feature).
"Global search" by what?
 
> Actual Results:  
> Offline file size increases

If Tb2, and if rebuild-index is executed repeatedly, file size of offline store increases upon each rebuild-index. (bug 487992. fixed bt Tb3.0rc1)
Problem of bug 487992?

Comment 3

[Frank Zimmer]

If i disable the global indexer and compress afterward the Inbox (Exchange)
the file size of the offline file stays stable and does not increase any longer!

I will attach the requested logfiles as soon as possible

Comment 4

[Frank Zimmer]

Correction to my former entry :

The Inbox file size is increasing without the global indexer enabled !
I see that TB is always refreshing around 100 entries of the IMAP folder (of around 3000) 

I will attach the logfiles

Comment 5

[Frank Zimmer]

Attachment: imap:5 logfile

imap:5 logfile
there shall be inside 2 sequences of manual refreshing the Inbox

Comment 6

[Ludovic Hirlimann [:Usul]]

Sorry I forgot could you log imap:5,imapAutoSync:5 both at the same time ?

Comment 7

[Frank Zimmer]

Attachment: imap:5 + ImapAutoSync:5

as requested with ImapAutoSync

Comment 8

[David :Bienvenu]

Sigh, in this case, Exchange is not correcting the rfc822 size when we fetch the whole message. I've seen Exchange do correct it; I don't know if was a newer version of Exchange or not. If you have any control over your Exchange server, you can configure it to report the correct rfc822 size - search the web for "exchange imap rfc822.size wrong"  or look here - http://social.technet.microsoft.com/Forums/en-US/exchangesvrgeneral/thread/21adbe96-e21b-458a-8242-2c3894b9d7cf

http://support.microsoft.com/kb/191504 has info as well, but it seems to be down right now.

I guess we'll need to add some more hacks to TB to deal with Exchange's mendacity.

Comment 9

[Frank Zimmer]

Ok thanks
but i didn't had the problem with TB 2 ;-)

Comment 10

[David :Bienvenu]

yes, tb 2 didn't sanity check the message size in the offline store. Tb 3 does. Not that I'm saying Exchange is insane or anything :-)

Comment 11

[Frank Zimmer]

So in fact for TB3 needs to be a workaround/hack for Exchange to behave like TB2 ?
Thanks Bill !

Will this be someday go into TB3 ?

Frank

Comment 13

[Ludovic Hirlimann [:Usul]]

(In reply to comment #12)
> *** Bug 535005 has been marked as a duplicate of this bug. ***

Bienvenu I dupped but you miaght want to have a peek at the log posted in comment #0

Comment 14

[David :Bienvenu]

Yes, I'm going to try to get a fix for 3.01 or 3.02. Ludo, the log in bug 535005 doesn't really show the problem in action, but it does show them running a beta version of Exchange (this bug's log doesn't show the same Exchange version).

Comment 15

[Frank Zimmer]

Thanks for providing a fix, so i don't need to go back to TB2

Comment 16

[virgo.stellar.stream]

FWIW in TB2 this was resolved by setting the following:
mail.imap.fetch_by_chunks = false
mail.server.default.fetch_by_chunks = false

Comment 17

[Frank Zimmer]

Hi
i tried this settings which had worked in TB2 with TB3 but they did not
help.
As soon as i start to get get Inbox of the Exchange account offline the
offline file size increases constantly ;-)
So this workaround does not help
and

of course i restarted TB after all changes

Comment 18

[David :Bienvenu]

Attachment: wip

This patch makes us remember the literal size on a full message fetch, and adds an xpcshell test to check that we handle a bad rfc822 size, which also involved extending the fake server to return an incorrect rfc822 size if asked.

Comment 19

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #4)
> Correction to my former entry :
> The Inbox file size is increasing without the global indexer enabled !

Removing "global search" from bug summary to avoid confusion, and adding "wrong rfc822 size" for ease of search.

Comment 20

[David :Bienvenu]

Attachment: proposed fix

The actual code change is two lines, but the test changes are a bit more extensive. I added the ability to override the message size on the imap fakeserver, so that it would lie about the message size, and I changed the imap download test to add a case where it lies, and make sure we handle that.

Comment 21

[David :Bienvenu]

this is something I feel we need for 3.01, given the failure mode, and the number of people that have run into it.

Comment 22

[Frank Zimmer]

I feel the same too ;-)
because i would like (better i must unfortunately) use the Exchange server

Comment 23

[Jasper Knockaert]

Related to https://bugzilla.mozilla.org/show_bug.cgi?id=532134 ?

Comment 24

[David :Bienvenu]

quite possibly, yes, since you're using Exchange.

Comment 26

[Mark Banner (:standard8)]

I'm just trying the test on trunk and I'm seeing:

###!!! ASSERTION: offline message doesn't start with From : 'VerifyOfflineMessage(m_offlineHeader, inputStream)', file /Users/moztest/comm/main/src/mailnews/base/util/nsMsgDBFolder.cpp, line 1698

Unfortunately it doesn't apply to branch due to bitrot

Comment 27

[David :Bienvenu]

Do you have a more complete stack trace? That's common code, used by news and imap, and I've seen it with news, with and w/o this patch...

Comment 28

[Mark Banner (:standard8)]

Stack trace of assertion (note this is on Mac):

#6  0x005c773a in NS_DebugBreak_P (aSeverity=1, aStr=0x1c437ac "offline message doesn't start with From ", aExpr=0x1c43778 "VerifyOfflineMessage(m_offlineHeader, inputStream)", aFile=0x1c425d4 "/Users/moztest/comm/main/src/mailnews/base/util/nsMsgDBFolder.cpp", aLine=1698) at /Users/moztest/comm/main/src/mozilla/xpcom/base/nsDebugImpl.cpp:356
#7  0x0182477e in nsMsgDBFolder::EndNewOfflineMessage (this=0x112a400) at /Users/moztest/comm/main/src/mailnews/base/util/nsMsgDBFolder.cpp:1697
#8  0x01a9fe5a in nsImapMailFolder::NormalEndMsgWriteStream (this=0x112a400, uidOfMessage=2, markRead=0, imapUrl=0x56558d0) at /Users/moztest/comm/main/src/mailnews/imap/src/nsImapMailFolder.cpp:4480
#9  0x005d53b9 in NS_InvokeByIndex_P (that=0x112a538, methodIndex=5, paramCount=3, params=0x8250b0) at /Users/moztest/comm/main/src/mozilla/xpcom/reflect/xptcall/src/md/unix/xptcinvoke_unixish_x86.cpp:179
#10 0x005c1bc2 in nsProxyObjectCallInfo::Run (this=0x8250e0) at /Users/moztest/comm/main/src/mozilla/xpcom/proxy/src/nsProxyEvent.cpp:181
#11 0x005b66da in nsThread::ProcessNextEvent (this=0x80f490, mayWait=1, result=0xbfffd0a4) at /Users/moztest/comm/main/src/mozilla/xpcom/threads/nsThread.cpp:527
#12 0x005d53b9 in NS_InvokeByIndex_P (that=0x80f490, methodIndex=8, paramCount=2, params=0xbfffd094) at /Users/moztest/comm/main/src/mozilla/xpcom/reflect/xptcall/src/md/unix/xptcinvoke_unixish_x86.cpp:179
#13 0x009ee175 in XPCWrappedNative::CallMethod (ccx=@0xbfffd2f0, mode=XPCWrappedNative::CALL_METHOD) at /Users/moztest/comm/main/src/mozilla/js/src/xpconnect/src/xpcwrappednative.cpp:2727
#14 0x009fae99 in XPC_WN_CallMethod (cx=0x11f5000, obj=0x55d8e60, argc=1, argv=0xbfffd3d0, vp=0xbfffd3d4) at /Users/moztest/comm/main/src/mozilla/js/src/xpconnect/src/xpcwrappednativejsops.cpp:1756
#15 0x0570d3f3 in ?? ()
#16 0x003caa2b in ExecuteTrace (cx=0x11f5000, f=0x12d68d4, state=@0xbfffd464) at /Users/moztest/comm/main/src/mozilla/js/src/jstracer.cpp:6524
#17 0x003fdef3 in ExecuteTree (cx=0x11f5000, f=0x12d68d4, inlineCallCount=@0xbfffd8b0, innermostNestedGuardp=0xbfffd548) at /Users/moztest/comm/main/src/mozilla/js/src/jstracer.cpp:6620
#18 0x003ff294 in js_MonitorLoopEdge (cx=0x11f5000, inlineCallCount=@0xbfffd8b0, reason=Record_Branch) at /Users/moztest/comm/main/src/mozilla/js/src/jstracer.cpp:7113
#19 0x00305f17 in js_Interpret (cx=0x11f5000) at jsops.cpp:360
#20 0x0032d7cd in js_Execute (cx=0x11f5000, chain=0x55c3a40, script=0x4def790, down=0x0, flags=0, result=0xbfffed08) at jsinterp.cpp:1618
#21 0x0029b379 in JS_ExecuteScript (cx=0x11f5000, obj=0x55c3a40, script=0x4def790, rval=0xbfffed08) at /Users/moztest/comm/main/src/mozilla/js/src/jsapi.cpp:4956
#22 0x00004f36 in ProcessFile (cx=0x11f5000, obj=0x55c3a40, filename=0x0, file=0xa02455e0, forceTTY=1) at /Users/moztest/comm/main/src/mozilla/js/src/xpconnect/shell/xpcshell.cpp:1010
#23 0x000050e7 in Process (cx=0x11f5000, obj=0x55c3a40, filename=0x0, forceTTY=1) at /Users/moztest/comm/main/src/mozilla/js/src/xpconnect/shell/xpcshell.cpp:1049
#24 0x000057b1 in ProcessArgs (cx=0x11f5000, obj=0x55c3a40, argv=0xbfffef2c, argc=15) at /Users/moztest/comm/main/src/mozilla/js/src/xpconnect/shell/xpcshell.cpp:1216
#25 0x000062a4 in main (argc=15, argv=0xbfffef2c, envp=0xbfffef6c) at /Users/moztest/comm/main/src/mozilla/js/src/xpconnect/shell/xpcshell.cpp:1879

Comment 29

[David :Bienvenu]

Attachment: fix test code that broke on mac...

this version should work on the mac - it's against 3.01

Comment 30

[Mark Banner (:standard8)]

Comment on attachment 420981 [details] [diff] [review]
fix test code that broke on mac...

r/sr/a=Standard8

Comment 31

[David :Bienvenu]

fixed on trunk and 3.01

Comment 33

[Blu3]

I'm running 3.01 and thunderbird is still redownloading my entire mailboxes.

Linux/x86_64, IMAP/s server is Dovecot.

Comment 35

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #8)
> Sigh, in this case, Exchange is not correcting the rfc822 size when we fetch
> the whole message. I've seen Exchange do correct it; I don't know if was a
> newer version of Exchange or not.

I saw report of next phenomenon in other bug.
  Fetched message headers is different from initial fetch just after new mail
  arrival (fetch via IDLE in that case).
  Header order, headers in message header part, are altered by Exchange.
  In that case, added headers by spam filter were seen.
  I thought different UID is assinged in that case, so I didn't request
  detailed IMAP log, but Exchange may replace mail data and use same UID.
If problem with newer Exchange, Exchange's behaviour like above may cause of prorblem.

Comment 36

[WADA:World Anti-bad-Duping Agency]

Following is Web page found by Google search for "ms exchange rfc822.size".
> http://lumbgaps.blogspot.com/2009/04/exchange-2007-imap-rfc822size-and.html
Wrong RFC822.SIZE after Exchange server 2007 seems intensional RFC violation for performance reason. MS Exchange server's option(2007 or later) of "Violate RFC 3502 or not" sounds -ImapSettings -EnableExactRFC822Size:$false/$true. So, it looks that problem occurs or not depends on option setting of MS Exchange server if 2007 or later.
> Build numbers and release dates for Exchange Server
> http://support.microsoft.com/kb/158530