offline folder size constantly increases using Exchange IMAP (Exchange returns wrong rfc822 size) - messages redownload

RESOLVED FIXED in Thunderbird 3.1a1

Status

MailNews Core
Networking: IMAP
--
major
RESOLVED FIXED
8 years ago
7 years ago

People

(Reporter: Frank Zimmer, Assigned: Bienvenu)

Tracking

({fixed-seamonkey2.0.3, perf})

unspecified
Thunderbird 3.1a1
x86
Windows XP
fixed-seamonkey2.0.3, perf

Firefox Tracking Flags

(blocking-thunderbird3.0 .1+, thunderbird3.0 .1-fixed)

Details

(Whiteboard: [gs], URL)

Attachments

(3 attachments, 2 obsolete attachments)

(Reporter)

Description

8 years ago
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; de-DE; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.22) Gecko/20090605 Thunderbird/2.0.0.22 Mnenhy/0.7.6.666

Using the Exchange Inbox (via IMAP) as Offline folder configured and Global search enabled the size of the local Offline file increases constantly
On my account the original size of the mailbox is ~100 MB 
The offline file is now at 250 MB and increases after each refresh interval
Another IMAP account on a Cyrus imap server is behaving normal.

Reproducible: Always

Steps to Reproduce:
1.Configure IMAP account on exchange server
2.Select Inbox as Offline available
3.Enable Global search
Actual Results:  
Offline file size increases

Expected Results:  
Offline files size increases
Frank global search should only touch the global-messages-db.sqlite file. It should not interfere with the local folders.

If you disable the global indexer (see https://wiki.mozilla.org/Thunderbird:Using_Gloda) - how do your local folder behave , do they still grow ?

While your at playing with prefs you might want to clean up your version string as described in https://bugzilla.mozilla.org/show_bug.cgi?id=533651#c4 .

Could you provide us with an imap log as described at https://wiki.mozilla.org/MailNews:Logging#Windows so we can try to analyse what is going on. We would need Imap:5 and ImapAutoSync:5 log files. You can attach the logs using the add attachment link above.
See Also: → bug 532134
(In reply to comment #0)
> Gecko/20090605 Thunderbird/2.0.0.22 Mnenhy/0.7.6.666

> Steps to Reproduce:
>(snip)
> 3.Enable Global search

Tb 2 doesn't have "Global Search and Indexer" capability(it's Tb3's feature).
"Global search" by what?
 
> Actual Results:  
> Offline file size increases

If Tb2, and if rebuild-index is executed repeatedly, file size of offline store increases upon each rebuild-index. (bug 487992. fixed bt Tb3.0rc1)
Problem of bug 487992?
(Reporter)

Comment 3

8 years ago
If i disable the global indexer and compress afterward the Inbox (Exchange)
the file size of the offline file stays stable and does not increase any longer!

I will attach the requested logfiles as soon as possible
(Reporter)

Comment 4

8 years ago
Correction to my former entry :

The Inbox file size is increasing without the global indexer enabled !
I see that TB is always refreshing around 100 entries of the IMAP folder (of around 3000) 

I will attach the logfiles
(Reporter)

Comment 5

8 years ago
Created attachment 417665 [details]
imap:5 logfile

imap:5 logfile
there shall be inside 2 sequences of manual refreshing the Inbox
Sorry I forgot could you log imap:5,imapAutoSync:5 both at the same time ?
Component: General → Networking: IMAP
Product: Thunderbird → MailNews Core
QA Contact: general → networking.imap
(Reporter)

Comment 7

8 years ago
Created attachment 417671 [details]
imap:5 + ImapAutoSync:5

as requested with ImapAutoSync
Whiteboard: [has protocol logs]
(Assignee)

Comment 8

8 years ago
Sigh, in this case, Exchange is not correcting the rfc822 size when we fetch the whole message. I've seen Exchange do correct it; I don't know if was a newer version of Exchange or not. If you have any control over your Exchange server, you can configure it to report the correct rfc822 size - search the web for "exchange imap rfc822.size wrong"  or look here - http://social.technet.microsoft.com/Forums/en-US/exchangesvrgeneral/thread/21adbe96-e21b-458a-8242-2c3894b9d7cf

http://support.microsoft.com/kb/191504 has info as well, but it seems to be down right now.

I guess we'll need to add some more hacks to TB to deal with Exchange's mendacity.
Assignee: nobody → bienvenu
Status: NEW → ASSIGNED
(Reporter)

Comment 9

8 years ago
Ok thanks
but i didn't had the problem with TB 2 ;-)
(Assignee)

Comment 10

8 years ago
yes, tb 2 didn't sanity check the message size in the offline store. Tb 3 does. Not that I'm saying Exchange is insane or anything :-)
(Reporter)

Comment 11

8 years ago
So in fact for TB3 needs to be a workaround/hack for Exchange to behave like TB2 ?
Thanks Bill !

Will this be someday go into TB3 ?

Frank
Duplicate of this bug: 535005
(In reply to comment #12)
> *** Bug 535005 has been marked as a duplicate of this bug. ***

Bienvenu I dupped but you miaght want to have a peek at the log posted in comment #0
(Assignee)

Comment 14

8 years ago
Yes, I'm going to try to get a fix for 3.01 or 3.02. Ludo, the log in bug 535005 doesn't really show the problem in action, but it does show them running a beta version of Exchange (this bug's log doesn't show the same Exchange version).
(Reporter)

Comment 15

8 years ago
Thanks for providing a fix, so i don't need to go back to TB2
FWIW in TB2 this was resolved by setting the following:
mail.imap.fetch_by_chunks = false
mail.server.default.fetch_by_chunks = false
(Reporter)

Comment 17

8 years ago
Hi
i tried this settings which had worked in TB2 with TB3 but they did not
help.
As soon as i start to get get Inbox of the Exchange account offline the
offline file size increases constantly ;-)
So this workaround does not help
and

of course i restarted TB after all changes
(Assignee)

Comment 18

8 years ago
Created attachment 419629 [details] [diff] [review]
wip

This patch makes us remember the literal size on a full message fetch, and adds an xpcshell test to check that we handle a bad rfc822 size, which also involved extending the fake server to return an incorrect rfc822 size if asked.
(In reply to comment #4)
> Correction to my former entry :
> The Inbox file size is increasing without the global indexer enabled !

Removing "global search" from bug summary to avoid confusion, and adding "wrong rfc822 size" for ease of search.
Summary: offline folder size constantly increases using Exchange IMAP with global search enabled → offline folder size constantly increases using Exchange IMAP (Exchange returns wrong rfc822 size)
(Assignee)

Comment 20

8 years ago
Created attachment 419940 [details] [diff] [review]
proposed fix

The actual code change is two lines, but the test changes are a bit more extensive. I added the ability to override the message size on the imap fakeserver, so that it would lie about the message size, and I changed the imap download test to add a case where it lies, and make sure we handle that.
Attachment #419629 - Attachment is obsolete: true
Attachment #419940 - Flags: superreview?(bugzilla)
Attachment #419940 - Flags: review?(bugzilla)
(Assignee)

Comment 21

8 years ago
this is something I feel we need for 3.01, given the failure mode, and the number of people that have run into it.
blocking-thunderbird3.0: --- → .1+
Whiteboard: [has protocol logs] → [has patch for review]
(Reporter)

Comment 22

8 years ago
I feel the same too ;-)
because i would like (better i must unfortunately) use the Exchange server

Comment 23

8 years ago
Related to https://bugzilla.mozilla.org/show_bug.cgi?id=532134 ?
(Assignee)

Comment 24

8 years ago
quite possibly, yes, since you're using Exchange.
(Assignee)

Updated

8 years ago
Duplicate of this bug: 532134
I'm just trying the test on trunk and I'm seeing:

###!!! ASSERTION: offline message doesn't start with From : 'VerifyOfflineMessage(m_offlineHeader, inputStream)', file /Users/moztest/comm/main/src/mailnews/base/util/nsMsgDBFolder.cpp, line 1698

Unfortunately it doesn't apply to branch due to bitrot
(Assignee)

Comment 27

8 years ago
Do you have a more complete stack trace? That's common code, used by news and imap, and I've seen it with news, with and w/o this patch...
Stack trace of assertion (note this is on Mac):

#6  0x005c773a in NS_DebugBreak_P (aSeverity=1, aStr=0x1c437ac "offline message doesn't start with From ", aExpr=0x1c43778 "VerifyOfflineMessage(m_offlineHeader, inputStream)", aFile=0x1c425d4 "/Users/moztest/comm/main/src/mailnews/base/util/nsMsgDBFolder.cpp", aLine=1698) at /Users/moztest/comm/main/src/mozilla/xpcom/base/nsDebugImpl.cpp:356
#7  0x0182477e in nsMsgDBFolder::EndNewOfflineMessage (this=0x112a400) at /Users/moztest/comm/main/src/mailnews/base/util/nsMsgDBFolder.cpp:1697
#8  0x01a9fe5a in nsImapMailFolder::NormalEndMsgWriteStream (this=0x112a400, uidOfMessage=2, markRead=0, imapUrl=0x56558d0) at /Users/moztest/comm/main/src/mailnews/imap/src/nsImapMailFolder.cpp:4480
#9  0x005d53b9 in NS_InvokeByIndex_P (that=0x112a538, methodIndex=5, paramCount=3, params=0x8250b0) at /Users/moztest/comm/main/src/mozilla/xpcom/reflect/xptcall/src/md/unix/xptcinvoke_unixish_x86.cpp:179
#10 0x005c1bc2 in nsProxyObjectCallInfo::Run (this=0x8250e0) at /Users/moztest/comm/main/src/mozilla/xpcom/proxy/src/nsProxyEvent.cpp:181
#11 0x005b66da in nsThread::ProcessNextEvent (this=0x80f490, mayWait=1, result=0xbfffd0a4) at /Users/moztest/comm/main/src/mozilla/xpcom/threads/nsThread.cpp:527
#12 0x005d53b9 in NS_InvokeByIndex_P (that=0x80f490, methodIndex=8, paramCount=2, params=0xbfffd094) at /Users/moztest/comm/main/src/mozilla/xpcom/reflect/xptcall/src/md/unix/xptcinvoke_unixish_x86.cpp:179
#13 0x009ee175 in XPCWrappedNative::CallMethod (ccx=@0xbfffd2f0, mode=XPCWrappedNative::CALL_METHOD) at /Users/moztest/comm/main/src/mozilla/js/src/xpconnect/src/xpcwrappednative.cpp:2727
#14 0x009fae99 in XPC_WN_CallMethod (cx=0x11f5000, obj=0x55d8e60, argc=1, argv=0xbfffd3d0, vp=0xbfffd3d4) at /Users/moztest/comm/main/src/mozilla/js/src/xpconnect/src/xpcwrappednativejsops.cpp:1756
#15 0x0570d3f3 in ?? ()
#16 0x003caa2b in ExecuteTrace (cx=0x11f5000, f=0x12d68d4, state=@0xbfffd464) at /Users/moztest/comm/main/src/mozilla/js/src/jstracer.cpp:6524
#17 0x003fdef3 in ExecuteTree (cx=0x11f5000, f=0x12d68d4, inlineCallCount=@0xbfffd8b0, innermostNestedGuardp=0xbfffd548) at /Users/moztest/comm/main/src/mozilla/js/src/jstracer.cpp:6620
#18 0x003ff294 in js_MonitorLoopEdge (cx=0x11f5000, inlineCallCount=@0xbfffd8b0, reason=Record_Branch) at /Users/moztest/comm/main/src/mozilla/js/src/jstracer.cpp:7113
#19 0x00305f17 in js_Interpret (cx=0x11f5000) at jsops.cpp:360
#20 0x0032d7cd in js_Execute (cx=0x11f5000, chain=0x55c3a40, script=0x4def790, down=0x0, flags=0, result=0xbfffed08) at jsinterp.cpp:1618
#21 0x0029b379 in JS_ExecuteScript (cx=0x11f5000, obj=0x55c3a40, script=0x4def790, rval=0xbfffed08) at /Users/moztest/comm/main/src/mozilla/js/src/jsapi.cpp:4956
#22 0x00004f36 in ProcessFile (cx=0x11f5000, obj=0x55c3a40, filename=0x0, file=0xa02455e0, forceTTY=1) at /Users/moztest/comm/main/src/mozilla/js/src/xpconnect/shell/xpcshell.cpp:1010
#23 0x000050e7 in Process (cx=0x11f5000, obj=0x55c3a40, filename=0x0, forceTTY=1) at /Users/moztest/comm/main/src/mozilla/js/src/xpconnect/shell/xpcshell.cpp:1049
#24 0x000057b1 in ProcessArgs (cx=0x11f5000, obj=0x55c3a40, argv=0xbfffef2c, argc=15) at /Users/moztest/comm/main/src/mozilla/js/src/xpconnect/shell/xpcshell.cpp:1216
#25 0x000062a4 in main (argc=15, argv=0xbfffef2c, envp=0xbfffef6c) at /Users/moztest/comm/main/src/mozilla/js/src/xpconnect/shell/xpcshell.cpp:1879
(Assignee)

Comment 29

8 years ago
Created attachment 420981 [details] [diff] [review]
fix test code that broke on mac...

this version should work on the mac - it's against 3.01
Attachment #419940 - Attachment is obsolete: true
Attachment #420981 - Flags: superreview?(bugzilla)
Attachment #420981 - Flags: review?
Attachment #419940 - Flags: superreview?(bugzilla)
Attachment #419940 - Flags: review?(bugzilla)
Comment on attachment 420981 [details] [diff] [review]
fix test code that broke on mac...

r/sr/a=Standard8
Attachment #420981 - Flags: superreview?(bugzilla)
Attachment #420981 - Flags: superreview+
Attachment #420981 - Flags: review?
Attachment #420981 - Flags: review+
Attachment #420981 - Flags: approval-thunderbird3.0.1+
(Assignee)

Comment 31

8 years ago
fixed on trunk and 3.01
Status: ASSIGNED → RESOLVED
Last Resolved: 8 years ago
status-thunderbird3.0: --- → .1-fixed
Resolution: --- → FIXED
Whiteboard: [has patch for review]
Target Milestone: --- → Thunderbird 3.1a1
(Assignee)

Updated

8 years ago
Duplicate of this bug: 534297

Comment 33

8 years ago
I'm running 3.01 and thunderbird is still redownloading my entire mailboxes.

Linux/x86_64, IMAP/s server is Dovecot.
Keywords: perf
Summary: offline folder size constantly increases using Exchange IMAP (Exchange returns wrong rfc822 size) → offline folder size constantly increases using Exchange IMAP (Exchange returns wrong rfc822 size) - messages redownload
Whiteboard: [gs]
Duplicate of this bug: 542816
Depends on: 544396

Updated

8 years ago
Keywords: fixed-seamonkey2.0.3
(In reply to comment #8)
> Sigh, in this case, Exchange is not correcting the rfc822 size when we fetch
> the whole message. I've seen Exchange do correct it; I don't know if was a
> newer version of Exchange or not.

I saw report of next phenomenon in other bug.
  Fetched message headers is different from initial fetch just after new mail
  arrival (fetch via IDLE in that case).
  Header order, headers in message header part, are altered by Exchange.
  In that case, added headers by spam filter were seen.
  I thought different UID is assinged in that case, so I didn't request
  detailed IMAP log, but Exchange may replace mail data and use same UID.
If problem with newer Exchange, Exchange's behaviour like above may cause of prorblem.
Following is Web page found by Google search for "ms exchange rfc822.size".
> http://lumbgaps.blogspot.com/2009/04/exchange-2007-imap-rfc822size-and.html
Wrong RFC822.SIZE after Exchange server 2007 seems intensional RFC violation for performance reason. MS Exchange server's option(2007 or later) of "Violate RFC 3502 or not" sounds -ImapSettings -EnableExactRFC822Size:$false/$true. So, it looks that problem occurs or not depends on option setting of MS Exchange server if 2007 or later.
> Build numbers and release dates for Exchange Server
> http://support.microsoft.com/kb/158530
You need to log in before you can comment on or make changes to this bug.