231349 - mail.imap.use_status_for_biff causes general IMAP instability

Status: RESOLVED WORKSFORME

(MailNews Core :: Networking: IMAP, defect)

Description

[Bill Gianopoulos [:WG9s]]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.6b) Gecko/20031218 Firebird/0.7+
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.6b) Gecko/20031218 Firebird/0.7+

Having mail.imap.use_status_for_biff (which is the default) appears to cause
IMAP to be much less stable.  Some fof the symptoms are:

1.  It stops moving junk mail to the junk folder.

2.  It stops checking for new mail

3.  I end up with messages being downloded minus the body.  I have the header
and the attachments, but the body is empty.  I think trying to move such
messages to Junk triggers some of the other issues.

4.  Message counts on folders end up being incorrect.

I am seeing these issues in Thunderbird 0.5a.  They usuually occur after it has
been running a couple of hours.  I also have multiple IMAP accounts on multiple
servers.

Setting mail.imap.use_status_for_biff to false appers to correct all these issues.

Reproducible: Sometimes

Steps to Reproduce:
1.
2.
3.

Comment 1

[Bill Gianopoulos [:WG9s]]

Should ahve included that the preference mail.imap.use_status_for_biff was added
by the fix for bug 224381.

Comment 2

[David :Bienvenu]

try updating to a newer 1.7 build - it has a couple fixes, especially for 2 and
4, I believe. I'm not sure what you mean by 1), but it is the case that moving
junk mail to the junk folder in folders other than the inbox will be delayed
until you open the folder, if we use STATUS instead of updating the folder and
downloading the headers and applying the junk filter, when checking for new
mail. That's the trade-off - since we use STATUS, we don't download the headers
or the message bodies, so we obviously can't apply the junk filter.

I'm not sure what you mean by 3). Do you mean, downloaded for offline use, or
just downloaded for display? Using STATUS shouldn't have any affect, except that
the old method of selecting the folder would cause the message bodies to be
downloaded and put in the memory cache when checking for new mail, and using
STATUS means that we don't download the message until you open the folder. Do
you have the menu item view | Display Attachments Inline set to true or false?
Is the message body blank?

Comment 3

[Bill Gianopoulos [:WG9s]]

Actually this is a bug I kind of had wanted to, but was hesitiant to file for a
long time.  I like bugs where you can say if you do this then this happens. 
Less stable if you set this option is kind of way too subjective, but that is
really all I havwe.  Especially since this is kind of something that happens
about once per day.  Sometimes not at all for a couple of days, sometimes 3 or 4
times in the same day.

I am running Mozilla Thunderbird 0.5a (20040103) which, I believe, includes the
relevent IMAP fixes you mentioned.

It is possible that issue #2 and #4 do not occur in this version as I am not
sure when the last time I saw them is.  I know I have not had an ocuurance of
either since changing the indicated preference.

So, the only real issue might be number #3.  I think issue #1 occurs if you try
to move one of these ill downloaded messages.  This appears to make it no longer
able to move any messages even manually, so that would kind of also stop the
automatic moving of junk messages.

I am also not sure there is really anything wrong with the code for using status
inste3ade of select.  It is possible that  this begins as a server issue because
the server does something wrong if multiple connections are doing things in
parrelel on the Inbox.  It could just bet hat using status makes the client more
efficiant so it is more likely to triggwer a serer bug with handling multiple
things at once.

I have 3 IMAP accounts and I really only see this issue on the Lotus/IBM Domino
version 6 server.  The problem appears to be aggravated by having scads of junk
mail in the inbox, so that aftwer a weekend, for example (this is a work
account) on first login it takes longer to move all the junk messages to the
junk folder than the interval between checks for new mail on the server, so that
it tries to load new mail while it is still moving junk mail out of the inbox.

Things that have sweemed to make this less likely to occur that I have tried in
the past are:

1.  Increasing the interval between check for new mail on the server.

2.  Staggering the times between mail checks on the 2 IMAP servers that I
automatilcally check for new mail on to minimize the possibility of having both
servers checking for new mail at the same time.

3.  Moving the Junk mail folder off of the Lotus server.

4.  A recent change that has sped up the moving of mail to junk has also helped.
 Previously no matter how small the messages were it never seeemd to be able to
move more than 1 message per second, so with a lot of junk it was really easy to
NOT be able to process it before the next new mail check.

5.  Reducing the number of cached connection to the lotus server also helped a bit.

To answer your other questions, I do have Display Attachments Inline checked. 
The messages that download without the body do, in fact, have body text. 
Dweleting the IMAP account and readding it (which is the only way I have figured
out to get it to redo the download, results in a corrwect download of the
message with a body being present.  I beleive, but am not sure, that I have also
seen this issue on messages without attachments.

If I selecdt a group of messages includeding thwe screwed up one to move to
anothwer foldeer it starts moving messages then kind of hangs when it gets tot
he bad one.

Selecting that message individually results in a pop-up error saying something
like the server reported nothing to append.

I have set my clint back to caching 5 connections, using status for biff and
have it checking for mail once a minute to try to re-create the issue so I can
get the exact pop-up.

If you weed through the really whiney posts and try to finsd the ones with
useful information in them from this
http://forums.mozillazine.org/viewtopic.php?t=40286 forum thread, you will see
that others appear to be having similar issues with IMAP servers other than
Lotus Domino.

Comment 4

[Aleksey Nogin]

Reporter, is #4 the same as described in bug 231207, or something else?

As for #3, my guess is that it could be a consequence of #1 - junk controls have
a side-effect of downloading the messages (so that the filter can evaluate the
body for "junkinnes"), so essentially with old biff + junk, the messages get
downloaded right away, but with status it does not happen. If this is indeed the
case, then bug 147427 (asking for new messages to be always downloaded right
away) should be the proper one.

> moving
> junk mail to the junk folder in folders other than the inbox will be delayed
> until you open the folder, if we use STATUS instead of updating the folder and
> downloading the headers and applying the junk filter, when checking for new
> mail. That's the trade-off - since we use STATUS, we don't download the
> headers or the message bodies, so we obviously can't apply the junk filter.

IMHO it does not have to be this way - once STATUS reports new messages, Mozilla
could go ahead and d/l those messages. You still get the advantage of STATUS
being fast (since most of the time most of the folders would not have anything
new) without any of the disadvantages.

Comment 5

[Bill Gianopoulos [:WG9s]]

> Reporter, is #4 the same as described in bug 231207, or something else?

Well that bug seems to be about the unread count.  My issue was with the count
of total messages in the mailbox being incorrect.  To the point that after I
deleted all messages inthe mailbox it still had a message count of 2.  It could
very well be the same issue though.

> As for #3, my guess is that it could be a consequence of #1 - junk controls have
> a side-effect of downloading the messages (so that the filter can evaluate the
> body for "junkinnes"), so essentially with old biff + junk, the messages get
> downloaded right away, but with status it does not happen. If this is indeed the
> case, then bug 147427 (asking for new messages to be always downloaded right
> away) should be the proper one.

Well, my issue is not with when the mesages are downloaded.  It is that they get
marked as being downloaded but are somehow downloaded incorrectly.  Once this
happens there is no way to re-download them, and trying to move such an ill
downloaded messaged gets Thunderird in such a state that you must exit and
restart it or if will not move any messages from one folder to another.  This
does not appear to be what bug 147427 is about.

Comment 6

[Bill Gianopoulos [:WG9s]]

I just experienced the issue of a bodyless message being downloaded using the
latest 0120 Thunderbuild (which should have all the latest IMAP fixes).  The
problem occurred during a period in which we were experiencing excessive network
delays between the location where my client is and where the server is.  So,
this might be network error triggered.  I usually experience this issue when
telecommuting, which would also be when my network connetion is slower than when
I am in the office.

The exact message that I get in the pop-up when attempting to copy or move these
messages once the problem occurs is:

The current comand did not succeed.  The mail server responded : Empty message
to APPEND.

Comment 7

[Bill Gianopoulos [:WG9s]]

Well, today I ended up with problems #1, 2 and 3 from my original report
occurring even though I had mail.imap.use_status_for_biff set to false.  So,
either this preference has no effect on these issues, or setting it to true only
makes them less likely to occcur.  Setting the preference to false clearly does
not eliminate the problem.

Comment 8

[David :Bienvenu]

yes, as I said, it really shouldn't have any direct effect. In general, it
should result in more stability, not less, because it reduces the amount of imap
work we do. 

Comment 9

[Bill Gianopoulos [:WG9s]]

I am beginning to think the real issue here is how many unread messages are
onthe server.  If I don't log in over the weekend to check mail, there are over
1000 new messages in my inbox and at leat 90% of them are junk mail (I am on
many mailing lists including [e-mail] and
[e-mail] so I get lots of SPAM).  I think it is dealing with
this large a vlume of new and/or junk emails that triggers these issues.  I
think over the last couple of weeks I was logging in over the weekend and
reading my mail so I was fooled into thinking the problem had gone away cause I
had not seen it in awhile, but I am beginning to realize that I almost always
had this issue on Mondays.

Is there some option I can set in Thunderbird to cause it to create some debug
logging information so that if I can get it to do this agian, I can submit
something that could be usefull in resolving this issue?

Comment 10

[David :Bienvenu]

William, yes, that's much more likely to be the cause of problems.

You can turn on imap protocol logging:

http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html#imap

However, it will generate a huge log in your situation, on Monday morning. You
might be able to remove some of the stuff from the log (like the fetching of the
flags), and zip it up, and e-mail it to me.

Comment 11

[Ronny Adsetts]

Having seen the latest traffic on this bug, I want to add my tuppence...

I can confirm that changing the status of mail.imap.use_status_for_biff to false
makes a difference to bug #221792.

All my email is filtered using procmail on the server. I then read my email via
a courier-imap-ssl server.

In order to see when new email is delivered to a folder, I set that folder's
properties to 'Check this folder for new mail'.

With the biff key set to true, not all folders are checked. With it set to
false, they are all checked.

Comment 12

[Bill Gianopoulos [:WG9s]]

re comment #10.  E-mail on the way with zipped inbox and IMAP log.

Comment 13

[Bill Gianopoulos [:WG9s]]

Some of the recent IMAP fixes appear to have corrected the issues I was seeing here.