103401 - mozilla cannot open imap folders that have too many UID gaps

Status: RESOLVED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[jmorzins]

From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.4) Gecko/20010913
BuildID:    2001091311


Once an IMAP mailboxe becomes "big enough", you can continue to open it in the
Mozilla that you regularly use but you *cannot* open it in a new Mozilla on a
different machine.  Mozilla will spin its progress bar forever instead of
opening the folder.  Quitting and restarting doesn't help -- once a folder is
"too big", it can only be read with the particular Mozilla binary that made it
that big.

The user can continue to access folders that are "small enough".


Reproducible: Always
Steps to Reproduce:

1. quit Mozilla
2. add 3000 messages to an IMAP mailbox
3. delete every other message, leaving 1500
4. verify that the messages have non-sequential UIDs
5. verify that you don't have a cache file (*.msf) for the folder
6. start Mozilla and launch the Mail&News task
7. try to open the folder in the Mailreader.


I have seen the problem even down as low as 400 messages, but 1500 messages is a
more typical size for my inbox, and I'm more confident in reproducibility with
large mailboxes.  (Tip: create message "A", then message "B", and then double
the pair, redouble, and continue doubling until you have enough messages.  Then
delete all "B".)

I verified UIDs using the unix utility "imtest":
. fetch 390:400 (uid)
* 390 FETCH (UID 8106)
* 391 FETCH (UID 8108)
* 392 FETCH (UID 8110)
* 393 FETCH (UID 8112)
* 394 FETCH (UID 8114)
* 395 FETCH (UID 8116)
* 396 FETCH (UID 8118)
* 397 FETCH (UID 8120)
* 398 FETCH (UID 8122)
* 399 FETCH (UID 8124)
* 400 FETCH (UID 8126)
. OK Completed
Server logs probably would have worked as well.

The point of deleting the cache file is to force mozilla to re-generate it. 
That's the bug: if the folder is too big, mozilla fails to generate the cache
file, and the user can't access the folder.


Actual Results:  
Mozilla will not be able to complete its scan of the test folder.  Sometimes it
will process a few hundred headers before it fails (the status line may read "47
of 748"), other times it fails even before processing a single message.  It will
never be possible to view the body of one of any of the test messages, even if
Mozilla managed to read some headers.

You can still continue to view other, smaller folders.


Expected Results:  
Mozilla should sucessfully read the message headers from the server, should
display them in the message list, and should allow users to read every message.



It may be worth your time to read about UIDs in the IMAP RFC (RFC2060) before
you read any farther.  The most critical lines are:

 Unique identifiers are assigned in a strictly ascending fashion
 in the mailbox; as each message is added to the mailbox it is
 assigned a higher UID than the message(s) which were added
 previously.

 Unlike message sequence numbers, unique identifiers are not
 necessarily contiguous.  Unique identifiers also persist across
 sessions.




My suspicion is that the bug is caused by Mozilla being unable to cache server
replies that have too many gaps in the UID sequence.  If a user primarily
handles mail on one machine, they will never add more than a few dozen messages
to a mailbox cache at a time, and UID gaps in the new mail will also be rare. 
But note that every time they delete a message from the mailbox they leave a UID
gap.  When the user tries to run Mozilla on a second machine, the new Mozila
will try to fetch the user's entire folder into the new machine's cache in one
gulp.  Mozilla often chokes on this mouthful, particularly if the folder it's
trying to swallow was a heavily-used one.


The critical step in reproducing this bug is to use a folder that has gaps in
the UIDs.  If you merely throw a thousand messages into a new folder, the server
will number the messages sequentially, and Mozilla will cache the server reply
with no trouble.

Heavily-used inboxes will have plenty of UID gaps; one for each time the user
deleted their incoming mail.  People who filter their mail also generate UID
gaps at a large rate.  Sysadmins tend to fulfill both criteria, and are likely
to evaluate Mozilla poorly if they find it can't even load their INBOX.

I'll rank "Mozilla can't generate a fresh cache for large INBOXes" as Major
severity, and let you downgrade it if you want.

Comment 1

[jmorzins]

I've just tested and confirmed that Build 2001091303 on Windows 2000 shares the
bug.  My test folder has 600 non-contiguous messages, and Mozilla won't open the
folder.

I'm not sure what exact number of messages triggers the bug.  I haven't seen 100
non-contiguous messages trigger it -- 100 messages work fine.  I think I have
seen 200 non-contiguous go both ways -- sometimes they're visible, sometimes
they lock up the folder.  600 messages seems to be pretty safely in the buggy
regime, and 1500 should definitely trigger it.

Comment 2

[jmorzins]

I've confirmed that Macintosh build 2001091311 also exhibits the bug.  I've now
tested on Macos9, on Windows2000, on Linux, and on Irix.  I'm going to mark the
bug as "all platforms, all OS's."

Mozilla is unable to build cache files for folders with too many UID gaps.

Speaking from a sysadmin's perspective, this bug places restrictions on how
widely I can recommend Mozilla to our users.

Comment 3

[jmorzins]

Bleah, I don't understand how bugzilla works.  This bug doesn't show up in
default queries, apparently because it's marked as unconfirmed?  Is it marked as
unconfirmed because I made the foolish mistake of appending comments to the bug
before a developer had the chance to append a comment?

I can't tell; I'll try reassigning the but to the MailNews-Networking-IMAP
contact, to see if if that changes the bugs classification and allows it to
appear in default queries again.

Comment 4

[jmorzins]

Hm.  It's possible that bug 48550 is the same as this bug.

Comment 5

[David :Bienvenu]

we might be generating a command that's too long for your imap server.
Generating a protocol log would tell us:

http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html#imap

Comment 6

[jmorzins]

Actually, you generate a line that's too long for your own protocol logger.  The
logfile's lines are truncated at about 500 bytes in length.  Note that the "best
practices" rfc for IMAP suggests not sending lines longer than 1024 bytes in length.

I haven't yet figured out how to get Mozilla running under a debugger, so I
don't know exactly where the problem is.  Poking around by hand, the "UID FETCH"
code near line 2688 of nsImapProtocol seems suspicious.  It readies a
dynamically-sized buffer to hold the uid fetch command, which could be trouble,
because the server might not be using a dynamically-sized buffer to receive the
command.

http://lxr.mozilla.org/seamonkey/source/mailnews/imap/src/nsImapProtocol.cpp#2688

I believe the imap protocol has a mechanism where you send long lines by first
telling the server: "attention! long line coming up, length XXXX bytes", and
then sending the line.  One way to solve this bug would be to implement this
mechanism.

I'll attach a logfile.

Comment 7

[jmorzins]

Attachment: imap logfile

Line 5113 is the relevant line.  The log file has only got around to requesting
800 messages before the logger truncated the line.  I've definitely seen this
bug on mailboxes with as few as 250 messages, although those mailboxes had 5-
or 6-digit UID's instead of the 4-digit UIDs presented in this logfile.  (The
server only communicates over the SSL port, so I can't send a network trace
either.)

Comment 8

[jmorzins]

Oops, I was too hasty when announcing the "800 messages" number.  Actually, only
about 90 UIDs are visible on line 5113 in the logfile.  But note that in order
to request all 1500 UIDs in my test mailbox, Mozilla would need to send a
7500-byte line.  That's a lot of data.

I'll also point out that I'm still having problems with this bug in real-world
situations.  I walked over to a computer that I hadn't used in a few months, and
couldn't read mail on it, because Mozilla was unable to pull the UIDs for the
new messages in my Inbox.  My inbox was only 250 messages at the time!  That's
_small_.

Comment 9

[David :Bienvenu]

before the speculation gets too out of hand here, bear in mind that we only ask
for 200 headers at a time for this very reason, and that this works on most
servers. However, if you do have the scenario you describe, 200 headers will
overflow the 1024 byte buffer size that it seems your server has, so we need to
rewrite the code to take into account the length of the generated strings.

I don't know of any imap command to alert the server that a long command is coming.

Comment 10

[jmorzins]

I checked rfc2060 again, and it appears that I was confused.  A very long string
can be entered as a tagged-length "literal", but the FETCH command doesn't use
strings.  It uses "set"s and "sequence_num"s.

Sorry for the confusion.

Comment 11

[David :Bienvenu]

taking

Comment 12

[jmorzins]

I'll add the comment that about six months ago our site upgraded to a newer
version of Cyrus imapd, and we haven't had the problem since then.  I still
think mozilla shouldn't send out overly-long lines, but the upgraded server no
longer chokes itself on the commands that Mozilla sends, so we're fine at our end.

Comment 13

[David :Bienvenu]

accepting; I'm going to try to fix this for correctness' sake.

Comment 14

[David :Bienvenu]

Attachment: proposed fix

this patch has some cleanup, but the main fix is to pass in the key array to
AllocateImapUidString, so that we can tell when there are uid gaps in the
mailbox, and still ask for ranges. E.g., if the mailbox contains UID's 1,3,5,
and we're trying to create a uid string to fetch those same UIDs, we'll
generate 1:5.

Comment 15

[David :Bienvenu]

Attachment: correct fix

whoops, that last diff was the wrong diff.

Comment 16

[David :Bienvenu]

fixed, r/sr=mscott

Comment 17

[David :Bienvenu]

*** Bug 186658 has been marked as a duplicate of this bug. ***

Comment 18

[Aleksey Nogin]

This commit have added a warning on brad TBox:

mailnews/imap/src/nsImapProtocol.cpp:3696
Unused variable `const PRInt32 kHdrsToFetchAtOnce'

Indeed, all usages of the variable were removed, but the variable is still there.