Closed Bug 103401 Opened 19 years ago Closed 17 years ago
mozilla cannot open imap folders that have too many UID gaps
From Bugzilla Helper: User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.4) Gecko/20010913 BuildID: 2001091311 Once an IMAP mailboxe becomes "big enough", you can continue to open it in the Mozilla that you regularly use but you *cannot* open it in a new Mozilla on a different machine. Mozilla will spin its progress bar forever instead of opening the folder. Quitting and restarting doesn't help -- once a folder is "too big", it can only be read with the particular Mozilla binary that made it that big. The user can continue to access folders that are "small enough". Reproducible: Always Steps to Reproduce: 1. quit Mozilla 2. add 3000 messages to an IMAP mailbox 3. delete every other message, leaving 1500 4. verify that the messages have non-sequential UIDs 5. verify that you don't have a cache file (*.msf) for the folder 6. start Mozilla and launch the Mail&News task 7. try to open the folder in the Mailreader. I have seen the problem even down as low as 400 messages, but 1500 messages is a more typical size for my inbox, and I'm more confident in reproducibility with large mailboxes. (Tip: create message "A", then message "B", and then double the pair, redouble, and continue doubling until you have enough messages. Then delete all "B".) I verified UIDs using the unix utility "imtest": . fetch 390:400 (uid) * 390 FETCH (UID 8106) * 391 FETCH (UID 8108) * 392 FETCH (UID 8110) * 393 FETCH (UID 8112) * 394 FETCH (UID 8114) * 395 FETCH (UID 8116) * 396 FETCH (UID 8118) * 397 FETCH (UID 8120) * 398 FETCH (UID 8122) * 399 FETCH (UID 8124) * 400 FETCH (UID 8126) . OK Completed Server logs probably would have worked as well. The point of deleting the cache file is to force mozilla to re-generate it. That's the bug: if the folder is too big, mozilla fails to generate the cache file, and the user can't access the folder. Actual Results: Mozilla will not be able to complete its scan of the test folder. Sometimes it will process a few hundred headers before it fails (the status line may read "47 of 748"), other times it fails even before processing a single message. It will never be possible to view the body of one of any of the test messages, even if Mozilla managed to read some headers. You can still continue to view other, smaller folders. Expected Results: Mozilla should sucessfully read the message headers from the server, should display them in the message list, and should allow users to read every message. It may be worth your time to read about UIDs in the IMAP RFC (RFC2060) before you read any farther. The most critical lines are: Unique identifiers are assigned in a strictly ascending fashion in the mailbox; as each message is added to the mailbox it is assigned a higher UID than the message(s) which were added previously. Unlike message sequence numbers, unique identifiers are not necessarily contiguous. Unique identifiers also persist across sessions. My suspicion is that the bug is caused by Mozilla being unable to cache server replies that have too many gaps in the UID sequence. If a user primarily handles mail on one machine, they will never add more than a few dozen messages to a mailbox cache at a time, and UID gaps in the new mail will also be rare. But note that every time they delete a message from the mailbox they leave a UID gap. When the user tries to run Mozilla on a second machine, the new Mozila will try to fetch the user's entire folder into the new machine's cache in one gulp. Mozilla often chokes on this mouthful, particularly if the folder it's trying to swallow was a heavily-used one. The critical step in reproducing this bug is to use a folder that has gaps in the UIDs. If you merely throw a thousand messages into a new folder, the server will number the messages sequentially, and Mozilla will cache the server reply with no trouble. Heavily-used inboxes will have plenty of UID gaps; one for each time the user deleted their incoming mail. People who filter their mail also generate UID gaps at a large rate. Sysadmins tend to fulfill both criteria, and are likely to evaluate Mozilla poorly if they find it can't even load their INBOX. I'll rank "Mozilla can't generate a fresh cache for large INBOXes" as Major severity, and let you downgrade it if you want.
I've just tested and confirmed that Build 2001091303 on Windows 2000 shares the bug. My test folder has 600 non-contiguous messages, and Mozilla won't open the folder. I'm not sure what exact number of messages triggers the bug. I haven't seen 100 non-contiguous messages trigger it -- 100 messages work fine. I think I have seen 200 non-contiguous go both ways -- sometimes they're visible, sometimes they lock up the folder. 600 messages seems to be pretty safely in the buggy regime, and 1500 should definitely trigger it.
I've confirmed that Macintosh build 2001091311 also exhibits the bug. I've now tested on Macos9, on Windows2000, on Linux, and on Irix. I'm going to mark the bug as "all platforms, all OS's." Mozilla is unable to build cache files for folders with too many UID gaps. Speaking from a sysadmin's perspective, this bug places restrictions on how widely I can recommend Mozilla to our users.
OS: Linux → All
Hardware: PC → All
Summary: folder hang if new imap messages have too many UID gaps → mozilla cannot open imap folders that have too many UID gaps
Bleah, I don't understand how bugzilla works. This bug doesn't show up in default queries, apparently because it's marked as unconfirmed? Is it marked as unconfirmed because I made the foolish mistake of appending comments to the bug before a developer had the chance to append a comment? I can't tell; I'll try reassigning the but to the MailNews-Networking-IMAP contact, to see if if that changes the bugs classification and allows it to appear in default queries again.
Hm. It's possible that bug 48550 is the same as this bug.
Status: UNCONFIRMED → NEW
Ever confirmed: true
we might be generating a command that's too long for your imap server. Generating a protocol log would tell us: http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html#imap
Actually, you generate a line that's too long for your own protocol logger. The logfile's lines are truncated at about 500 bytes in length. Note that the "best practices" rfc for IMAP suggests not sending lines longer than 1024 bytes in length. I haven't yet figured out how to get Mozilla running under a debugger, so I don't know exactly where the problem is. Poking around by hand, the "UID FETCH" code near line 2688 of nsImapProtocol seems suspicious. It readies a dynamically-sized buffer to hold the uid fetch command, which could be trouble, because the server might not be using a dynamically-sized buffer to receive the command. http://lxr.mozilla.org/seamonkey/source/mailnews/imap/src/nsImapProtocol.cpp#2688 I believe the imap protocol has a mechanism where you send long lines by first telling the server: "attention! long line coming up, length XXXX bytes", and then sending the line. One way to solve this bug would be to implement this mechanism. I'll attach a logfile.
Line 5113 is the relevant line. The log file has only got around to requesting 800 messages before the logger truncated the line. I've definitely seen this bug on mailboxes with as few as 250 messages, although those mailboxes had 5- or 6-digit UID's instead of the 4-digit UIDs presented in this logfile. (The server only communicates over the SSL port, so I can't send a network trace either.)
Oops, I was too hasty when announcing the "800 messages" number. Actually, only about 90 UIDs are visible on line 5113 in the logfile. But note that in order to request all 1500 UIDs in my test mailbox, Mozilla would need to send a 7500-byte line. That's a lot of data. I'll also point out that I'm still having problems with this bug in real-world situations. I walked over to a computer that I hadn't used in a few months, and couldn't read mail on it, because Mozilla was unable to pull the UIDs for the new messages in my Inbox. My inbox was only 250 messages at the time! That's _small_.
before the speculation gets too out of hand here, bear in mind that we only ask for 200 headers at a time for this very reason, and that this works on most servers. However, if you do have the scenario you describe, 200 headers will overflow the 1024 byte buffer size that it seems your server has, so we need to rewrite the code to take into account the length of the generated strings. I don't know of any imap command to alert the server that a long command is coming.
I checked rfc2060 again, and it appears that I was confused. A very long string can be entered as a tagged-length "literal", but the FETCH command doesn't use strings. It uses "set"s and "sequence_num"s. Sorry for the confusion.
Assignee: mscott → bienvenu
I'll add the comment that about six months ago our site upgraded to a newer version of Cyrus imapd, and we haven't had the problem since then. I still think mozilla shouldn't send out overly-long lines, but the upgraded server no longer chokes itself on the commands that Mozilla sends, so we're fine at our end.
accepting; I'm going to try to fix this for correctness' sake.
Status: NEW → ASSIGNED
this patch has some cleanup, but the main fix is to pass in the key array to AllocateImapUidString, so that we can tell when there are uid gaps in the mailbox, and still ask for ranges. E.g., if the mailbox contains UID's 1,3,5, and we're trying to create a uid string to fetch those same UIDs, we'll generate 1:5.
whoops, that last diff was the wrong diff.
Status: ASSIGNED → RESOLVED
Closed: 17 years ago
Resolution: --- → FIXED
*** Bug 186658 has been marked as a duplicate of this bug. ***
This commit have added a warning on brad TBox: mailnews/imap/src/nsImapProtocol.cpp:3696 Unused variable `const PRInt32 kHdrsToFetchAtOnce' Indeed, all usages of the variable were removed, but the variable is still there.
You need to log in before you can comment on or make changes to this bug.