Closed Bug 244303 Opened 21 years ago Closed 16 years ago

IMAP synchronise cannot recover from failure

Categories

(MailNews Core :: Networking: IMAP, defect)

Product:
MailNews Core
Component:
Networking: IMAP
Platform:
x86
Linux
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: minfrin, Assigned: Bienvenu)

Details

(Keywords: perf)

User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7b) Gecko/20040514 Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7b) Gecko/20040514 If the "make my messages available when I am working offline" option is selected, and the following is true: - There is a large mailbox to synchronise (38MB) - The is a slow link (dialup speed) Mozilla cannot recover gracefully from transfer failures. During a transfer this big, at some point, a download failure occurs (probably the fault of the IMAP server, courier-imapd). Instead of resuming the download of messages where it left off, Mozilla chooses to recommence the mailbox synchronise _from scratch_. With each failure, Mozilla chooses to try again with a new temporary file, keeping the old ones intact and ignored. After three days of continuous downloads, Mozilla has not managed to synchronise the mailbox, despite having downloaded a few hundred MB of mail. The original mailbox on the IMAP server is 38MB in size. -rw------- 1 minfrin fma 0 Feb 8 18:38 filterlog.html -rw-r--r-- 1 minfrin fma 251623997 May 21 22:26 INBOX -rw-r--r-- 1 minfrin fma 1056825 May 21 22:23 INBOX.msf drwxrwxr-x 3 minfrin fma 4096 May 20 21:02 INBOX.sbd -rw-r--r-- 1 minfrin fma 415 May 21 14:22 msgFilterRules.dat -rw-r--r-- 1 minfrin fma 29844648 May 21 22:23 nstmp -rw-r--r-- 1 minfrin fma 0 May 20 15:27 nstmp-1 -rw-r--r-- 1 minfrin fma 21633924 May 21 22:23 nstmp-10 -rw-r--r-- 1 minfrin fma 10889237 May 21 22:23 nstmp-11 -rw-r--r-- 1 minfrin fma 10788760 May 21 22:23 nstmp-12 -rw-r--r-- 1 minfrin fma 10343305 May 21 22:10 nstmp-13 -rw-r--r-- 1 minfrin fma 14436334 May 21 22:26 nstmp-14 -rw-r--r-- 1 minfrin fma 11341895 May 21 22:23 nstmp-15 -rw-r--r-- 1 minfrin fma 17126552 May 21 22:23 nstmp-16 -rw-r--r-- 1 minfrin fma 11960680 May 21 22:23 nstmp-17 -rw-r--r-- 1 minfrin fma 10890187 May 21 22:23 nstmp-18 -rw-r--r-- 1 minfrin fma 11112882 May 21 22:23 nstmp-19 -rw-r--r-- 1 minfrin fma 9543192 May 21 22:23 nstmp-2 -rw-r--r-- 1 minfrin fma 9852573 May 21 22:23 nstmp-20 -rw-r--r-- 1 minfrin fma 9679525 May 21 22:22 nstmp-21 -rw-r--r-- 1 minfrin fma 164864 May 21 22:23 nstmp-22 -rw-r--r-- 1 minfrin fma 7234303 May 21 22:23 nstmp-23 -rw-r--r-- 1 minfrin fma 1653083 May 21 22:23 nstmp-24 -rw-r--r-- 1 minfrin fma 294597 May 21 22:23 nstmp-25 -rw-r--r-- 1 minfrin fma 307023 May 21 22:22 nstmp-26 -rw-r--r-- 1 minfrin fma 174410 May 21 22:23 nstmp-27 -rw-r--r-- 1 minfrin fma 145245 May 21 22:23 nstmp-28 -rw-r--r-- 1 minfrin fma 134754 May 21 22:22 nstmp-29 -rw-r--r-- 1 minfrin fma 29315798 May 21 22:23 nstmp-3 -rw-r--r-- 1 minfrin fma 180981 May 21 22:23 nstmp-30 -rw-r--r-- 1 minfrin fma 135667 May 21 22:23 nstmp-31 -rw-r--r-- 1 minfrin fma 23988716 May 21 22:23 nstmp-4 -rw-r--r-- 1 minfrin fma 23961567 May 21 22:23 nstmp-5 -rw-r--r-- 1 minfrin fma 21869371 May 21 22:23 nstmp-6 -rw-r--r-- 1 minfrin fma 21936090 May 21 22:23 nstmp-7 -rw-r--r-- 1 minfrin fma 9806071 May 21 22:23 nstmp-8 -rw-r--r-- 1 minfrin fma 9009973 May 21 22:23 nstmp-9 This bug renders this feature useless on large mailboxes. Reproducible: Always Steps to Reproduce: xxx
Have you shutdown and restarted at any point after a failure? I don't really understand the temp files since we download directly into the INBOX file, not a temp file, unless somehow we think the INBOX file is locked. That lock would be cleared by shutting down and restarting. What kind of failure are you referring to? Do you get an error of some sort? Or does the process just stop, and you try again?
Subsequent to filing this bug, I did the following: - Shut down mozilla - Changed to the ImapMail directory - Deleted all nstmp* files - Deleted INBOX (left behind INBOX.msf) Then, I cranked up Mozilla again. I clicked on get messages, and the following happened: - Mozilla picked up the message headers from INBOX.msf/sbd - Mozilla recreated the file INBOX, and started downloading all messages from the IMAP server. - After approx 6MB or so had been transferred, the message "the remote server has disconnected" (or words to that effect) was issued, and I was asked to click ok. - After clicking ok, my packet sniffer showed Mozilla starting to download mail again. - After another while, "the remote server has disconnected". Clicked ok, packet sniffer shows that Mozilla is starting to download mail for a third time. A check in the ImapMail directory shows this: drwxrwxr-x 3 minfrin fma 4096 May 22 00:43 . drwxrwxr-x 6 minfrin fma 4096 Mar 23 17:25 .. -rw------- 1 minfrin fma 0 Feb 8 18:38 filterlog.html -rwx------ 1 minfrin fma 22269035 May 22 01:05 INBOX -rw-r--r-- 1 minfrin fma 1337282 May 22 01:04 INBOX.msf drwxrwxr-x 3 minfrin fma 4096 May 20 21:02 INBOX.sbd -rw-r--r-- 1 minfrin fma 415 May 21 14:22 msgFilterRules.dat -rw-r--r-- 1 minfrin fma 8427145 May 22 01:04 nstmp -rw-r--r-- 1 minfrin fma 5918571 May 22 01:04 nstmp-1 The file INBOX is continuously getting bigger. The nstmp files seem to be copies of the files before they broke. INBOX continuously increases in size, way beyond the actual size of the IMAP INBOX, which is 38MB on the server. When I originally deleted my first INBOX, it was 250MB in size and had been downloading constantly over a modem for 3 days.
I'm surprised that we restart the downloading after the server disconnected, though that's probably a desired behaviour. But it sounds like we're not committing the db (the .msf file) in this case, so we don't realize that we've already downloaded some of the messages. Or else whatever is going on with the temp files is obscuring the fact that we've downloaded the messages. It's going to be tricky to try to reproduce this here, since my server doesn't drop the connection like that, but I'll see if I can fake it out. In the meantime, do you have some large messages, or lots of tiny messages? You can configure offline to not download the larger messages, which might make sense with your low speed connection.
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
The mailbox contains about 2500 messages, the vast majority small, with a few messages of a MB or two each. Setting the server to not download anything bigger than 100k, and filing the messages bigger than 500k away results in mozilla being a lot more responsive. The packet sniffer however shows that mozilla is still downloading mail at full tilt, with an INBOX size that has now reached 35MB and climbing. I am using courier-imapd v3.0.3, on a 64kbs connection. It has so far interrupted the download 4 times, and there are 4 nstmp* files.
Another possiblity: Mozilla is set to check for new mail every 15 minutes. However - it takes longer than 15 minutes to download the mail. Mozilla may be trying to download the mail multiple times, without checking first that a mail download is already running. iptraf shows two connections with traffic flowing towards mozilla, and netstat shows 4 connections open.
Another observation: If you restart mozilla, the imap works normally, fetching messages and downloading them as you click on them, and storing them locally. As soon as you select "compact folders", suddenly mozilla kicks off it's constant download from the IMAP server, immediately opening two connections to the IMAP server and downloading messages continuously using both connections. The only way to stop this is to exit mozilla.
That's an interesting data point - the compact tries to repair the offline store, if there are messages marked as being stored offline, but they're not correctly in the offline store - that might be a result of the tmp file issue. Well, I'll try to recreate this when I can...
Product: MailNews → Core
David do you see any reason to keep this given reporter writes "have super slow speeds, so I cannot confirm if this is still the case"?
Status: ASSIGNED → NEW
QA Contact: grylchan → networking.imap
I think the download stuff is going to be refactored by Emre such that this may not be an issue going forward. I still find it weird that it's creating nstmp files but the reporter definitely saw them, so I believe it's an issue.
(In reply to comment #9) > I think the download stuff is going to be refactored by Emre such that this may > not be an issue going forward. FWIW reporter writes "I've moved countries to one that doesn't have a broken internet infrastructure, so there isn't a proper way for me to test this"
Briefly off the broadband and onto 3G while it is repaired, which is dead slow by comparison. It looks like the problem is still there - when I checked a short while ago Thunderbird was downloading like crazy. I am glad I checked, otherwise I would be facing an enormous bandwidth bill (3G is paid for by volume in this case).
Product: Core → MailNews Core
Graham, do you still see problem using current nightly? backup your profile before using ... http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-trunk/
Keywords: perf
Need Graham's input to proceed but he seems to be gone. => incomplete
Status: NEW → RESOLVED
Closed: 16 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.