Closed Bug 332309 Opened 19 years ago Closed 16 years ago

IMAP connection times out when server takes too long for a large operation

Categories

(Thunderbird :: General, defect)

Product:
Thunderbird
Component:
General
Platform:
x86
Linux
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED DUPLICATE of bug 409259

People

(Reporter: bugzilla+mozilla, Unassigned)

References

Details

(Whiteboard: dupem)

User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.8.0.1) Gecko/20060324 Ubuntu/dapper Firefox/1.5.0.1 Build Identifier: Thunderbird version 1.5 (20060309) Ubuntu/dapper 1.5-0ubuntu6 I don't know the specific number, but trying to move more than about 50 messages from one IMAP folder to another (on the same server) causes an IMAP server timed out message. I've tried increasing the TCP timeout several times (have it set to 300 seconds now). This causes thousands of duplicate copies of the messages in the destination folder after each attempt. I don't believe that the server is actually timing out, it just takes a long time to work and move that many messages, especially on a server with slow drives, etc. (like mine) I don't know how Thunderbird works internally, but it seems like it should move messages incrementally so as to avoid this problem. It shouldn't try to do it all at once within the timeout value. This is EXTREMELY annoying. Also moving these messages seems to take forever which is what causes it to timeout in the first place. Is Thunderbird actually copying them on the server, or is it retrieving them locally and then saving them again in the new folder? I'm trying to move over from Evolution and have not had these kinds of problems. Reproducible: Always Steps to Reproduce:
Related to Core bug 299332?
It sounds identical. I guess that patch is not in 1.5 yet? I tried to trace the server communication without encryption and found the problem: client sends: 9 uid copy 1:933,1598:2538,2543:2920 "INBOX.Trash".. Then after my timeout setting, I get back the "server timeout" message. I never get any other response from the server. It looks like the copy is not split up at all, just a huge number of messages and it takes my poor server a long time to copy them all. They don't show up in the folder right away, but even after Thunderbird times out, I find them on my server in the Maildir /tmp/ folder until the server finishes copying and then they show up. I'll try to test this with a newer build.
tbe fix for bug 299332 is in 1.5 - that fix was merely to generate shorter commands, not neccesarily fewer messages per copy command. Most imap servers can do a copy like "uid copy 1:933,1598:2538,2543:2920 "INBOX.Trash"" relatively quickly. But you can increase the timeout to as large a value as you want. An imap protocol log would tell you for sure if it's really just the server taking a really long time: http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html#imap
I did capture the log (with ngrep) last time, I only included the relvent (one) line in my previous post, since I get nothing after that on that connection. I can also confirm this bug occurs with latest trunk as well. "10 uid copy 1:3185 "INBOX.temp-delete".." times out again. It seems to me increasing the timeout value is not an appropriate fix. The server has *not* stopped responding, it is simply working. Ideally, the server should communicate this somehow, like with a response when each message is successfully copied, but I suppose it's not part of imap4. Would it slow down TB substantially to only copy a few messages at a time, or even one at a time and wait for a response from each one? If that wouldn't work, perhaps the timeout could be disabled (or increased to 30 minutes or something) when moving >100 messages or when the total size of the messages is over 100M or something? I don't know. I realise my old PIII 500 isn't the greatest mail server in the world but it should still be powerful enough to handle it's only user--me. And the server itself is doing nothing wrong, it's the client that is terminating the connection abruptly. I just think there should be some solution to this aside from "increase your timeout setting by guessing how long it will take your server to do what you tell it to."
Summary: Moving a large number of messages causes IMAP timeout → IMAP connection times out when server takes too long for a large operation
>Would it slow down TB substantially to only copy a few messages at a time, or >even one at a time and wait for a response from each one? Yes, that would be awful, for both the client and the server... If the server is going to go away for five minutes, that's really out of the ordinary. I'm not suggesting you guess how long things are going to take - I'd just set the timeout to some extremely high value, like 10000, and in essence just live without timeouts.
I understand what you are saying and I will do this and see if it fixes the problem for me, it just seems rather unintuitive for an ordinary user as "timeout" is usually associated with a something not working right, when that's not the case. Is it really that uncommon for a server to take 5+ minutes to copy a huge message folder, especially a multi-user server with 10's or hundreds of other users on it? It seems like that would always take a lot of time for just disk i/o alone If that's not so I guess I'll shut up and get myself a better server. If an imap5 ever comes out, I hope it supports some kind of ping during large operations to tell the client it hasn't timed out. Thanks
You're very right about this: >it just seems rather unintuitive for an ordinary user as >"timeout" is usually associated with a something not working right, when that's >not the case." but an ordinary user wouldn't be setting up his own imap server :-) >"Is it really that uncommon for a server to take 5+ minutes to copy a huge >message folder" Yes, that's really slow - I can copy 1000 messages in 10 seconds on my local machine with TB, and presumably, an imap server would have much better performance. Granted, you'd have multiple users using a server, but servers are usually much more powerful machines.
dupe?
Whiteboard: dupem
Ahhh! I just tested this with TB 2.0.0.12 and actually the problem is much worse now! When I set mailnews.tcptimeout to 3000 seconds, I was able to copy a 100M folder with 16,674 messages in about 5 minutes with no timeout errors. Once I reset the timeout to the default 60 seconds, however, I ran into major trouble. I no longer received any timeout errors, however when it did timeout (silently), TB would actually disconnect and start a new connection to the server and try to repeat the copy operation. However, even though TB had disconnected, the original imapd process on the server was still running, dutifully copying away... By the time I finally figured out what was going on, I had 12 imapd processes running concurrently, each copying the same 16,000 messages into the same folder! That folder was quickly growing in size, so I had to kill the imapd server and restart it. While TB was copying, I noticed that every so often the status bar message changed from "Copying messages" to "Checking server capabilities/Sending login information" again. This is why I assumed TB was establishing a new connection. Even though I killed the imapd processes before they finished, I already had tons of duplicate messages in the new folder I was copying to--as many as 12 dupes (one for each imapd process TB launched) of a single message. In my mind, the problems is even worse now than it was before. Imagine if you were copying an even larger folder! Or a smaller folder on a shared hosting provider with limited disk space--you would very, very easily eat up your storage quota. Perhaps there is also some misconfiguration of courier-imapd on my end--I'll look into it. In the meantime, I do think perhaps the severity of this bug needs to be raised. If I have a chance, I will try to test this on TB3 nightly as well.
dup/related bug 409259 ?
Assignee: mscott → nobody
Ryan, please check and write back what happens after bug 409259 gets fixed
Depends on: 409259
Seems to be the same as bug 40925 (symptoms are identical), suggest to DUP/merge (or at least set to "confirmed" status). Bug 446411 is another duplicate.
(sorry, same as bug 409259. cut&paste error)
Status: UNCONFIRMED → RESOLVED
Closed: 16 years ago
Resolution: --- → DUPLICATE
You need to log in before you can comment on or make changes to this bug.