476338 - infinite retry loop with IMAP server on a big offline folder (due to server crash)

Status: RESOLVED DUPLICATE of bug 409259

(MailNews Core :: Networking: IMAP, defect)

Description

[Roberto Colmegna]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; it; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5
Build Identifier: 

I'm executing a stress test from TB and an IMAP server.  I have a single IMAP-folder which contains 6000 msgs with a total size of 3GBytes (yes .. giga!).

With the default IMAP config TB works well (prefetching headers). But If I mark the folder as "offline usable" TB execute a command like this:

tf65 UID FETCH 30000:36000 (BODY[])

The server crash down due to huge msgs-set (a server bug). But TB retry immidialety with the same cmd (on another session), recrashing the server.
This generate an infinite loop.

I think that splitting command on huge continuous interval as a lot of commands which works on small interval (possible auto-tunneable) could improve the TB efficiency.  This is also useful in case of slow/unstable connections. 

Reproducible: Always

Steps to Reproduce:
1. <not appliable>
2.
3.
Actual Results:  
infinite wait

Expected Results:  
finite wait

Comment 1

[WADA:World Anti-bad-Duping Agency]

Even if server won't crash, if "UID FETCH 30000:36000 (BODY[])" won't complete within a threshold time, similar retry of command will probably occur. See Bug 410692 for timeout/retry in UID COPY case.
Setting dependency to Bug 410692.

Comment 2

[Wayne Mery (:wsmwk)]

Reberto, please retest after fix for bug 410692 lands
ftp://ftp.mozilla.org/pub/thunderbird/nightly/latest-trunk/

Comment 3

[Roberto Colmegna]

With the new version the performance have improved of a "perceived" factor of 3x.

But ... I have tried to COPY 4000 mails from a folder to another (src and dst
on the imap server) and I obtain this effect:
1) CPU burning (100%) on the client for 4 secs
2) no network sent to IMAP srv (network sniff)
3) error: "A script on this page may be busy ..." (I click "continue")
4) command completed with SUCCESS

I will retry with 200,000 mails in folder.

Comment 4

[Roberto Colmegna]

I have also noted that when I move tons of mail TB execute very frequent 
"compact folder" which degrade the general performance.

Previous msg note: with 8,000 mails the moving became near to impossible due
to a lot of warn msg.

Comment 5

[Wayne Mery (:wsmwk)]

Roberto, can you retest using version 3?  your test could not have exercised the patch which fixed bug 410692, because it was not fixed until   2009-04-15. after testing version 3, and if you no longer see the effects of comment 0 now that Bug 410692 is fixed, then we can close this WFM. Or we can dupe this to that bug.  (there is also bug 403603, which needs another look)

There are other bugs that deal with the performance issues of moving many messages.

Comment 6

[Roberto Colmegna]

The IMAP server is now fixed and not crash under hi load.

About performance: I have tried with a 40,000 msg folder and I noticed that 
TB require less than one minute to retrieve headers but a lot of minutes (about 
7) to execute "Indexing" step (without any move operation).

Testing used TB/RC1.

Comment 7

[Wayne Mery (:wsmwk)]

bievenu, with the fixing of bug 409259, does the following sound good to you?

(In reply to comment #6)
> The IMAP server is now fixed and not crash under hi load.
> 
> About performance: I have tried with a 40,000 msg folder and I noticed that 
> TB require less than one minute to retrieve headers but a lot of minutes (about 
> 7) to execute "Indexing" step (without any move operation).
> 
> Testing used TB/RC1.

Comment 8

[David :Bienvenu]

wsmwk - yes, sounds good.