525707 - problems with imap/mailbox msgs flags

Status: RESOLVED INCOMPLETE

(MailNews Core :: Backend, defect)

Description

[Roberto Colmegna]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; it; rv:1.9.1.4) Gecko/20091016 Firefox/3.5.4 (.NET CLR 3.5.30729)
Build Identifier: TB 3.0 beta 4

I have a big imap stress-test folder with about 50,000 small messages 
(<3 Kbytes everyone).

I have a lot of very stranges behavior:
1) very slow "indexing" operation (minutes after FETCH/IMAP command is terminated)
2) changing flags msgs status ("readed") seem to operate only on client (no STORE/IMAP cmd sent to server)
3) "FETCH BODY" command (to scan folder) restart when arrive at about 30,000 msgs 

Reproducible: Always




It seem exists a lot of problem with big folder (no speaking abount 100,000 msg folders).

Note that the TB 2.0.0.23 works correctly, against the same folder/IMAP server.

Comment 1

[Wayne Mery (:wsmwk)]

this really needs split into 2-3 bugs.
Roberto, can you give more detail about your points #1 and #3?

Comment 2

[Roberto Colmegna]

point #3 appears randomically, while #1 is produced from this interaction:

1) user select a big-mailbox (50,000 msgs)

2) TB send to IMAP server a cmd "UID fetch 1598397:1598496,1602475:1647714 (UID RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type)"

3) after FETCH is completed user change to a small mailbox

4) TB start an "indexing" operation on the big mailbox (no net traffic, only loal CPU)

5) this operation works at about 300msg every 5 sec (on a AthlonX2-6400CPU).
   With a saturation of one CPU-core.  50000/300*5=13 min to complete indexing

6) if WHILE indexing user launch an "index rebuild" (via folder properties dialog box), TB restart to FETCH msg from the server (while continuous the other
local-index op. in step 5)

Comment 3

[Wayne Mery (:wsmwk)]

Roberto, what parts of this do you still see?
And, to what extent is bug 479450 involved?

Comment 4

[Roberto Colmegna]

Hi,

I'm sorry but in 2010 I totally dismissed the original stress-test environment, so I can't give you any information.

Comment 5

[Wayne Mery (:wsmwk)]

wada, irving, Is there something worth keeping and confirming here that isn't in bug 479450 ?

Comment 6

[WADA:World Anti-bad-Duping Agency]

I believe this bug is a report of mail data corruption in the past what is already resolved(filter move wrote wrong data in .msf, so Compact corrupted data in mail folder file, then Repair Folder(aka "Rebuild Index") detected From line at funny place/position).
I don't think it's need to keep open even though problem is already not reproducible in bug opener's environment.

Comment 7

[:Irving Reid (No longer working on Firefox)]

I don't think this has anything to do with the other folder corruption bugs. If we already have a bug for indexing performance, we should; we should also test to make sure that an existing background indexing job is cancelled when either the underlying message folder is rebuilt, or another index rebuild is manually requested.

Also, rebuilding the index should not fetch messages if the offline storage is up to date.

Comment 8

[WADA:World Anti-bad-Duping Agency]

(Correction of comment #6)
Sorry, I seem to have posted to wrong bug. That comment was for other mail data corruption problem. Please ignore my comment.

Comment 9

[Wayne Mery (:wsmwk)]

Per Roberto the testcase no longer exists, so => incomplete.
And this was reported against version 3.0, which had problems of this sort that have since been fixed.