48550 - Cannot open my Inbox of 17000 messages

Status: RESOLVED FIXED

(MailNews Core :: Networking: IMAP, defect, P3)

Description

[Frank Tang]

I am using nsbeta2 w/ IMAP to read my mail. My Inbox have 17000 messages . When
I click Inbox, it try to download some headers (up to about 6600) and then
nothing happen. No warrning no error. If I look at other folder which have 10
messages it will show me. I look at the process viewer in my NT (ctl+alt+del)
the memory usage is hitting 25M in netscp6.exe . I am running WinNT4 with 80M RAM.
I try several times but nothing happen when I try to view my giant Inbox. I
cannot trash 4.x and use SeaMonkey because of this reason. mark dogfood.

Comment 1

[Frank Tang]

jar ask me to file dogfood bug for any reason why I don't trash 4.x but use
SeaMonkey for my daily life. This is one of the main reason. I can use it at my
work since the machine at my office have more RAM. I run happy there although
the process view show I used 125M while I view/reply message of my giant Inbox
at my office. I cannot use SeaMonkey at home since my home machine do not have
125M RAM to assign to netscp6.exe 

Comment 2

[Scott MacGregor]

I'm pretty sure we don't have any plans for seamonkey to scale to 17,000
messages for a folder. You just aren't in our target audience for this mail
client. That doesn't mean you shouldn't be using us at work though!!!

But our decisions to use RDF, our in memory database, the tree widget, etc. just
aren't designed to handle or perform for 17,000 messages. I believe our targed
scale was around 9000. I'm sure marketing could correct me on that number.

In short, I'm afraid this bug is probably going to be a won't fix or a future
for 7.0......

Comment 3

[Jim Roskind]

One easy work-around (assuming your incoming rate is not incredibly incredibly
incredibly high) is to have an "archive" folder.  You can periodically select a
LOT of the messages in your inbox, and move them to the archive folder. The
result is that your inbox stays small enough, but only messages over a month or
two old get put into the "archive" folder.

I developed this trick in the early 4.x days, when large IMAP folders caused
tremendous performance problems.  With this sort of work-around, this bug stops
being dogfood pretty quickly (in fact, I bet our early 4.x imap cliets couldn't
handle handle this inbox count well!)

Hope that helps,

Jim (big inbox guy) Roskind

Comment 4

[Jim Roskind]

I should have mentioned... I have over 10000 messages in my inbox.  If your
browser stalls after downloading 6600, I suspect there is either a problematic
message arriving, or your local mail db has been corrupted.  I think I've faced
both problems (although the real mail guru's may jump in and correct me).

I can certainly believe that with 17000 messages you are using a LOT of memory,
but I suspect something else is tickling the situation.  With 10000 messages, I
startup the browser and jump to 20Megs of memory, then I open my inbox, and jump
to about 60-70 megs (this means that 10000 messages cost me about 40-50 megs).
My guess is that I could do another 10000 before I hit 120 megs (although memory
does get used up as a read the messages :-(.

Jim

Comment 5

[David :Bienvenu]

1. the 4.x slowlness that jar refers to was mainly a slowness with the 3.x
netscape imap server, which had n-squared imap select performance based on the
number of messages in a folder!. Once the 4.0 imap server rolled out, then you
could have giant folders and not run into server performance problems (though
you would start to run into client problems!)

2. The problem that Frank is seeing, is, I suspect, real, and doesn't have to do
with 17,000 messages. I think there's some sort of race condition that happens
when you download a large number of imap messages all at once - things just
stall out. It happens more on a high speed connection (i.e., I can't reproduce
it at home). 

Comment 6

[Jaime Rodriguez, Jr.]

Strongly suggest JAR's workaround.  Other's developer's need to be seeing this
problem to make this dogfood+.

Comment 7

[lchiang]

Will an IMAP log help in case it's due to a particular msg that Frank's Inbox is 
stalling out on?

Frank: http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html#imap

Comment 8

[leger]

Putting on nsbeta3 for Mail/News triage review.  Adding relnote3, in case we 
need to add this for PR3.

Comment 9

[lchiang]

Jan - we would like to get Frank's IMAP log (as per my comments) before we 
review.  

Comment 10

[Frank Tang]

I will then try to open some mailbox which have 9000 and see I can do that or 
not.

Comment 11

[lchiang]

Perhaps, we'll need Frank to go to someone's system and create a new profile to 
debug this?

Comment 12

[lchiang]

- per mail triage
Frank - please provide the info as requested in the last few comments.  Once 
done, we will look at this again for nsbeta3.

Comment 13

[laurel]

removing relnote3 keyword -- we don't know what happened here and we don't have
any other/current reports of not being able to open inbox.

Comment 14

[Frank Tang]

AFter I cut my inbox to 3643 messages (around that number), I can open my Inbox.
It is not fast but at least I can use it from home now.
The Process viewer show 11M when I see the Profile Manager. and show 15.5M after
I open my Inbox and view one message. Not too bad.
I believer our Mailer does not scale up like 4.x, but I don't think that is
important for SeaMonkey now.

Comment 15

[David :Bienvenu]

adding keyword - I still believe this is an actual bug.

Comment 16

[scottputterman]

marking nsbeta1-. We don't expect most people to have that many messages. If we
can find a case where this happens in a smaller folder then we should consider this.

Comment 17

[[e-mail] (gone - use [e-mail] instead)]

*** Bug 66258 has been marked as a duplicate of this bug. ***

Comment 18

[Ludovic Hirlimann]

Attachment: Ziped Log files

Comment 19

[Ludovic Hirlimann]

I have 7000+ mails on a notes 4.6x server.
When I try to connect to it It goes Wild and stop around 3795.
I already had that kind of problem see bug 66258, and readded a lot of mails in 
my Mail box to have some logs to give the mozilla team.
By the time of Bug 66258  I played on my server with the server's admin and we 
saw that problem occured with the imap daemon on that server , so it migth be 
server related ....

Ludo 

Comment 20

[selmer (gone)]

marking nsdogfood-, looks like a workaround has been suggested.

Comment 21

[jmorzins]

It's possible that Bug 103401 is a duplicate of this bug.  My belief is that
this bug is triggered by folders with too many gaps in the folder's UID sequence.  

I provided reproduction instructions in bug 103401.  I have often seen this bug
when trying to open my normal 2000-message INBOXes, and I can hand-craft a
400-message folder which always reproduces the bug.

I'll attach a log file of a session where I fail to read my folder named
"INBOX.2003", but I'll point out that the log file is ambiguous.  There's a "UID
FETCH" line in the logfile which is truncated at 510 characters, naming only 87
of the messages in my 400-message test folder.  Unfortunately, it appears that
the truncation at 510 characters was done by the logging routine; there's no way
of telling what MailNews actually sent to the server.  (grumble useless logger
grumble)
If it sent a single-line UID FETCH naming all 400 messages, the line would have
been more than 2000 bytes long.  It may not be safe to send servers lines this long.

Comment 22

[jmorzins]

Attachment: MailNews fails to open the folder named INBOX.2003

Comment 23

[Wayne Mery (:wsmwk)]

David, close this out?
Easily opens big mailboxes these days

Comment 24

[David :Bienvenu]

yeah, sure, wfm.