104676 - Local folder mail viewing of a large mailbox is too slow due to mcafee virusscan

Status: VERIFIED FIXED

(SeaMonkey :: MailNews: Message Display, defect, P1)

Description

[Katsuhiko Momoi]

** Observed with 20011013 Win32 trunk build  **

Problem description:

0. I have an archived mailbox containing about 13,000 messages.
   My machine has Intel 850 MHz processor w/390 MB RAM. So this is
   a pretty fast machine.
1. When I view a locally saved mailbox with about 13,000 messages 
   in it, the loading speed is extremely slow. 
   I measured loading times in Local Folder with 2 different sizes
   of mailboxes:

   Typical 20-30 line msgs:

   13,000 msg mailbox (no image): about 5-6 seconds
   13,000 msg mailbox (1 image): about 8-9 seconds
      300 msg mailbox: about 1 second

   By comparison, 

   5000 msg IMAP mailbox: less than 0.1 seconds
     66 msg POP mailbox: less than 0.4 seconds (Is there a difference between
                                                 Local Folder and this?)

Generally, loading of local messages is terrible. It is actually slower
than 4.x peformance slightly. 

Certain functions associated with mail headers are also very slow:

A. deleting 1 message with about 50 lines in 13,000 msg mailbox

   10 seconds. Doing this 2-3 times bumps up CPU consumption to
   about 90%, sometimes to 100%.

B. Copy/Moving and doing anything to do with local msgs takes a long
   time.

** The only excpetion is searching. Bug 104243 truly improved 
search speed as of the 20011012 trunk builds. Local Search is
now usable.

Comment 1

[Katsuhiko Momoi]

I should add that repeated local message viewing (normal activity
if you're looking at archived messages) bumps up CPU 
consumption considerably and slows down my machine. 

Comment 2

[Karl Johan Kleist]

Does this problem really deserve the severity level "critical"?
---
Severity
This field describes the impact of a bug.

Blocker - Blocks development and/or testing work
Critical - crashes, loss of data, severe memory leak
Major - major loss of function
Minor - minor loss of function, or other problem where easy workaround is present
Trivial - cosmetic problem like misspelled words or misaligned text
Enhancement - Request for enhancement
---
Seems more like "major" to me.

Comment 3

[Håkan Waara]

This is probably something that perf. people should purify.

Not a critical bug, though, afaik.

Comment 4

[Katsuhiko Momoi]

There is severe memory leak while viewing msgs and performing
normal functions such as deleting a few msgs.
I'm afraid I disagree with both of you. Try similar 
expriment and see for yourself.

Comment 5

[[e-mail] (gone - use [e-mail] instead)]

Are you on the trunk or 0.9.4 commercial branch?  We've fixed a _lot_ of leaks
lately (though we still have much more to go).

Comment 6

[Håkan Waara]

A purify log for momoi's testcase would probably help here, so we can see the
culprit.

Comment 7

[[e-mail] (gone - use [e-mail] instead)]

I need to learn to read.  You clearly said trunk, sorry.

Comment 8

[[e-mail] (gone - use [e-mail] instead)]

Kat, can you send me via mail your local folder that is giving you performance
issues?  I'll both Purify it and measure performance.  Thanks.

Should be similar to this structure:
C:\Documents and Settings\Administrator\Application
Data\Mozilla\Profiles\stephend\xcov9trc.slt\Mail\Local Folders

Comment 9

[[not reading bugmail]]

this seems to be because the storage method in use here.. if you have
attachments and many main messages.. it all gets put in that file in you mail
folder, they dont seem to be saved outside the file.. (I bet if we change some
way the indexing is handled inside the file, either thru perf updates.. ie
finding a faster work around  for indexing..  or by changing the way mail is
stored on the computer. 

Comment 10

[scottputterman]

Adding nsbeta for investigation of the leaks mentioned.  I don't think
optimizing the performance of a 13000 msg folder is critical. But if we are
seeing the slow downs mentioned in the 100 message folder, I think we should
investigate.

Comment 11

[scottputterman]

I'm experiencing this on a much smaller folder. I have a 930 msg folder in my
Local Folders.  I'm on a 700 MHz machine running Win 2K.  Every time I try to
read a msg (and they are only about 1KB) it takes about 3-5 seconds to read it.
It didn't used to be this way.  I also noticed a long delay when trying to copy
messages to this folder.

Comment 12

[scottputterman]

reassigning to naving

Comment 13

[Navin Gupta]

I think reading local msgs has always been this slow. I am saying this based
on the performance data that stephend collects every week and also because 
I haven't seen any change from a user's perspective. I see whenever we 
read a message cpu goes upto 100% and this is also true for imap. This is why
when you move a message it seems slow because we select the next message. 
copy doesn't cause any spike in cpu. 

Also just compared it with 4x and 4x is really very fast here and there is 
no spike in the cpu. 

Comment 14

[scottputterman]

It hasn't always been this slow.  My machine at home seems to freeze for a few
seconds while reading messages in my local folder.  Imap messages take a
fraction of the time.  Also, my machine at work isn't exhibiting this behavior.

Comment 15

[scottputterman]

There seems to be a possibility that there's something wrong with the file
itself. Navin mentioned to me that a couple of other people who experienced
these problems were having problems with their file systems.  So, I went to
mailbox file that was having the problem and sure enough, selecting it in the
filesystem froze Windows Explorer for a noticeable amount of time. Same thing
when I tried to get the properties for the file.  I went to my degfragmenter and
found that this file is severely fragmented. I then brought this file over to
another machine with the same build and it worked as expected without any delay
in reading messages.

It's a little suspicious that this problem started occurring to a number of
people at the same time though.  When we read a message, do we open and close
the mailbox each time? Or do we only close it when switching folders?

Comment 16

[Navin Gupta]

>When we read a message, do we open and close
>the mailbox each time? Or do we only close it when switching folders?

We always make a new connection when we read local messages and get rid 
of the connection once we are done reading the message. The fileTransport 
reads the data from the berkeley mailbox and hands it to us. In short words
we do open and close the mailbox, each time we read. 

We should reuse connections like imap, this can be a big performance win.

Comment 17

[scottputterman]

I only mention that because it seems that my very fragmented file is having
trouble being opened.  

Comment 18

[Navin Gupta]

So I have investigated the problem and the majority of the time is spent in 
opening(PR_Open) & closing mailbox (PR_Close). So the files which take longer
to open and close exacerbate this problem. Some files really take long time to
open 

There is now some discussion about reusing the connections that will allow us
not to open and close mailbox unless needed. This should improve performance on
all local folders operations. 

Comment 19

[timeless]

heh, the bugs original description matches my experience this week (n6.2)
exactly except that it was my inbox, not some archive.

Comment 20

[(not reading, please use [e-mail] instead)]

we figured it out that it is the virus scanner.

naving, can you add some info?

Comment 21

[Navin Gupta]

The files which take long time to open and close are because the virus scanner
running in background gets triggered on opening and closing the file. This is 
deep within OS, most probably when OS does an open/fopen. So the workaround is
to rename those folders to have .txt extension, which atleast on  my machine does
not cause the virus scanner to jump in. you can look into your virus scanner and
see what extensions are safe. I agree with scott that this happens most likely
when you have copied folders manually. It doesn't happen when folders are
created by mozilla. 

Comment 22

[Katsuhiko Momoi]

> I agree with scott that this happens most likely
> when you have copied folders manually. It doesn't 
> happen when folders are created by mozilla. 

Question: What counts as "created by Mozilla". I created the 
problem mailboxes in the following way:
1. Using Mozilla, create a local mailbox.
2. Copy all the msgs from the current IMAP Inbox into
   the local mailbox created above.

Comment 23

[Navin Gupta]

>Question: What counts as "created by Mozilla". I created the 
>problem mailboxes in the following way:
>1. Using Mozilla, create a local mailbox.
>2. Copy all the msgs from the current IMAP Inbox into
>   the local mailbox created above.

That is "created by mozilla", but you will have to look at your virus settings 
and task manager to verify if this also triggers the scan and change 
your settings appropriately.

Comment 24

[scottputterman]

This is happening to a number of people's Inbox.  If they rename it to Inbox.txt
will it stop behaving as the Inbox?

Comment 25

[scottputterman]

moving to 0.9.7.  If we can come up with something ther than a release note,
then we can try to get this into 0.9.6.  

Comment 26

[Navin Gupta]

Summarizing again, this slowness depends upon your virus scan settings which 
determines whether your folders created by mozilla will be affected or not.

Comment 27

[Navin Gupta]

All the people within netscape who are experiencing this slowness is due
to McAfee Anti Virus Scan 4.5.1. There could be other point releases. 

I am using mcAfee anti virus scan 6.01 and experience no slowness, and i was 
experiencing slowness when I was using 4.5.1. Please download 6.01 or some
other virus scanner. esther has verified this finding. 

Comment 28

[[e-mail] (gone - use [e-mail] instead)]

Esther, Sheela, myself and Navin all looked at this on various machines, and by
all accounts this seems 'fixed' when those experiencing the issue upgraded their
anti-virus client.

Verified