547950 - Eating too much disk i/o when global search and indexing is on

Status: RESOLVED DUPLICATE of bug 530098

(MailNews Core :: Backend, defect)

Description

[Abdullah AYKIR]

User-Agent:       Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/532.9 (KHTML, like Gecko) Chrome/5.0.307.7 Safari/532.9
Build Identifier: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.7) Gecko/20100204 Lightning/1.0b2pre Icedove/3.0.1 ThunderBrowse/3.2.8.1

I have about 10GB of an inbox size with tens thousands of email in several folders. When I first upgraded from 2.x to 3.x it started to index my folders and finished indexing successfully bu when i opened thunderbird yesterday it again started to index and it was not seeming to be finished so i looked at the memory usage by linux top command. What i saw was a strange behaviour. Ram usage was going up to 2.4 GB and than falling back to 1.2 GB and there was a high disk i/o. I tried to open it in safe mode and i oberved the status bar. It was indexing 80 messages when it reached 100% it started indexing 120 messages at 65% and after reaching 100% it raised the number to be indexed again. So i closed thunderbird and googled for similar problems. I found a page suggesting to turn off global indexing and filing a bug if problem goes away and i did so. I turned global indexing off and everything is normal now. The directive was at address https://wiki.mozilla.org/Thunderbird:Testing:Memory_Usage_Problems diagnosis step 8

Reproducible: Always

Steps to Reproduce:
1.Turn on global search and indexing
2.restart thunderbird
3.
Actual Results:  
It takes too long and never finishes to index.

Expected Results:  
It should show a progress bar about what is going on and allow us to cancel

I use debian. My default character set is utf8 for general system.

Comment 1

[Wayne Mery (:wsmwk)]

Abdullah, with indexing off and your largest folder displayed, how much memory is used?

Comment 2

[Kris Boulez]

I have exactly the same problem with TB3 on Windows (Vista). The initial indexing is taking too much CPU/RAM and make the program completely unresponsive ["program stopped responding"] for multiple minutes

Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.9pre) Gecko/20100219 Shredder/3.0.3pre

Comment 3

[Wayne Mery (:wsmwk)]

me mutters /why do people say "too much" and then not say how much, as though we know how much they mean?/

please spin through https://wiki.mozilla.org/Thunderbird:Testing:Memory_Usage_Problems

Comment 4

[Kris Boulez]

me not saying it is high memory usage.
me just saying that is completely reproducable and making TB3 completely unusable for multiple minutes and is to be fixed.

This is a plain install, no add-ons or whatsoever.

Comment 5

[Abdullah AYKIR]

(In reply to comment #1)
> Abdullah, with indexing off and your largest folder displayed, how much memory
> is used?
Hi Wayne,
Thanks for your reply.

Approximate normal usage of ram values are according to top linux command:
VIRT: 700MB
RES: 200MB
SHR: 25MB

and CPU usage is 0% when idle and not usually exceeding 30% when in action but it stuck 100% while i have the global search and indexing on.

I had turned off all my extensions and plugins before I knew it was caused by indexing and after discovering it I turned on all my extensions and plugins back and everything is normally functioning for me now.

Comment 6

[Wayne Mery (:wsmwk)]

200MB with indexing on or off strikes me as being high. 
is this imap account(s)?  can you get a msgdb log?

i have in the past experienced unnecessary reindexing, which is what your comment zero sounds like.
do you still reach 1.2 - 2gb memory usage?
how big is your largest folder on disk (file size)?

Comment 7

[Wayne Mery (:wsmwk)]

(In reply to comment #4)
> me not saying it is high memory usage.
> me just saying that is completely reproducable and making TB3 completely
> unusable for multiple minutes and is to be fixed.

without high memory usage your problem is likely a differnt bug

Comment 8

[Abdullah AYKIR]

(In reply to comment #6)
> 200MB with indexing on or off strikes me as being high. 
> is this imap account(s)?  can you get a msgdb log?
> 
> i have in the past experienced unnecessary reindexing, which is what your
> comment zero sounds like.
> do you still reach 1.2 - 2gb memory usage?
> how big is your largest folder on disk (file size)?

My biggest folder has a size about 3GB but wierdly it reports only 36kb when i check it with du -hs. This may cause the problems and may be related to OS I used. Probably the indexer is unable to find an EOF.

When I type ls -la it reports the file size correctly but if I type du -hs * I see a 0 in front of Inbox file.

I do not see any high memory usage at all.

RES memory is going up and down from 100MB to 200MB in different times but there  seems to be no problem. It reached 400MB while checking for new mails but dropped back to approximately 140MB without any problem.

I have about 10 accounts and all of them are pop3 accounts. I do not use imap.

I give the commands below to start logging
export NSPR_LOG_MODULES=MSGDB:5,timestamp
export NSPR_LOG_FILE=/var/log/msgdb.log
but the file is empty. So I tried all:1 and it worked but there is nothing about indexing, just normal activities.
After starting logging and turning the global indexing on abnormal memory usage seem to be gone away but cpu usage is high with weird status notification reaching upto 100% and going down to somewhere 65% 87% etc. with target number of messages to be indexed increased everytime.

I have unset the exported variables and tried again with global indexing on and nothing has changed except one thing it warned me about a running instance when I tried to restart it with global indexing turned off and I had to kill the process.

Now global indexing is off and it seems to be  working normal for me.

Comment 9

[Ludovic Hirlimann [:Usul]]

Are you synching your content on disk ? or are your message left on the server ? (looks like that by the size of your df output) ?

Comment 10

[Abdullah AYKIR]

(In reply to comment #9)
> Are you synching your content on disk ? or are your message left on the server
> ? (looks like that by the size of your df output) ?

I do not leave the message on the server I download them and the file size is reported correctly with "ls -la" command but it is reported with zero size when i used "df -h *". 

Maybe this is another problem about OS. 

I tried to tar my inbox to back it up and it did not finish even after creating an output which is bigger than my inbox. My disk filled up at the and and I cancelled trying to tar it. 

When I copied it with cp this is the result correct size with ls and wrong size with du. 

My file system is ext3.

I have noticed this problem while I was trying to back up my inbox when I started to move from Thunderbird 2.x to 3.x

I had no noticeable problem in 2.x about that

Comment 11

[Wayne Mery (:wsmwk)]

ludo, what do you think is next step?  

Just to be clear, Adullah's range of 100MB-200MB for RES column of top is fine for memory.  if it were not I would have suggested Bug 540214 (fixed in v3.0.3?) might be relevant.

Comment 12

[Andrew Sutherland [:asuth] (he/him)]

The I/O complaint sounds like high deletion costs (bug 530098); deletion will also exhibit the constantly increasing indexing count (intentionally), with jumps of 50 messages at a time (or 20? a constant increment most of the time anyways).  It may also have some bursty memory usage depending on the conversations involved.

Comment 13

[Wayne Mery (:wsmwk)]

andrew, thanks for the comment. So let's confirm, hang our hat on bug 530098 (which is blocking v3.1).

Comment 14

[Andrew Sutherland [:asuth] (he/him)]

The deletion cost bug has been fixed; the fix will be available in tomorrow's 3.1 nightlies.  Abdullah, if you could try one of these builds tomorrow and see how it improves your I/O situation, that would be great.  Note that at first start-up of the new version we will either 1) blow away your database if you haven't been using recent nightlies or 2) add a new index to the database synchronously that can will involve a fair amount of I/O.  Each of these things should only happen once, at least as long as you stick to the new nightly and don't jump around between versions.

The nightlies will show up here, but again you don't want the April 4th ones, you will need to wait for April 5th ones:
ftp://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-comm-1.9.2/

Comment 15

[Ludovic Hirlimann [:Usul]]

Abdullah ?

Comment 16

[Abdullah AYKIR]

Hi All,
Thanks for your kind replies. I checked the link Andrew provided for nightly download. There are only 32bit windows and i686 Linux packages in the list and I think it is the source code. I use 64bit debian. I will try to compile and use it and I will send you the results.
Thanks & best regards.

Comment 17

[Abdullah AYKIR]

Hi All Again,
I know I am very late for replying and thanks for your patient. The testing process takes about 10 hours of time for me and I could not have so much time since.
What I have done is shortly:
I tried to back up all of my existing files under .icedove folder in my home folder to en external drive. I faced with some problems caused by character sets. I used Turkish characters in the past to give folder names and I have changed my system to UTF8 later. Some letters turned to question marks in the file system and it was not possible to copy them to the backup destination.

Then I opened Icedove(Thunderbird's name on Debian) renamed folders with Turkish characters and searched in the file system recursively to find the files and renamed them.  I could be able to reach the contents of the folders and backup them.

The reporting total file size was different on my local disk and external backup. The external backup was about 2 times of the local one. It was because of some files reported 0 size. Then I deleted all my local files. Restored the files from backup and tried to open Icedove and open the global indexing. I saw the strange behaviour again as I have mentioned before. So I visited the url given above to get the daily snapshot and downloaded the one compatible with my system. 

I opened it and saw its name as Lanikai. Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.2.5pre) Gecko/20100509 Lanikai/3.1pre

It started to index my emails. It take some time and crashed after a while. I send the bug report via the dialog box that appeared telling me about the crash than I restarted Lanikai. It started to index again all of my emails and now it has finished indexing successfuly.

I will know turn back to Icedove and turn off global indexing. The addons I use are incompatible with the nightly.

I will wait for a new stable version to use this function.
By the way there are some other advancements on the GUI and it makes sense such as setting apart the "search" and "filter in the folder" and "quick filter part"

Thanks to you all for your great support.

Comment 18

[Wayne Mery (:wsmwk)]

Andrew is often spot on, so duping to bug 530098. Abdullah, please comment on your results after you start using version 3.1