Indexing causes high CPU usage

RESOLVED INCOMPLETE

Status

MailNews Core
Database
RESOLVED INCOMPLETE
8 years ago
7 years ago

People

(Reporter: David Hollman, Unassigned)

Tracking

({perf})

1.9.1 Branch
x86
Windows Vista

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [closeme 2011-02-11])

Attachments

(2 attachments)

(Reporter)

Description

8 years ago
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 ( .NET CLR 3.5.30729)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.10) Gecko/20100512 Thunderbird/3.0.5

I moved about 450 messages from a folder on one IMAP server to a folder on a different IMAP server.

This appeared to trigger indexing.  The Thunderbird.exe process would sporadically pin the CPU to 100%.  The UI did not become totally unusable, as I was able to close the program, but this did cause general system sluggishness.  The high CPU usage would fluctuate, lasting perhaps 10-20 seconds before returning to a lower value, then spiking up again, etc.

Ideally indexing should be more of a background task which never takes up a high percentage of resources when running...

Reproducible: Always

Comment 1

8 years ago
Maybe fixed by bug 530098 ; Firefox 3.1 will now delete faster, which should help in your case. But there are other issues like bug 547950 (heavy I/O)

Updated

8 years ago
Component: Search → Database
Keywords: perf
Product: Thunderbird → MailNews Core
QA Contact: search → database
David can you try 3.1rc1 and tell us if it's less CPU intensive for you ?

Comment 3

8 years ago
make sure your test is not using drag and drop - to avoid confusing with d&d performance issues. IOW, use command menu Message|Move|... or equivalent context menu.
Whiteboard: [needs retest v3.1 or trunk]
Version: unspecified → 1.9.1 Branch
(In reply to comment #0)
> I moved about 450 messages from a folder on one IMAP server to a folder on a
> different IMAP server.
> This appeared to trigger indexing.

As "move of mails" consists of (a) "copy of mails to move target folder" and (b) "delete(Shift+Delete) of mails from move source folder", needless to say, indexing always occurs.
  - Indexing of .msf,  at both move source folder and move target folder
  - Indexing by Gloda, at both move source folder and move target folder
David Hollman(bug opener), which indexing are you talking about? Which indexing did cause your high CPU usage?
At which step in "move mails"(a or b) does the "high CPU usage caused by indxing" occur?
It can be checked by;
(1) Keep the 450 messages in an IMAP folder of IMAP account-1(say folder A)
(2) Copy the 450 messages in folder A to folder B of IMAP account-1
(3) Move mails to other IMAP account's folder
(3-1) Copy 450 messages in folder B to folder C of IMAP account-2
(3-2) Select All, Shift+Delete at folder B of IMAP account-1
(4) Setup for next repeat:
    Select All, Shift+Delete at folder C of IMAP account-2
(5) Repeat step (2) to step (4), with Gloda=on/off, with auto-sync=on/off etc.
(Reporter)

Comment 5

8 years ago
I am now running:

Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.4) Gecko/20100608 Thunderbird/3.1

For some reason Thunderbird has started re-indexing my local folders which have thousands of messages.

I don't know why it is re-indexing... but this has reproduced the high CPU usage. So, no drag and drop involved, or any other UI interactions, as it started this on its own.  I don't think this is related to message movements in particular;  that just happened to be the trigger last time.

I have a dual-core machine and average total CPU utilization (as reported by Task Manager) is about 60-70%.  The thunderbird process itself reports upwards of 50%.

I have observed this in safe mode and normal mode.

Hope that is helpful...
(Reporter)

Comment 6

8 years ago
Created attachment 457086 [details]
Shows overall CPU activity
(Reporter)

Comment 7

8 years ago
Created attachment 457088 [details]
Shows tb process in particular;  no others are consuming much CPU
(Reporter)

Comment 8

8 years ago
One other thing I just noticed which could be relevant... when I shutdown TB (I used the Windows [X] button) the TB process takes a long time to actually stop even though the UI has disappeared.

Comment 9

8 years ago
(In reply to comment #5)
> I am now running: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.4) Gecko/20100608 Thunderbird/3.1
> 
> For some reason Thunderbird has started re-indexing my local folders which have thousands of messages.
> I don't know why it is re-indexing...

This would be normal. v3.1 has an updated schema for gloda - so the reindex is expected

> no drag and drop involved, or any other UI interactions, as it
> started this on its own.  I don't think this is related to message movements in
> particular;  that just happened to be the trigger last time.

correct. so the question now is, after the initial reindexing is done, does the cpu usage of "normal usage" cause a problem on your computer.


(In reply to comment #8)
> One other thing I just noticed which could be relevant... when I shutdown TB (I
> used the Windows [X] button) the TB process takes a long time to actually stop
> even though the UI has disappeared.

dunno
(In reply to comment #8)
> One other thing I just noticed which could be relevant... when I shutdown TB (I
> used the Windows [X] button) the TB process takes a long time to actually stop
> even though the UI has disappeared.

It's simply becuase that Fx's/Tb's Window is closed immediately regardless of required processes and/or required time to do shutdown normally.
See Bug 488809, please.
(Reporter)

Comment 11

8 years ago
Thanks for explaining why the reindex happened, and the shutdown behaviour.

I think though, the fact that the CPU usage is high enough that TB becomes sluggish (as well as other apps) is the crux of the matter.  To me, that seems like a bug, because indexing should be a "background" activity which does not impact usage of the computer.

Ideally, it would moderate itself so that wouldn't happen.

Thanks

Comment 12

8 years ago
For the purposes of this bug, we need to refocus on the original issue reported in comment 0, and the questions of comment 2, 3 and 4.

Comment 13

8 years ago
(In reply to comment #11)

> 
> Ideally, it would moderate itself so that wouldn't happen.

It actually tries to do that - but for some reason that moderation is not working on your machine+profile. But as Wayne said, we should focus on the issue reported initially.
(In reply to comment #11)
> I think though, the fact that the CPU usage is high enough that TB becomes
> sluggish (as well as other apps) is the crux of the matter.  To me, that seems
> like a bug, because indexing should be a "background" activity which does not
> impact usage of the computer.

"the fact that the CPU usage is high enough" is a penomenon of efficient use of CPU power, if CPU power is used for productive job only.
  Indexing by Gloda is CPU bound job. If (total required CPU power) =
  (CPU 100% of single Core)*(N seconds), (CPU 100%)*(N seconds) on sigle Core
  system, (CPU 100%)*(N/M seconds) on M Core, is an evidence of efficiecy.
Problem is unresponsiveness of main task(UI), or unresponsiveness of other application, when such high workload job of Tb exists.
An expample of it is bug 452221.
  "Delete of a mail" is usually never totally CPU bound job and "removing of
  entry for a mail" ends shortly, so UI simply waits for completion of the
  remove. It's usually never problem, because unresponsive time of UI is very
  small and unresponsiveness of UI is not visible for user.
  However, if bug 452221 occurs, it takes very long to remove entries for mails
  from .msf, and is CPU bound job. So, long unresponsive time of UI is exposed
  to user, and long period of CPU 100% occurs.
Generally speaking, to avoid such unresponsiveness, change like next may be required.
(a) Remove CPU usage for unproductive job as many as possible
    from the time consuming/CPU bound job.
(b) Make time consuming job "background job"(lower priority, separated task).
(c) Make such time consuming job asynchronous task for UI task,
    and UI waits for completion of the task asynchronously.
(d) Appropriate task priority setting within Tb, among Tb and other
    applications.
Without (a), action like (b)/(c)/(d) is nonsense.
David Hollman, my questios of comment #4 is to know;
  What job consumes 100% CPU power in your case? Is it for unproductive job?
  Is problem like bug 452221 involved in your problem?

(In reply to comment #0)
> The high CPU usage would fluctuate, lasting perhaps 10-20 seconds before
returning to a lower value, then spiking up again, etc.

If high CPU usage is by Gloda indexing, it's pobably due to periodical wait(speep) by indexer. It's to avoid flood of complaints of CPU hog from users just after migration to Tb 3(Note: Tb3 forces "Gloda enabled" upon migration).
You can see it by console log of Gloda.
> https://wiki.mozilla.org/Thunderbird:Debugging_Gloda
The wait(sleep) caused long time to complete full indexing of existent mails as initial indexing after migration. So, phenomenon like next was reported from a user who has very large volume of existent mail data(IMAP user, enabled offline-use=on, after all IMAP mails are downloaded by auto-sync).
  Initial indexing doesn't comeplete even after two days running of Tb.

David Hollman, you are MS Windows user. Ms Windows has feature of "Performance Counter log". Did you do effort such as "checking of CPU utilization of each Tb's thread by Performance Counter log" after opening of this bug at B.M.O for problem relevant to CPU utilization?
Initial report/feeedbak of "non-ordinal high CPU usage by Tb" is usefull.
But repeated complaints of "Tb consumed high CPU power of my PC!" only won't resolve any problem.

Comment 15

7 years ago
(In reply to comment #14)
> (In reply to comment #11)
>...
> Generally speaking, to avoid such unresponsiveness, change like next may be
> required.
> (a) Remove CPU usage for unproductive job as many as possible
>     from the time consuming/CPU bound job.
> (b) Make time consuming job "background job"(lower priority, separated task).
> (c) Make such time consuming job asynchronous task for UI task,
>     and UI waits for completion of the task asynchronously.
> (d) Appropriate task priority setting within Tb, among Tb and other
>     applications.
> Without (a), action like (b)/(c)/(d) is nonsense.
> David Hollman, my questios of comment #4 is to know;
>   What job consumes 100% CPU power in your case? Is it for unproductive job?
>   Is problem like bug 452221 involved in your problem?

David, ^^


> (In reply to comment #0)
> > The high CPU usage would fluctuate, lasting perhaps 10-20 seconds before
> returning to a lower value, then spiking up again, etc.
> 
> If high CPU usage is by Gloda indexing, it's pobably due to periodical
> wait(speep) by indexer. It's to avoid flood of complaints of CPU hog from users
> just after migration to Tb 3(Note: Tb3 forces "Gloda enabled" upon migration).
> You can see it by console log of Gloda.
> > https://wiki.mozilla.org/Thunderbird:Debugging_Gloda
> The wait(sleep) caused long time to complete full indexing of existent mails as
> initial indexing after migration. So, phenomenon like next was reported from a
> user who has very large volume of existent mail data(IMAP user, enabled
> offline-use=on, after all IMAP mails are downloaded by auto-sync).
>   Initial indexing doesn't comeplete even after two days running of Tb.
> 
> David Hollman, you are MS Windows user. Ms Windows has feature of "Performance
> Counter log". Did you do effort such as "checking of CPU utilization of each
> Tb's thread by Performance Counter log" after opening of this bug at B.M.O for
> problem relevant to CPU utilization?

David, ^^
Whiteboard: [needs retest v3.1 or trunk] → [closeme 2011-02-11]
RESOLVED INCOMPLETE due to lack of response to previous question. If you feel this change was made in error, please respond to this bug with your reasons why.
Status: UNCONFIRMED → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.