539194 - High CPU (100% for many seconds) caused by having news downloaded without bounds, i.e. no purging, resulting in large msf file. (pre-Gloda)

Status: NEW

(MailNews Core :: Networking: NNTP, defect)

Description

[Philip Chee]

+++ This bug was initially created as a clone of Bug #311774 +++

Possible bug: MailNews reported as high CPU being caused by having news downloaded without bounds, i.e. no purging. This could explain some people reporting 2.0 is slow as well due to large email or news folders causing blocked main threads when getting messages. Allegedly bug 311774 introduced the slowness.

Comment 1

[Hartmut Figge]

Attachment: log of CPU load during checking for new messages

Here is a log from the time before i get rid of this bug by unchecking '[x] Don't delete any messages' and using '[x] Delete messages more than 60 days old' for all of my new servers.

The log begins just before the start of an automatically check for new messages and ends just after this check has finished. Every second a new measurement during the log.

I had verified that bug 311774 is responsible for the high load by backing out the patch there. But i do not wish to do so because this patch is very nice otherwise. :)

Hence my other solution.

Comment 2

[Hartmut Figge]

(In reply to comment #1)

> Here is a log from the time before i get rid of this bug ...

Ehm, 'got' instead of 'get'. Well, i know why i am avoid writing in English. *g*

Comment 3

[Christian Mondrup]

(In reply to comment #2)
> (In reply to comment #1)
> 
> > Here is a log from the time before i get rid of this bug ...

I logged my SM 2.02 as installed on an Ubuntu 9.10 station. I followed the advice given in comment #1. But I still experience SM slow. For example my keyboard keying is delayed while I write this comment.

I'll attach my log.

Comment 4

[Christian Mondrup]

Attachment: For comment #3

Comment 5

[Hartmut Figge]

(In reply to comment #3)

> I logged my SM 2.02 as installed on an Ubuntu 9.10 station. I followed the
> advice given in comment #1. But I still experience SM slow.

Did you have the problem before upgrading to 2.0.2? I am asking this, because my first posting about it is from 27.9.2009 and i never have used a stable SM.

Comment 6

[Christian Mondrup]

(In reply to comment #5)
> (In reply to comment #3)
> 
> > I logged my SM 2.02 as installed on an Ubuntu 9.10 station. I followed the
> > advice given in comment #1. But I still experience SM slow.
> 
> Did you have the problem before upgrading to 2.0.2? I am asking this, because
> my first posting about it is from 27.9.2009 and i never have used a stable SM.

I've had the problem as least since I upgraded to SM 2.01. I'm not sure about version 2.0. But I never experienced that kind of slowness with pre 2.0 versions.

Comment 7

[Hartmut Figge]

(In reply to comment #6)

> I've had the problem as least since I upgraded to SM 2.01. I'm not sure about
> version 2.0. But I never experienced that kind of slowness with pre 2.0
> versions.

Then it is likely that you are also bitten by the patch. Wish, you could backout it by yourself to get a proof. *g*

For others my solution had worked. Not for you. Hm. Did you restart your SM after changing the settings?

Comment 8

[Hartmut Figge]

BTW, there is another bug related to this one. It may have existed before, but the occurrence has drastically increased.

If you attempt to look at a posting during the time of high load, chances are good, that you get an empty messagepane. Even the message source will be empty.

Clearing the cache, switching to another posting and then back fixes this. During the last days another solution was detected which worked for TB3 and for some others on SM2.

user_pref("browser.cache.memory.capacity", 0);

Comment 9

[Christian Mondrup]

(In reply to comment #5)
> (In reply to comment #3)
> 
> > I logged my SM 2.02 as installed on an Ubuntu 9.10 station. I followed the
> > advice given in comment #1. But I still experience SM slow.
> 
> Did you have the problem before upgrading to 2.0.2? I am asking this, because
> my first posting about it is from 27.9.2009 and i never have used a stable SM.

I've had the problem as least since I upgraded to SM 2.01. I'm not sure about version 2.0. But I never experienced that kind of slowness with pre 2.0 versions.

Comment 10

[Christian Mondrup]

Attachment: Logging of SM 1.1.18 CPU load

For comparison I installed SM 1.1.18. I was able to do that by adding libstdc++.so.5 from a Debian repository to my Ubuntu station. I've kept my home directory SM directory from previous installs and hence was able to run SM 1.1.18 for 'real use'

I attach a log of SM 1.1.18 CPU load. and had it running while using it actively. You'll notice a much more modest CPU load. While running SM 1.1.18 I did _not_ experience subjectively the 'slowness' I've reported for SM 2.0.2

Comment 11

[Wayne Mery (:wsmwk)]

how exactly does one see/force this?
how much cpu?

Comment 12

[Hartmut Figge]

You need enough subscribed news groups with a large enough amount of postings. Sorry, no precise values available. ;)

In the past a lot of users were affected and the issue could be solved for them by unchecking 'keep all messages' and checking 'delete messages more than 60 days old'.

To the cpu load, in my case i had 100% CPU for nearly 10 seconds. See the log in https://bug539194.bugzilla.mozilla.org/attachment.cgi?id=421489

Comment 13

[Wayne Mery (:wsmwk)]

does this need to be marked regression?
bad enough to block a release?  (I'm not clear how bad the performance is, but some people report unusable systems caused by combination of bad news activity and other performance bug)

ref bug 98534 Expired articles not automatically removed from list.

Comment 14

[David :Bienvenu]

Yes, this bug is real - if you haven't connected to your news server in a while, or have very high traffic groups, we now download all the new headers basically on startup, and tell gloda about the new headers, and things grind to a halt - I'm not sure if gloda is indexing the headers, or if it's just the notifications that make things really really slow.

Comment 15

[Jens Hatlak (:InvisibleSmiley)]

(In reply to comment #14)
> we now download all the new headers
> basically on startup, and tell gloda about the new headers

Note that this bug has first been reported for SM 2.0 which doesn't use gloda itself (and to my knowledge there are no extensions that activate it for SM).

Comment 16

[David :Bienvenu]

Ah, well, I was going to say, we also notify the folder pane about all the downloaded headers, but SM doesn't use the new folder pane either. In any case, we do start downloading a ton of news headers, and that bogs things down.

Comment 17

[Tilmann Reh]

I also noticed that SM gets extremely slow when news headers are polled. Here, it almost completely stops all UI functionality for several seconds (10-15 in my case, with only 12 groups subscribed) - even if only a few new news headers are downloaded. This is really boring - for example, if news polling starts in the background while you are writing a mail, and suddenly there is *no* reaction to your keypresses any more.

Appearantly, it's more related to the number of already stored/indexed message headers, than to the number of freshly loaded ones.

I can also see the CPU usage heavily increasing in the (Win XP) Task Manager. Due to the HT CPU, the Task Manager shows only 50 % but that is exactly full load.

But I have noticed something more: Looking at the file accesses of SM with Process Monitor while news are polled, revealed /thousands/ of accesses to the msf files of all groups. Even to those msf files that didn't get updated anyway. I think this might be one of the causes for the slow update process.

Finally, I think this was not introduced with SM2. With SM1, I already noticed that during news poll the program was not reactive to my keyboard input any more - for many seconds. I'd say that it got worse with the switch from Mozilla 1.6 to SM 1.0, but that's too long ago for exact references... Maybe it got even worse with SM2, but it's not new.

If I can help with further tests or investigations, I'd gladly do.

Comment 18

[Hartmut Figge]

(In reply to comment #17)
> I also noticed that SM gets extremely slow when news headers are polled. Here,
> it almost completely stops all UI functionality for several seconds (10-15 in
> my case, with only 12 groups subscribed) - even if only a few new news headers
> are downloaded. This is really boring ...

Yes, it is. Or was, because i was able to stop this annoyance. See comment #1.

> Finally, I think this was not introduced with SM2.

But i could identify the responsible patch. ;)

> With SM1, I already noticed
> that during news poll the program was not reactive to my keyboard input any
> more - for many seconds. I'd say that it got worse with the switch from Mozilla
> 1.6 to SM 1.0, but that's too long ago for exact references... Maybe it got
> even worse with SM2, but it's not new.

Mhm. After backout of the patch i haven't noticed this bug anymore. But it is surely possible, that this patch had worsened a formerly existent condition.

Comment 19

[Tilmann Reh]

>> This is really boring ...
>
> Yes, it is. Or was, because i was able to stop this annoyance. See comment #1.

You're lucky if that worked for you. It doesn't work for me, unfortunately. Changing those settings doesn't change anything here, so I don't think this is really fixed.

>> Finally, I think this was not introduced with SM2.
>
> But i could identify the responsible patch. ;)

That's why I explicitly mentioned that the bug/problem is older than that patch. It might have had some influence, but it's not the cause of the problem.

Just for completeness, I ran the Task Manager and Process Monitor (in sequence, not simultanously) during news polls with expiry set to 60 days. CPU usage 50% as before (meaning 100% due to HT mis-display), many seconds UI freeze, several thousands msf accesses without a single new header. (Tested with SM 2.0.4 under Win XP Pro SP3.)

Additionally, from a logic viewpoint it should be faster to not check for any expiry - scanning all msf files for possible expiries would explain those accesses (though they are far too many), but simply adding the new headers to the existing msf file would require only a single append operation.

Comment 20

[WADA:World Anti-bad-Duping Agency]

For high CPU% part.
Same problem as bug 541001(dup of bug 185634)?

Comment 21

[Wayne Mery (:wsmwk)]

can someone who sees this problem please reply about comment 20?  Do you see high memory usage?  (bug 185634)

Comment 22

[Philip Chee]

I have been experiencing this problem since upgrading to SeaMonkey 2.0.x including the symptoms in Comment 8. Consistent STR for me. (Win7/x64)

Open the browser surf around (open several tabs).
Open the mailnews window. (I have a large number of newsgroups some of which see heavy traffic).
The whole application including any open browser windows 'whiteouts" and is unusable for up to half a minute. Other open applications e.g. UltraEdit, TortoiseHg are not affected.

I think that this is certainly a regression in that previous versions don't freeze even when downloading headers or newsgroup bodies.

I don't think that this is a blocker in that there is no data loss, just a loss of productivity since the tabs I've been reading just freeze and whiteout for a while. But I believe that this has to be fixed sooner rather than later due to the sucky-UX.

Comment 23

[Wayne Mery (:wsmwk)]

joshua, if this is a regression, then it wouldn't be a simple mork issue, right?  (and for SM users wit wouldn't be gloda related)

What's needed?  A protocol log.  Or profile?