123922 - removing many expired messages can be very slow

Status: RESOLVED FIXED

(MailNews Core :: Networking: NNTP, defect)

Description

[Mark Hanson]

From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.7) Gecko/20011221
BuildID:    2001122108

In high traffic newsgroups, clicking on the link "click here to remove all
expired articles" can be extremely slow.  It can take tens of minutes to finish
on a 500MHz Celeron with plenty of memory (no swapping).  During this time, a
mozilla-bin process hogs the CPU and strace shows that it is executing many,
many redundant __llseek() and fstat() calls with the same arguments.

_llseek(60, 0, [835014], SEEK_CUR)      = 0
fstat(60, {st_mode=S_IFREG|0644, st_size=835014, ...}) = 0
_llseek(60, 835014, [835014], SEEK_SET) = 0
_llseek(60, 835014, [835014], SEEK_SET) = 0
fstat(60, {st_mode=S_IFREG|0644, st_size=835014, ...}) = 0
_llseek(60, 835014, [835014], SEEK_SET) = 0
_llseek(60, 835014, [835014], SEEK_SET) = 0
fstat(60, {st_mode=S_IFREG|0644, st_size=835014, ...}) = 0
_llseek(60, 835014, [835014], SEEK_SET) = 0
_llseek(60, 835014, [835014], SEEK_SET) = 0
_llseek(60, 835014, [835014], SEEK_SET) = 0
write(60, "\n@$${B031{@\n<(A4D9=5b7a)>[4A:^80"..., 55) = 55
_llseek(65, 0, [14783003], SEEK_CUR)    = 0
fstat(65, {st_mode=S_IFREG|0644, st_size=14783003, ...}) = 0
_llseek(65, 14783003, [14783003], SEEK_SET) = 0
_llseek(65, 14783003, [14783003], SEEK_SET) = 0
fstat(65, {st_mode=S_IFREG|0644, st_size=14783003, ...}) = 0
_llseek(65, 14783003, [14783003], SEEK_SET) = 0
_llseek(65, 14783003, [14783003], SEEK_SET) = 0
fstat(65, {st_mode=S_IFREG|0644, st_size=14783003, ...}) = 0
_llseek(65, 14783003, [14783003], SEEK_SET) = 0
_llseek(65, 14783003, [14783003], SEEK_SET) = 0
fstat(65, {st_mode=S_IFREG|0644, st_size=14783003, ...}) = 0
_llseek(65, 14783003, [14783003], SEEK_SET) = 0
_llseek(65, 14783003, [14783003], SEEK_SET) = 0
_llseek(65, 14783003, [14783003], SEEK_SET) = 0
write(65, "\n@$${2538{@\n[244C:m(^91=0)]\n{1:^"..., 220) = 220

What benefit is there to seeking to the same point in a file three times in a
row?  Looks like this can be optimized quite a bit.

Reproducible: Sometimes
Steps to Reproduce:
1. Find a newsgroup with a lot (thousands) of expired articles.
2. Click on an expired article.
3. When the message appears, click on the link to remove all expired articles.
4. Run strace on the mozilla-bin process which is using all the CPU.

I believe that the reproducability of this depends on the number of articles it
needs to process.  If there aren't many, the process can only take a few seconds.

Actual Results:  A bit of the strace output is in the Description section.

Expected Results:  It should have completed the task more quickly.

Comment 1

[David :Bienvenu]

Rob, that's not a database issue; it's a news issue. It sounds like news is
committing the db way too often in this scenario.

Comment 2

[[e-mail] (gone - use [e-mail] instead)]

-> seth, me

Comment 3

[David :Bienvenu]

Attachment: proposed fix

don't commmit the db for every header removed - the db will be committed at the
end of the process anyway

Comment 4

[(not reading, please use [e-mail] instead)]

Comment on attachment 82125 [details] [diff] [review]
proposed fix

sr=sspitzer

excellent!

Comment 5

[David :Bienvenu]

taking

Comment 6

[David :Bienvenu]

*** Bug 76232 has been marked as a duplicate of this bug. ***

Comment 7

[[e-mail] (gone - use [e-mail] instead)]

Dup'd bug was win2k, marking os-->all.

Comment 8

[[e-mail] (gone - use [e-mail] instead)]

*** Bug 124673 has been marked as a duplicate of this bug. ***

Comment 9

[[e-mail] (gone - use [e-mail] instead)]

David - you landed this on the trunk but haven't yet marked it, are you
intending this to be landed on the branch, too?

Comment 10

[David :Bienvenu]

no, I just forgot to mark it fixed. I'll nominate it for the branch, since it's
such a trivial fix, but I don't think anyone in charge is going to care.

Comment 11

[mg8]

This occurs still in RC2 (build 2002051005) on MacOS9.2. I see only "Loading
document ..." and then nothing more. I needed to use Force to quit. This sucks!

Comment 12

[David :Bienvenu]

the fix is only on the trunk, not in rc1 or rc2, as my previous comment should
have made clear.

Comment 13

[mg8]

What needs then to be done to get it into the releases?