127398 - Searching with "body" makes GUI unresponsive

Status: VERIFIED FIXED

(SeaMonkey :: MailNews: Message Display, defect)

Description

[Håkan Waara]

Searching a folder full of messages, with one or more body criteria freezes
Mozilla  for a while when it searches. The CPU usage goes up to 100%, I believe.

This is only when I search with one or more "body" criteria, not when searching
subject criteria, for instance.

Build 20020204

Comment 1

[laurel]

Using feb27 trunk, win98: Body search on POP/local folders does incapacitate the
system for awhile. Even at some intervals when you can do something like switch
a window, it is extreeeeemely slow and will go back to grabbing power until the
search completes. Even the barberpole in the search ui freezes up. So far I
haven't seen any hangs, eventually lets go when the search finishes.  Have been
able to Stop the search.

Navin: comments?  Nomination? 

Comment 2

[Navin Gupta]

Body search is not a common search, atleast for me. Therefore I don't know
if this should be nominated. But it should not take 100% cpu. 

Comment 3

[Håkan Waara]

I use body search pretty frequently, so this is definitely dogfood for me. :-(  

nsbeta1+?

Comment 4

[Stefan Magdalinski]

I use it all the time, and I'm sure this didn't used to happen, or at least got 
better for a while when that great search speedup happened.

I've got a mail archive that goes back 4 years, and contains a lot of useful 
information, that often I can onlyfind by searching the bodies. 

is this a regression? 

seen on winXP pro, Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:0.9.9+) 
Gecko/20020317

Comment 5

[laurel]

*** Bug 142469 has been marked as a duplicate of this bug. ***

Comment 6

[laurel]

*** Bug 145264 has been marked as a duplicate of this bug. ***

Comment 7

[laurel]

*** Bug 146278 has been marked as a duplicate of this bug. ***

Comment 8

[laurel]

*** Bug 152535 has been marked as a duplicate of this bug. ***

Comment 9

[Tuukka Tolvanen (sp3000)]

*** Bug 163263 has been marked as a duplicate of this bug. ***

Comment 10

[Mario Ludwig]

On my build Mozilla/5.0 (Windows; U; Win98; en-US; rv:1.1) Gecko/20020826 on
WIN98SE body search on local mails is freezing in about 9 of 10 attempts. in 1
luckily attempt i get the desired result. when it is freezing, there helps no
waiting for hours, only kicking it via the task manager will help.
This was the same with all former builds, incl. 1.0

this is a major bug for me, because i very often use (better: would like to
use...) body search, and i think, this should be possible and not treated as
"rarely used by users" by the developer!!!

Comment 11

[scottputterman]

Putting this on the radar assuming the same code will be used to do scoring
based on mail body content.

Comment 12

[laurel]

*** Bug 167726 has been marked as a duplicate of this bug. ***

Comment 13

[Bernd Kuemmerlen]

This is a big problem for me as well, using
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.1) Gecko/20020826

Apart from the unresponsive GUI, a progress bar would be nice, so even if it 
takes a very long time, I could estimate how long the search will take.

Comment 14

[Rene Pronk]

Fixing bug 166502 (Mail search does a complete boolean evaluation) should help a
lot.

Comment 15

[David :Bienvenu]

there must be a dup of this bug around somewhere, because I commented, in
another bug, that instead of just simply evaluating 100 messages or so every
time slice, we could keep track of how much time we've taken in the current time
slice and return when we've consumed XX milliseconds (where XX is a number to
experiment with and see how the UI responsiveness is affected). An alternative
would be to look at the search term (to see if there was a body term), and the
msg size, and tweak the 100 number based on that. But I think the time approach
is probably more precise, and probably easier to code on top of that.

Comment 16

[Stefan Magdalinski]

yep, it's http://bugzilla.mozilla.org/show_bug.cgi?id=155800

and here's what you said. 

"yes, see the comment in bug 104243 - we probably need to throttle down the
number of messages we look at per time slice quite a bit in the case of body
search (and not the other cases). We could either examine the search terms to
see if there's a body term, or we could set a threshhold in milliseconds as to
how long we want to spend in the time slice and break out once we've exceeded it
- I'm not sure what the threshhold would be - maybe 100 milliseconds. This
latter solution would be the most general since it would be independent of
processor speed."

but while fixing this would improve the responsiveness of body search, fixing
166502 would save thousands of percent of pointless cycles.

Is this a dependency? I'm not quite up to speed enough with etiquette round here
to know how to mark it. 

Comment 17

[Miloslaw Smyk]

*** Bug 155800 has been marked as a duplicate of this bug. ***

Comment 18

[Miloslaw Smyk]

Updating Summary, as I believe that during search CPU use should actually go to
100%. The problem is that GUI is now (after a fix to bug #104243) handled a
little too seldom.

Comment 19

[Koike Kazuhiko]

I've found nsMsgSearchOfflineMail::ConstructExpressionTree() takes a lot of time.
If nsMsgSearchOfflineMail::MatchTerms() doesn't call it, the search finishes very
quickly.

http://lxr.mozilla.org/seamonkey/source/mailnews/base/search/src/nsMsgLocalSearch.cpp#529

Comment 20

[Koike Kazuhiko]

I expected replacing the recursive call in ConstructExpressionTree() with normal
loop
would improve performance, but no improvement happened.

Is it possible to execute the search in background?

Comment 21

[David :Bienvenu]

trust me, the problem is that we're reading too many message body lines in a
single timer event hander. Please see comment 15 again. I'd be happy to help
anyone who wants to try to fix this.
http://bugzilla.mozilla.org/show_bug.cgi?id=127398#c15

Comment 22

[Koike Kazuhiko]

Attachment: patch

I took bienvenu's advice.

Comment 23

[David :Bienvenu]

that looks like the right idea. I think I'd do more than 5 milliseconds worth
(and I can't remember if PRIntervalTime is in milliseconds or microseconds). If
you change it to 50 or 100 or more, does the UI feel sluggish? Also, how does it
affect search time, say, searching a large local folder?

Comment 24

[Koike Kazuhiko]

PRIntervalTime is in milliseconds. The threshold value in the patch should be 5000.
(I thought it was in seconds.)

I think the patch doesn't affect search time, but makes UI responsive. 

Comment 25

[David :Bienvenu]

so you mean 500, not 5000, I assume. 5000 would be 5 seconds, which would make
things pretty unresponsive :-)

Comment 26

[Koike Kazuhiko]

If the threshold is 500 milliseconds, only 2 or 3 messages are handled in each
time slice.
That makes the fix for bug 104243 meaningless.

Anyway it seems to me whether 500 or 5000 doesn't matter. In both cases UI can
respond, but very slowly.

Comment 27

[David :Bienvenu]

No, the fix for bug 104243 is very meaningful. Remember that not every search
searches over message bodies. Many searches just search over message headers,
which is very fast compared to searching message bodies, and I would think we
could do more than a 2 or 3 headers in half a second - I would think more like a
few hundred header comparisons in half a second. It has to be 500 milliseconds
or less - 5000 milliseconds is way too long to go without responding to a ui event.

Comment 28

[Koike Kazuhiko]

I see. threshold should be 500. Please review/superreview the patch. 

Comment 29

[Stefan Magdalinski]

While looking at this, you might want to have a look at 

http://bugzilla.mozilla.org/show_bug.cgi?id=166502 too. That bugfix could lead
to an actual performance increase in the case where people try to combine
criteria to make body searching less slow. 

Comment 30

[u69748]

Mozilla searches the text in multipart block,
even if it is base64-encoded  ;-(

Comment 31

[David :Bienvenu]

taking - I have a fix coded up - the attached fix was wrong because
PRIntervalTime is in microseconds, as I thought it might be. And that explains
why 5 or 5000 didn't matter, because 5 microseconds vs 5000 microseconds is not
an important distinction. It also explains why you were only getting 1 or 2
message bodies with each time slice. The upcoming patch set the time slice at
200 milliseconds - we can look at 40-80 message bodies per time slice (on my
machine) and 600-900 msg hdrs

Comment 32

[David :Bienvenu]

Attachment: proposed fix

set time slice interval to 200 milliseconds, and remove some unused code (make
footprint go down for a change :-) )

Comment 33

[David :Bienvenu]

Navin, can I get a review? thx.

Comment 34

[Navin Gupta]

Comment on attachment 107051 [details] [diff] [review]
proposed fix

looks good. r=naving

Comment 35

[(not reading, please use [e-mail] instead)]

Comment on attachment 107051 [details] [diff] [review]
proposed fix

sr=sspitzer

Comment 36

[David :Bienvenu]

fix checked in.

Comment 37

[laurel]

OK using dec19 commercial trunk: win98, linux rh8.0, mac OS 10.2