120712 - Enormous memory usage during bug search

Status: RESOLVED FIXED

(SeaMonkey :: Search, defect)

Description

[Dom Incollingo]

From Bugzilla Helper:
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:0.9.7+)
Gecko/20020116
BuildID:    2002011604

During a specific bug search on http://bugzilla.mozilla.org/, Mozilla memory
usage skyrocketed to approximately 200 megabytes!

I am running Mozilla build 2002011604 on Windows 2000.  My computer has 512 meg
of memory.

Reproducible: Always
Steps to Reproduce:
1.Launch Mozilla and navigate to http://bugzilla.mozilla.org/
2.Launch Windows Task Manager.   Mozilla is using about 20 meg of memory.
3.Enter a forward slash (/) in the field entitled "Enter a bug # or some search
terms".   (I was searching for a bug whose error message begins with a slash).
4.Press the 'Show' button.
5.Go back to Windows Task Manager and watch Mozilla's memory consumption.
6.Memory rises steadily to about 40 - 50 meg until the browser displays the next
page with the bug list.
7. After the browser displays the next page with the bug list, memory
consumption CONTINUES to rise quickly and steadily.  In a few seconds, Mozilla
is using approximately 200 meg of memory! 
8.Memory usage remains at about 200 meg long after the bug list is displayed.

Actual Results:  Memory usage quickly went from 20 meg to approximately 200 meg.

Expected Results:  Memory usage should have climbed slightly, but not nearly to
200 meg.

I have done the exact same bug search using Netscape 6.2.1 and the Mozilla 0.9.7
monthly milestone build.  In both cases, memory usage climbs to about 40 meg,
and then levels off.   It does not climb higher than 40 meg.  This leak appears
to have been introduced after the 0.9.7 Mozilla build.

Although this situation (a bug search using a /) is uncommon, I believe that
this bug is critical because it is indicative of a serious memory leak somewhere
in browser processing.

Comment 1

[Dom Incollingo]

Attachment: Screenprint showing memory consumption after bug search

Comment 2

[Matthias Versen [:Matti]]

wfm with win2k build 20020117.. (59MB RAM)

Comment 3

[Dom Incollingo]

I can re-create this problem using build 2002011803.  Also, I posted a message
on the Mozilla newsgroup, and another user with W2K-SP2 / 512MB (and the latest
nightly build as of this morning) was also able to re-create this problem. 
Perhaps  the problem has something to do with the fact that our PCs have 512 MB
of RAM.

Comment 4

[Matthias Versen [:Matti]]

no, my system have also 512BM RAM 
BTW: I can't see your screenshot because I don't have PPT and many other people 
can't see it because they use linux..

Comment 5

[Dom Incollingo]

Attachment: Here is a JPG screenshot of the memory leak

If you check the netscape.public.mozilla.browser newsgroup for today (1/19/02)
at about 12:05 PM, you will see where I posted this problem.  Someone with the
same machine configuration as mine responded that Mozilla was at 213MB when
they followed the steps listed in this bug.

Comment 6

[timeless]

This is *not* mozilla's fault.  If anything, it /might/ be bugzilla's.

you search probably resulted in a pull of all bugs you can read in the 
database.
[nc4 top output]
Bug List Sat Jan 19 18:48:13 PST 2002 This list is too long for bugzilla's 
little mind; the Next/Prev/First/Last buttons won't appear.

Comment 7

[Dom Incollingo]

Attachment: JPG screenshot of the same Bugzilla search using Netscape 6.2.1

If this is * Not * the fault of Mozilla, then why does Netscape 6.2.1 use only
40 MB of memory when doing the exact same search?  (As illustrated by the third
attachment)   The search done with NS 6.2.1 used the same Bugzilla processing
and returned the same number of bugs.  The only difference was the version of
the browser.  I have also done the same search using Mozilla 0.9.7 (monthly
milestone),  and it tops off at 50 MB even though Bugzilla returns the same
number of bugs and (presumably) performs the same processing.

Comment 8

[Dave Miller [:justdave]]

per additional information provided by the reporter I concur that he is
complaining about a Mozilla issue, not a Bugzilla one.  The "list is too big for
Bugzilla's little mind" message isn't what he's complaining about.  The amount
of memory used by the browser to process the returned data is what he's
complaining about (specifically, a 160 MB difference in memory usage between NC
6.2.1 and the current Mozilla for the same data).

Moving back to Browser...

Comment 9

[Jonas Jørgensen]

Reproduced on build 2002011604, Win2k. Confirming.

Comment 10

[Travis Crump]

this also occurs on Linux(Debian/woody, Gnome 1.4)with build 20020115.  Memory
usage as measured by top(s=0.5) jumps 160M after the bug list finishes loading
in 20-30M increments.  Presumably since the page has finished loading Mozilla
shouldn't be doing anything computationally/memory intensive.

Comment 11

[Asa Dotzler [:asa]]

not a leak, just a high memory usage. Over to Layout folks. 

Comment 12

[anthonyd]

Attachment: my mem usage on win2k for same query

for what its worth, my screenshot of my memusage for this query (35 megs).

Comment 13

[anthonyd]

markling this bug WFM on win2k.  i use up 35 megs (see attachemnt screenshot).

Comment 14

[eberry]

The steps still reproduce the bug with today's Mozilla trunk build from 
2002-02-04-09 under W2K SP2 with a completely new profile.  It might be related 
to parsing the bug list for the sidebar, and this feature might not be in 
Netscape branches.  Note also that Moz now crashes sometimes as the memory is 
sucked up.  See talkback ID: TB2554040H

Comment 15

[Dom Incollingo]

Attachment: Mozilla memory usage at 203 meg. after search

As the attachment I am submitting shows, Mozilla memory is still at 203 meg
following this search.

As the attachment indicates, I am using Mozilla build 2002020409.

Please note (from the attachment) that Task Manager shows Mozilla.exe at
203,324K.  I looked at attachment number 68020, and noticed that netscap6.exe
(not Mozilla.exe) is using 35 MB.  This search does not use high memory on
Netscape 6.2.1.  But it does use very high memory on Mozilla.  Something must
have changed since the branch was created for Netscape 6.2.1.

Comment 16

[Brant Gurganus]

In a second tabbed browser, I followed the instructions and mozilla's memory
usage only rose from about 50MB to 60MB.
I am running Windows XP Home, mozilla build 2002021303, 512 MB RAM, 60 GB HD.

Comment 17

[Dom Incollingo]

I reran my original test and a couple others using build 2002021503.  All tests
used Win 2K, 512MB, and non-tabbed browsing.  For each test, I launched the
browser prior to the test, so that the initial memory footprint would start at
the same size for all tests.  Here are the results:

Initial memory footprint: 19.5 MB

Bugzilla search on a forward slash (/):
           Memory footprint after search: 35 MB
           Bugs found: 2,156

Bugzilla search on the word 'mail':
           Memory footprint after search: 53 MB
           Bugs found: 4,890

Bugzilla search on the word 'a':
           Memory footprint after search: 152 MB
           Bugs found: 21,696 

I also ran the same tests using NS 6.2.1, and the memory footprints after
corresponding searches were similar.  In fact, the Mozilla footprint was a bit
smaller after all searches!

The initial problem I reported seems to have been solved.   Please close this
bug if you feel appropriate.  And please let me know if there is anything I can
do to help.   Thanks.

Comment 18

[eberry]

Sorry, but this isn't fixed with yesterday's nightly under W2K.  The issue 
is that showing the bug list in the Search sidebar that is a memory hog, not the 
page rendering itself.  Please test again with a recent nightly build with the 
sidebar showing, and have the search tab open with "Open the Search tab in the 
sidebar when search results are available" checked in the search prefs.  When I 
do this, I get 110+ MB of memory usage with a brand-new profile.  Most of the 
consuming comes after the  page is rendered, which again points to the search 
tab as the culprit.  Hiding the search tab or hiding the sidebar after this 
jump doesn't decrease memory usage.  Note that the same search when the sidebar 
was never shown doesn't result in high memory usage.

Comment 19

[Dom Incollingo]

I set my preferences to 'Open the Search tab in the sidebar when search results
are available', but it is not working.   Either I have a corrupted file
somewhere or there is a bug in build 2002021503.  I will try again on Monday
with another build.  In the meantime, could you send a screenshot (JPG) so I can
see what your browser screen looks like when this problem occurs?  Thanks.

Comment 20

[Ere Maijala (slow)]

I was browsing some bugs changed in the last few days when I noticed this one. I
noticed before that Mozilla became unresponsive for a good while after the page
loading had apparently completed and sidebar was being displayed. Of course I
had to check the memory usage now. 170 megs... Build 20020214 Win2K. 

Comment 21

[James "Kovu" Russell]

*** Bug 129985 has been marked as a duplicate of this bug. ***

Comment 22

[Cees T.]

Is this related to bug 98835?

Comment 23

[Boris Zbarsky [:bzbarsky]]

*** Bug 139447 has been marked as a duplicate of this bug. ***

Comment 24

[John Levon]

This has sent my machine into terminal swap storm a couple of times
necessitating a sysrq-k to kill X ...

Comment 25

[harm]

>
>------- Additional Comment #11 From Asa Dotzler 2002-01-25 20:36 -------
>
>not a leak, just a high memory usage. Over to Layout folks. 

Not a leak ? I don't see the memory beeing reclaimed if I continue browsing
random other sites after I did a bugzilla query and mozilla 'ate' 200 Mbyte.

Comment 26

[harm]

For reference  bugzilla query resulted in:
about 3000 bugs found
about 200 Mbyte memory used
about 250 Kbyte worth of text in the bugzilla result

Comment 27

[Darrel Yurychuk]

Is there any work being done on this?  Has this finally been acknowledged as
confirmed and is reproducible?

Here's my experience for build 2002042706 RC1.0.0:

When running the following bugzilla query:
http://bugzilla.mozilla.org/buglist.cgi?bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&field0-0-0=short_desc&type0-0-0=substring&value0-0-0=/&field0-0-1=status_whiteboard&type0-0-1=substring&value0-0-1=/
mozilla chews up nearly all virtual memory and most of the CPU (from MS WINTOP -
it's all I got).  This even after it appears the query is finished and the
message comes back with the "This list is too long for bugzilla's little mind;
the Next/Prev/First/Last buttons won't appear." message.  Eventually the system
settles down (CPU usage near idle and VM stabilizes), I do a reload of the query
and mozilla crashes (talkback incidents: TB5763283Y and TB5763039M), most likely
from lack of free memory.

I know enough about CGI to say that except for displaying the result, all the
processing happens on the server not the client, therefore the >2700 results
displayed should take up no more resources on the client machine than displaying
a multi-page html doc.  Something very bad is happening with Mozilla in this
case and it's possible it's not restricted to just for queries on the
bugzilla.mozilla.org site.

We resource constrained users (Windows 95b, 133 Pentium, 64 MB RAM, 600 MB HD)
are the ones that always notice these bugs and of course complain the loudest ;-)  

Comment 28

[Andrew Schultz]

I am unable to reproduce this loading a buglist in a local file.  I stuck it on
my localhost http server, and still couldn't reproduce it.

I tried disabling my memory/disk cache and tried pulling the buglist from
bugzilla, and it still happened.

Why is this in Search?

Comment 29

[Andrew Schultz]

linux build 20020504
under gdb, I caught Mozilla while the memory usage was spiking and caught it
with a stack that looked like:
#0  0x40602044 in memcpy () at memcpy:-1
#1  0x4a54b008 in ?? ()   <-- should be LiteralImpl::Create, I think
#2  0x40ef094a in RDFServiceImpl::GetLiteral (this=0x810bde8, aValue=0x45dd4b1a,
aLiteral=0xbffebea0) at nsRDFService.cpp:1215

I put a printf statement in LiteralImpl::Create to see what it was doing.  It
was allocating memory in approximately 1.4MB chunks.  100 times.  It also
allocated a bunch of smaller arrays.  When it was all done, it was using ~170MB.
 I'll attach the output.

updating URL to the one I used.  It returns 1800 bugs.
http://bugzilla.mozilla.org/buglist.cgi?chfield=%5BBug+creation%5D&chfieldfrom=03%2F01%2F2002&chfieldto=03%2F12%2F2002

==> RDF

Comment 30

[Andrew Schultz]

Attachment: memory allocated in LiteralImpl::Create

Comment 31

[Chris Waterson]

This is a known problem with search.

Comment 32

[Andrew Schultz]

search is grabbing the memory for the "HTML response chunk"
(InternetSearchDataSource::ParseHTML).  It looks like it's trying to make a copy
of each individual search hit, but includes a copy of the whole page below as well.

resultItemStart and resultItemEnd seem reasonable, and resultItem is ok.  Then
it does:
const PRUnichar *htmlResponseUni = resultItem.get();

the result is not null-terminated (at resultItemEnd), so when RDF checks strlen,
it gets the whole page.  Right?

The 100 I mentioned in comment 29 is the max number of hits that get saved.

Comment 33

[Andrew Schultz]

Attachment: copy resultItem instead of point at it

Comment 34

[Andrew Schultz]

this should make it into 1.0.1 if not 1.0.

looks like a regression from bug 77460. see comment 77

can I get an review here?

Comment 35

[Suresh Duddi (gone)]

rjc would be the right person to review this (maybe samir if he has taken over
all that from rjc.

Comment 36

[Robert John Churchill]

Comment on attachment 82752 [details] [diff] [review]
copy resultItem instead of point at it

(Shame about having to byte-copy the string. <sigh> Oh well.)

Move the
  free(htmlResponseUni)
up inside of the
  if (htmlResponseUni)
and then
  r=rjc

Comment 37

[Andrew Schultz]

Attachment: patch v2

don't free null pointers!

Comment 38

[Robert John Churchill]

Comment on attachment 84881 [details] [diff] [review]
patch v2

r=rjc

Comment 39

[Dom Incollingo]

Attachment: Sites unrelated to Bugzilla also cause memory usage to skyrocket

There are a few sites unrelated to Bugzilla that cause Mozilla memory usage to
skyrocket.  For example, rendering the Time Warner Road Runner site
http://dialaccess.rr.com/servlet/dr/AccessNumber causes memory usage to
increase from 21.9 meg to 92.2 meg.   (Please see attachment.)	Can someone
confirm whether this high memory usage is related to this bug?	Thanks very
much.

BTW, I'm using Moz build 2002060104 on Win 2K with 512 meg of RAM.

Comment 40

[Andrew Schultz]

Dom: the internet search component is not activated on the rr page, so it's not
this bug.  Please file a new bug.

Comment 41

[Matthias Versen [:Matti]]

isn't this bug 49539 ?

Comment 42

[Andrew Schultz]

bug 49539 started out different than this bug, but morphed into this one after
bug 49539 comment 45.

Comment 43

[Andrew Schultz]

*** Bug 147115 has been marked as a duplicate of this bug. ***

Comment 44

[gavin long]

*** Bug 150581 has been marked as a duplicate of this bug. ***

Comment 45

[eberry]

Andrew, have you requested a sr= for this 2-line patch?  It has an r= and fixes
a 100-megabyte memory hog/leak.  Who is the super-reviewer covering this
component? blaker? jag? alecf? brendan?

Comment 46

[Andrew Schultz]

Samir (bug assignee) is the person to super-review this

Comment 47

[Jonas Jørgensen]

Andrew: No, super-review can only be given by the people listed at
<http://mozilla.org/hacking/reviewers.html>.

Comment 48

[Andrew Schultz]

argh!
looking for sr=? here.  thanks.

Comment 49

[Keyser Sose]

1.0.1 has passed moving milestones.

Comment 50

[Keyser Sose]

Andrew, it might be wise to email some people on the
http://mozilla.org/hacking/reviewers.html list asking for a super review

Comment 51

[jag (Peter Annema)]

Comment on attachment 84881 [details] [diff] [review]
patch v2

The problem here is that |resultItem| is being lied to by the CBufDescriptor it
gets, causing the nsAutoString to not be properly zero terminated, thereby
breaking its interface promise. This causes its use below to "fail".

Since we now end up copying the substring anyway, why not just ditch the
CBufDescriptor and copy into the nsAutoString itself?

Comment 52

[jag (Peter Annema)]

Copy into |resultItem|, that is.

Comment 53

[Andrew Schultz]

Attachment: patch v3

patch to copy into |resultItem|

Comment 54

[jag (Peter Annema)]

Comment on attachment 100366 [details] [diff] [review]
patch v3

sr=jag

Comment 55

[Robert John Churchill]

Comment on attachment 100366 [details] [diff] [review]
patch v3

r=rjc  IFF you add a comment  detailing why "CBufDescriptor" can't be used

Comment 56

[Philip K. Warren]

Is this fix going to get checked in before the freeze for Mozilla 1.2 beta?

Comment 57

[Robert John Churchill]

I just checked this patch in, so marking FIXED.