Enormous memory usage during bug search

RESOLVED FIXED in mozilla1.0.2

Status

SeaMonkey
Search
--
critical
RESOLVED FIXED
16 years ago
10 years ago

People

(Reporter: Dom Incollingo, Assigned: Samir Gehani)

Tracking

({memory-footprint})

Trunk
mozilla1.0.2
memory-footprint

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [Need SR=], URL)

Attachments

(8 attachments, 2 obsolete attachments)

(Reporter)

Description

16 years ago
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:0.9.7+)
Gecko/20020116
BuildID:    2002011604

During a specific bug search on http://bugzilla.mozilla.org/, Mozilla memory
usage skyrocketed to approximately 200 megabytes!

I am running Mozilla build 2002011604 on Windows 2000.  My computer has 512 meg
of memory.

Reproducible: Always
Steps to Reproduce:
1.Launch Mozilla and navigate to http://bugzilla.mozilla.org/
2.Launch Windows Task Manager.   Mozilla is using about 20 meg of memory.
3.Enter a forward slash (/) in the field entitled "Enter a bug # or some search
terms".   (I was searching for a bug whose error message begins with a slash).
4.Press the 'Show' button.
5.Go back to Windows Task Manager and watch Mozilla's memory consumption.
6.Memory rises steadily to about 40 - 50 meg until the browser displays the next
page with the bug list.
7. After the browser displays the next page with the bug list, memory
consumption CONTINUES to rise quickly and steadily.  In a few seconds, Mozilla
is using approximately 200 meg of memory! 
8.Memory usage remains at about 200 meg long after the bug list is displayed.

Actual Results:  Memory usage quickly went from 20 meg to approximately 200 meg.

Expected Results:  Memory usage should have climbed slightly, but not nearly to
200 meg.

I have done the exact same bug search using Netscape 6.2.1 and the Mozilla 0.9.7
monthly milestone build.  In both cases, memory usage climbs to about 40 meg,
and then levels off.   It does not climb higher than 40 meg.  This leak appears
to have been introduced after the 0.9.7 Mozilla build.

Although this situation (a bug search using a /) is uncommon, I believe that
this bug is critical because it is indicative of a serious memory leak somewhere
in browser processing.
(Reporter)

Comment 1

16 years ago
Created attachment 65563 [details]
Screenprint showing memory consumption after bug search
wfm with win2k build 20020117.. (59MB RAM)
(Reporter)

Comment 3

16 years ago
I can re-create this problem using build 2002011803.  Also, I posted a message
on the Mozilla newsgroup, and another user with W2K-SP2 / 512MB (and the latest
nightly build as of this morning) was also able to re-create this problem. 
Perhaps  the problem has something to do with the fact that our PCs have 512 MB
of RAM.
no, my system have also 512BM RAM 
BTW: I can't see your screenshot because I don't have PPT and many other people 
can't see it because they use linux..
(Reporter)

Comment 5

16 years ago
Created attachment 65790 [details]
Here is a JPG screenshot of the memory leak

If you check the netscape.public.mozilla.browser newsgroup for today (1/19/02)
at about 12:05 PM, you will see where I posted this problem.  Someone with the
same machine configuration as mine responded that Mozilla was at 213MB when
they followed the steps listed in this bug.

Comment 6

16 years ago
This is *not* mozilla's fault.  If anything, it /might/ be bugzilla's.

you search probably resulted in a pull of all bugs you can read in the 
database.
[nc4 top output]
Bug List Sat Jan 19 18:48:13 PST 2002 This list is too long for bugzilla's 
little mind; the Next/Prev/First/Last buttons won't appear.
Assignee: asa → justdave
Component: Browser-General → Bugzilla-General
Product: Browser → Bugzilla
QA Contact: doronr → matty
Summary: Enormous Memory Leak During Bug Search → Quick Search for / results in ... list is too long for bugzilla's little mind
Version: other → 2.15
(Reporter)

Comment 7

16 years ago
Created attachment 65796 [details]
JPG screenshot of the same Bugzilla search using Netscape 6.2.1

If this is * Not * the fault of Mozilla, then why does Netscape 6.2.1 use only
40 MB of memory when doing the exact same search?  (As illustrated by the third
attachment)   The search done with NS 6.2.1 used the same Bugzilla processing
and returned the same number of bugs.  The only difference was the version of
the browser.  I have also done the same search using Mozilla 0.9.7 (monthly
milestone),  and it tops off at 50 MB even though Bugzilla returns the same
number of bugs and (presumably) performs the same processing.
per additional information provided by the reporter I concur that he is
complaining about a Mozilla issue, not a Bugzilla one.  The "list is too big for
Bugzilla's little mind" message isn't what he's complaining about.  The amount
of memory used by the browser to process the returned data is what he's
complaining about (specifically, a 160 MB difference in memory usage between NC
6.2.1 and the current Mozilla for the same data).

Moving back to Browser...
Assignee: justdave → asa
Component: Bugzilla-General → Browser-General
Product: Bugzilla → Browser
QA Contact: matty → doronr
Summary: Quick Search for / results in ... list is too long for bugzilla's little mind → Enormous memory leak during bug search
Version: 2.15 → other

Comment 9

16 years ago
Reproduced on build 2002011604, Win2k. Confirming.
Status: UNCONFIRMED → NEW
Ever confirmed: true

Comment 10

16 years ago
this also occurs on Linux(Debian/woody, Gnome 1.4)with build 20020115.  Memory
usage as measured by top(s=0.5) jumps 160M after the bug list finishes loading
in 20-30M increments.  Presumably since the page has finished loading Mozilla
shouldn't be doing anything computationally/memory intensive.

Comment 11

16 years ago
not a leak, just a high memory usage. Over to Layout folks. 
Assignee: asa → karnaze
Component: Browser-General → HTMLTables
QA Contact: doronr → amar
Summary: Enormous memory leak during bug search → Enormous memory usage during bug search

Comment 12

16 years ago
Created attachment 68020 [details]
my mem usage on win2k for same query

for what its worth, my screenshot of my memusage for this query (35 megs).

Comment 13

16 years ago
markling this bug WFM on win2k.  i use up 35 megs (see attachemnt screenshot).
Status: NEW → RESOLVED
Last Resolved: 16 years ago
Resolution: --- → WORKSFORME

Updated

16 years ago
Whiteboard: [awd:tbl]

Comment 14

16 years ago
The steps still reproduce the bug with today's Mozilla trunk build from 
2002-02-04-09 under W2K SP2 with a completely new profile.  It might be related 
to parsing the bug list for the sidebar, and this feature might not be in 
Netscape branches.  Note also that Moz now crashes sometimes as the memory is 
sucked up.  See talkback ID: TB2554040H
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
(Reporter)

Comment 15

16 years ago
Created attachment 68039 [details]
Mozilla memory usage at 203 meg. after search

As the attachment I am submitting shows, Mozilla memory is still at 203 meg
following this search.

As the attachment indicates, I am using Mozilla build 2002020409.

Please note (from the attachment) that Task Manager shows Mozilla.exe at
203,324K.  I looked at attachment number 68020, and noticed that netscap6.exe
(not Mozilla.exe) is using 35 MB.  This search does not use high memory on
Netscape 6.2.1.  But it does use very high memory on Mozilla.  Something must
have changed since the branch was created for Netscape 6.2.1.

Comment 16

16 years ago
In a second tabbed browser, I followed the instructions and mozilla's memory
usage only rose from about 50MB to 60MB.
I am running Windows XP Home, mozilla build 2002021303, 512 MB RAM, 60 GB HD.
(Reporter)

Comment 17

16 years ago
I reran my original test and a couple others using build 2002021503.  All tests
used Win 2K, 512MB, and non-tabbed browsing.  For each test, I launched the
browser prior to the test, so that the initial memory footprint would start at
the same size for all tests.  Here are the results:

Initial memory footprint: 19.5 MB

Bugzilla search on a forward slash (/):
           Memory footprint after search: 35 MB
           Bugs found: 2,156

Bugzilla search on the word 'mail':
           Memory footprint after search: 53 MB
           Bugs found: 4,890

Bugzilla search on the word 'a':
           Memory footprint after search: 152 MB
           Bugs found: 21,696 

I also ran the same tests using NS 6.2.1, and the memory footprints after
corresponding searches were similar.  In fact, the Mozilla footprint was a bit
smaller after all searches!

The initial problem I reported seems to have been solved.   Please close this
bug if you feel appropriate.  And please let me know if there is anything I can
do to help.   Thanks.

Comment 18

16 years ago
Sorry, but this isn't fixed with yesterday's nightly under W2K.  The issue 
is that showing the bug list in the Search sidebar that is a memory hog, not the 
page rendering itself.  Please test again with a recent nightly build with the 
sidebar showing, and have the search tab open with "Open the Search tab in the 
sidebar when search results are available" checked in the search prefs.  When I 
do this, I get 110+ MB of memory usage with a brand-new profile.  Most of the 
consuming comes after the  page is rendered, which again points to the search 
tab as the culprit.  Hiding the search tab or hiding the sidebar after this 
jump doesn't decrease memory usage.  Note that the same search when the sidebar 
was never shown doesn't result in high memory usage.

Updated

16 years ago
Component: HTMLTables → Search
(Reporter)

Comment 19

16 years ago
I set my preferences to 'Open the Search tab in the sidebar when search results
are available', but it is not working.   Either I have a corrupted file
somewhere or there is a bug in build 2002021503.  I will try again on Monday
with another build.  In the meantime, could you send a screenshot (JPG) so I can
see what your browser screen looks like when this problem occurs?  Thanks.

Comment 20

16 years ago
I was browsing some bugs changed in the last few days when I noticed this one. I
noticed before that Mozilla became unresponsive for a good while after the page
loading had apparently completed and sidebar was being displayed. Of course I
had to check the memory usage now. 170 megs... Build 20020214 Win2K. 

Updated

16 years ago
Target Milestone: --- → mozilla1.2

Comment 21

16 years ago
*** Bug 129985 has been marked as a duplicate of this bug. ***

Comment 22

16 years ago
Is this related to bug 98835?

Updated

16 years ago
Keywords: footprint
*** Bug 139447 has been marked as a duplicate of this bug. ***

Comment 24

16 years ago
This has sent my machine into terminal swap storm a couple of times
necessitating a sysrq-k to kill X ...
OS: Windows 2000 → All

Comment 25

16 years ago
>
>------- Additional Comment #11 From Asa Dotzler 2002-01-25 20:36 -------
>
>not a leak, just a high memory usage. Over to Layout folks. 

Not a leak ? I don't see the memory beeing reclaimed if I continue browsing
random other sites after I did a bugzilla query and mozilla 'ate' 200 Mbyte.

Comment 26

16 years ago
For reference  bugzilla query resulted in:
about 3000 bugs found
about 200 Mbyte memory used
about 250 Kbyte worth of text in the bugzilla result

Comment 27

16 years ago
Is there any work being done on this?  Has this finally been acknowledged as
confirmed and is reproducible?

Here's my experience for build 2002042706 RC1.0.0:

When running the following bugzilla query:
http://bugzilla.mozilla.org/buglist.cgi?bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&field0-0-0=short_desc&type0-0-0=substring&value0-0-0=/&field0-0-1=status_whiteboard&type0-0-1=substring&value0-0-1=/
mozilla chews up nearly all virtual memory and most of the CPU (from MS WINTOP -
it's all I got).  This even after it appears the query is finished and the
message comes back with the "This list is too long for bugzilla's little mind;
the Next/Prev/First/Last buttons won't appear." message.  Eventually the system
settles down (CPU usage near idle and VM stabilizes), I do a reload of the query
and mozilla crashes (talkback incidents: TB5763283Y and TB5763039M), most likely
from lack of free memory.

I know enough about CGI to say that except for displaying the result, all the
processing happens on the server not the client, therefore the >2700 results
displayed should take up no more resources on the client machine than displaying
a multi-page html doc.  Something very bad is happening with Mozilla in this
case and it's possible it's not restricted to just for queries on the
bugzilla.mozilla.org site.

We resource constrained users (Windows 95b, 133 Pentium, 64 MB RAM, 600 MB HD)
are the ones that always notice these bugs and of course complain the loudest ;-)  

Comment 28

16 years ago
I am unable to reproduce this loading a buglist in a local file.  I stuck it on
my localhost http server, and still couldn't reproduce it.

I tried disabling my memory/disk cache and tried pulling the buglist from
bugzilla, and it still happened.

Why is this in Search?

Comment 29

16 years ago
linux build 20020504
under gdb, I caught Mozilla while the memory usage was spiking and caught it
with a stack that looked like:
#0  0x40602044 in memcpy () at memcpy:-1
#1  0x4a54b008 in ?? ()   <-- should be LiteralImpl::Create, I think
#2  0x40ef094a in RDFServiceImpl::GetLiteral (this=0x810bde8, aValue=0x45dd4b1a,
aLiteral=0xbffebea0) at nsRDFService.cpp:1215

I put a printf statement in LiteralImpl::Create to see what it was doing.  It
was allocating memory in approximately 1.4MB chunks.  100 times.  It also
allocated a bunch of smaller arrays.  When it was all done, it was using ~170MB.
 I'll attach the output.

updating URL to the one I used.  It returns 1800 bugs.
http://bugzilla.mozilla.org/buglist.cgi?chfield=%5BBug+creation%5D&chfieldfrom=03%2F01%2F2002&chfieldto=03%2F12%2F2002

==> RDF
Assignee: karnaze → waterson
Status: REOPENED → NEW
Component: Search → RDF
QA Contact: amar → tever

Comment 30

16 years ago
Created attachment 82435 [details]
memory allocated in LiteralImpl::Create

Comment 31

16 years ago
This is a known problem with search.
Assignee: waterson → sgehani
Component: RDF → Search
QA Contact: tever → claudius
Whiteboard: [awd:tbl] → DUPEME

Comment 32

16 years ago
search is grabbing the memory for the "HTML response chunk"
(InternetSearchDataSource::ParseHTML).  It looks like it's trying to make a copy
of each individual search hit, but includes a copy of the whole page below as well.

resultItemStart and resultItemEnd seem reasonable, and resultItem is ok.  Then
it does:
const PRUnichar *htmlResponseUni = resultItem.get();

the result is not null-terminated (at resultItemEnd), so when RDF checks strlen,
it gets the whole page.  Right?

The 100 I mentioned in comment 29 is the max number of hits that get saved.

Comment 33

16 years ago
Created attachment 82752 [details] [diff] [review]
copy resultItem instead of point at it

Updated

16 years ago
Keywords: patch, review
Hardware: PC → All
Whiteboard: DUPEME

Comment 34

16 years ago
this should make it into 1.0.1 if not 1.0.

looks like a regression from bug 77460. see comment 77

can I get an review here?
Keywords: mozilla1.0.1
Target Milestone: mozilla1.2alpha → mozilla1.0.1

Comment 35

16 years ago
rjc would be the right person to review this (maybe samir if he has taken over
all that from rjc.
Comment on attachment 82752 [details] [diff] [review]
copy resultItem instead of point at it

(Shame about having to byte-copy the string. <sigh> Oh well.)

Move the
  free(htmlResponseUni)
up inside of the
  if (htmlResponseUni)
and then
  r=rjc
Attachment #82752 - Flags: review+

Comment 37

16 years ago
Created attachment 84881 [details] [diff] [review]
patch v2

don't free null pointers!
Comment on attachment 84881 [details] [diff] [review]
patch v2

r=rjc
Attachment #84881 - Flags: review+

Updated

16 years ago
Attachment #82752 - Attachment is obsolete: true
(Reporter)

Comment 39

16 years ago
Created attachment 85910 [details]
Sites unrelated to Bugzilla also cause memory usage to skyrocket

There are a few sites unrelated to Bugzilla that cause Mozilla memory usage to
skyrocket.  For example, rendering the Time Warner Road Runner site
http://dialaccess.rr.com/servlet/dr/AccessNumber causes memory usage to
increase from 21.9 meg to 92.2 meg.   (Please see attachment.)	Can someone
confirm whether this high memory usage is related to this bug?	Thanks very
much.

BTW, I'm using Moz build 2002060104 on Win 2K with 512 meg of RAM.

Comment 40

16 years ago
Dom: the internet search component is not activated on the rr page, so it's not
this bug.  Please file a new bug.
isn't this bug 49539 ?

Comment 42

16 years ago
bug 49539 started out different than this bug, but morphed into this one after
bug 49539 comment 45.

Updated

16 years ago
Blocks: 92580

Comment 43

16 years ago
*** Bug 147115 has been marked as a duplicate of this bug. ***

Comment 44

16 years ago
*** Bug 150581 has been marked as a duplicate of this bug. ***

Updated

16 years ago
Whiteboard: [Need SR=]

Updated

16 years ago
Keywords: mozilla1.1

Comment 45

16 years ago
Andrew, have you requested a sr= for this 2-line patch?  It has an r= and fixes
a 100-megabyte memory hog/leak.  Who is the super-reviewer covering this
component? blaker? jag? alecf? brendan?

Comment 46

16 years ago
Samir (bug assignee) is the person to super-review this

Comment 47

16 years ago
Andrew: No, super-review can only be given by the people listed at
<http://mozilla.org/hacking/reviewers.html>.

Comment 48

16 years ago
argh!
looking for sr=? here.  thanks.

Updated

16 years ago
Keywords: mozilla1.1 → mozilla1.2

Comment 49

16 years ago
1.0.1 has passed moving milestones.
Target Milestone: mozilla1.0.1 → mozilla1.0.2

Comment 50

16 years ago
Andrew, it might be wise to email some people on the
http://mozilla.org/hacking/reviewers.html list asking for a super review

Comment 51

16 years ago
Comment on attachment 84881 [details] [diff] [review]
patch v2

The problem here is that |resultItem| is being lied to by the CBufDescriptor it
gets, causing the nsAutoString to not be properly zero terminated, thereby
breaking its interface promise. This causes its use below to "fail".

Since we now end up copying the substring anyway, why not just ditch the
CBufDescriptor and copy into the nsAutoString itself?
Attachment #84881 - Flags: needs-work+

Comment 52

16 years ago
Copy into |resultItem|, that is.

Comment 53

16 years ago
Created attachment 100366 [details] [diff] [review]
patch v3

patch to copy into |resultItem|
Attachment #84881 - Attachment is obsolete: true

Comment 54

16 years ago
Comment on attachment 100366 [details] [diff] [review]
patch v3

sr=jag
Attachment #100366 - Flags: superreview+

Updated

16 years ago
Attachment #100366 - Flags: review+
Comment on attachment 100366 [details] [diff] [review]
patch v3

r=rjc  IFF you add a comment  detailing why "CBufDescriptor" can't be used

Comment 56

16 years ago
Is this fix going to get checked in before the freeze for Mozilla 1.2 beta?
I just checked this patch in, so marking FIXED.
Status: NEW → RESOLVED
Last Resolved: 16 years ago16 years ago
Resolution: --- → FIXED
Product: Core → SeaMonkey
You need to log in before you can comment on or make changes to this bug.