329876 - Virtual memory size of Mail&News/Tb increases on each mail viewing, but doesn't decrease, then infinite increase

Status: RESOLVED WORKSFORME

(MailNews Core :: Backend, defect)

Description

[WADA:World Anti-bad-Duping Agency]

Memory usage (Virtual memory size of Mail&News or Thunderbirdin Task Manager) increases on each mail view, but doesn't decrease even when Mail&News window of Seamonkey is closed.
This may cause infinite virtual memory size increase, until shutdown. 

This phenomenon was observed on both Semonkey trunk and Thunderbird trunk.
 - Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9a1)
   Gecko/20060307 SeaMonkey/1.5a
   (Current profile, current account definition,             )
   (LiveHTTPHeaders and Tab Browser Extentions are installed.)
 - Thunderbird version 3 alpha 1 (20060308)
   (New profile, new account definition, no extension, no theme,)
   (No IMAP account, No News account, Dummy POP3 account only,  )  
   (No server communication because dummy POP3 only.            )    

Leak Gauge says : (No leak at global object level)
  Summary:
  Leaked 0 out of 179 DOM Windows
  Leaked 0 out of 214 documents
  Leaked 0 out of 12 docshells
 (checked Seamonkey case only)
Something wrong such that "Eventlistener is not removed for mail related resource"?
> Bug 253688 : memory leak in Mozilla when adding a eventListener
>              from JavaScript and not removing it
Or leak of some minor level objects?
> Bug 262234 : removeChild() doesn't free memory
Or simply "Heap is not freemain'ed until shutdown"? (If so, why not re-used?)
Bug 206502 has relation to this bug's phenomenon?
> Bug 206502 : Checking all IMAP folders for new mail uses lots of memory

[Steps to reproduce]

(0) 'Home Page'  : A small local HTML.
    Startup page : 'Home Page'.
    Open window on startup : Navigator only
    Mail&New's appearance  : Virtical 3 pane layout
    Mail start page(Thunderbird start page) : None
(1) Start Seamonkey or Thunderbird
    => When Seamonkey, Navigator window is opened(single Tab only)
(2) Open Mail&News window (Seamonkey only)
(3) Open an account
(4) Open a mail folder (only 2 text/plain mails are viewed)
   (4-1) View mail No.1 (By up   arrow, or by click)
   (4-2) View mail No.2 (By down arrow, or by click) 
   (4-3) Repeat (4-1) thru (4-2) many times  
(5) Switch to "Local Folders" and view a mail
(6) Repeat (3) thru (5) many times
(7) Close Mail&News window         (Seamonkey only)
(8) Repeat (2) thru (7) many times (Seamonkey only)

[Actual results]

1. {Virtual memory size} increases on each mail viewing.
   (several K bytes to several 10K bytes per a view)
2. {Virtual memory size} doesn't decrease even when Mail&News window is closed.

[Additonal test results]

 Test Case : Displaying large plain/text mail(7MB) by Thunderbird
 (1) View a small  mail in folder-1       : VS = 17MB
 (2) View the long mail in folder-2       : VS = 36MB 
 (3) View a small  mail in folder-1 again : VS is nealy equal to 17MB
 (4) View the long mail in folder-2 again : VS is nealy equal to 36MB 
 (5) View a small  mail in folder-1 again : VS is nealy equal to 17MB
 => Resource not released in this bug is not mail data related resource.
    (Larger VS increase than mail size is phenomenon reported to Bug 266017)

Comment 1

[WADA:World Anti-bad-Duping Agency]

Attachment: Mail folder file (contains two small text/plain mail)

[ Problem recreation procedure with attached mails ]
 1. Create new profile
 2. Create a POP3 account(Dummy server, say "x.y.z". No Global Inbox use)
 3. Disable automatic mail downloading (because "dummy server")
 4. Terminate Thunderbird
 5. Copy the mail folder file in "x.y.z" directry under Mail directry(say TEST)
 6. Start Task Manager (view CPU, Virtual Memory Size)
 7. Start Thunderbird
 8. Click mail folder of "TEST"
 9. Click mail-1 in folder of "TEST", and wait for CPU 0% again
10. Click mail-2 in folder of "TEST", and wait for CPU 0% again
11. Repeat 9 & 10 many times, with watching "Virtual Memory Size" of each step

Comment 2

[Wayne Mery (:wsmwk)]

WADA - fixed by Bug 266679?

Comment 3

[WADA:World Anti-bad-Duping Agency]

(In reply to Comment #2)
> fixed by Bug 266679?

Wayne Mery, thanks for watching this bug.
Bug 266679 was fixed on 2005-09-20, and problem can be recreated by test procedure of comment #1 when my environment, with both latest Seamonkey trunk nightly and latest Thunderbird *Trunk" nightly, and was recreatable even with 20060410 latest trunk Tb nightly build.  
And Bug 266679 is problem of holding of file handle of ".msf", but only one ".msf" is opened in my test case in comment #0.
So I can't think this bug's issue is problem of Bug 266679, although I'm suspecting similar problem is involved, resource is not freed.

Wayne Mery, does your comment posting mean "Problem was not recreated with test case of comment #1 in your environment"?
If yes, what OS do you use? I'm suspecting Win-2K only problem.

Comment 4

[r.navalinskas]

Confirming the same behaviour for WinXP SP2, build 2006091308. Just tried simple "open mail" + "close mail" operation. "Open mail" eats additional 3.5 MB, "close mail" does not release any. Memory usage - as shown in Task Manager.

Comment 5

[WADA:World Anti-bad-Duping Agency]

cc-ing to David to ask what data is required to analyze this problem.

I suspect next instead of real memory leak(not free'ed) due to non-released reference to an object.
 1. An object whose size increases forever(e.g. object holds historical data)
 2. Object is updated(data is added, size increases), and old version is
    destroyed, then old version(smaller size) is free'ed in heap.
 3. Heap is fragmented, and fragmented free areas in heap are not combined.
 4. Then free'ed space in heap can not be re-used by the object.
This is because;
 a.If step 9&10 of scenario in comment #1 is executed just after restart, ten to
   twenty mail switch is sufficient to observe starting of memory increase.
   But, if some other operations are executed before step 9&10, and if virtual
   memory size for mail&news is larger than size just after start-up
   (realtively large heap space is used and freed after use),
   memory size increase on step 9&10 is not observed until many many repeating
   of step 9&10.
 b.During repeating step 9&10, memory size increase sometimes stops a while,
   and starts increasing again after several execution of step 9&10.

David, what data is required to analyze this kind of problem?
Can user(nightly tester) produce required data?

Comment 6

[Wayne Mery (:wsmwk)]

(In reply to comment #0)
>...
> Something wrong such that "Eventlistener is not removed for mail related
> resource"?
> > Bug 253688 : memory leak in Mozilla when adding a eventListener
> >              from JavaScript and not removing it

dveditz comments this may be gone after blocker bug 241518 got fixed.


> Or leak of some minor level objects?
> > Bug 262234 : removeChild() doesn't free memory

goes back to Bug 83635, which doesn't seem to be making any progress.


> Or simply "Heap is not freemain'ed until shutdown"? (If so, why not re-used?)
> Bug 206502 has relation to this bug's phenomenon?
> > Bug 206502 : Checking all IMAP folders for new mail uses lots of memory

... looks like that bug might get closed
 

> [Actual results]
> 
> 1. {Virtual memory size} increases on each mail viewing.
>    (several K bytes to several 10K bytes per a view)
> 2. {Virtual memory size} doesn't decrease even when Mail&News window is closed.
> 
> [Additonal test results]
> 
>  Test Case : Displaying large plain/text mail(7MB) by Thunderbird
>  (1) View a small  mail in folder-1       : VS = 17MB
>  (2) View the long mail in folder-2       : VS = 36MB 
>  (3) View a small  mail in folder-1 again : VS is nealy equal to 17MB
>  (4) View the long mail in folder-2 again : VS is nealy equal to 36MB 
>  (5) View a small  mail in folder-1 again : VS is nealy equal to 17MB
>  => Resource not released in this bug is not mail data related resource.


>     (Larger VS increase than mail size is phenomenon reported to Bug 266017)

*bug 266017 is a good reference*. Note especially bug 266017 comment 8 and bug 266017 comment 10 - David's comments. Unfortunately it doesn't go into detail or list references. 

Some these might be relevant, especially the first 2 (I didn't give them a good read). There may be others.

*bug 116399
*bug 34941
bug 166906
bug 209699
bug 248846

Wada comment #3
> Wayne Comment #2
> > fixed by Bug 266679?
> 
> Wayne Mery, does your comment posting mean "Problem was not recreated with test
> case of comment #1 in your environment"?
> If yes, what OS do you use? I'm suspecting Win-2K only problem.

Sorry for not replying sooner. None of the above - I haven't done any testing. I've got 2k and XP.

(Do you think this is w2k only?)

Comment 7

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #6)
> > [Additonal test results]
> > 
> >  Test Case : Displaying large plain/text mail(7MB) by Thunderbird
> >  (1) View a small  mail in folder-1       : VS = 17MB
> >  (2) View the long mail in folder-2       : VS = 36MB 
> >  (3) View a small  mail in folder-1 again : VS is nearly equal to 17MB
> >  (4) View the long mail in folder-2 again : VS is nearly equal to 36MB 
> >  (5) View a small  mail in folder-1 again : VS is nearly equal to 17MB
> >  => Resource not released in this bug is not mail data related resource.
> >     (Larger VS increase than mail size is phenomenon reported to Bug 266017)

> *bug 266017 is a good reference*.
> Some these might be relevant, ...

No, no. My "Additonal test results" is an evidence of "mail data related problem is NOT involved in this bug" and is saying "memory for mail data itself is successfuly freeed then freemain'ed". So there is no need to care on bugs like Bug 266017 during analysis of this bug. However, we should be careful so that phenomenon like Bug 266017 will not appear in observed data(measured memory size) in test for this bug. Attached test data(text/plain) to comment #1 is for this purpose.

Comment 8

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #6)
> (Do you think this is w2k only?)
According to #3, phenomenon can be observed on Win-XP too.
I can say nothing about Linux or Mac OS X, although I think ;
  If problem relates to JavaScript object only, problem will occur on Linux too.
  But if problem relates to heap compaction, it possibly depends on OS.

Comment 9

[WADA:World Anti-bad-Duping Agency]

(Correction of comment #8)
> According to #3
This should be "According to comment #4". Sorry for spam.

Comment 10

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #4)
> Just tried simple "open mail" + "close mail" operation. "Open mail" eats
> additional 3.5 MB, "close mail" does not release any.

r.navalinskas, is this a result of repeated "open mail"+"close mail" operations? Or result of single "open mail"+"close mail" operations?
Test result of test scenario of comment #1 with test data of comment #1? 

This bug's issue is : "Free'ed heap area looks to be not re-used."
(although memory leak by not-released reference is also suspectable) 
External observation of "Virtual memory size increases but doesn't decrease" by single operation can not be a reason to suspect memory leak problem.
"Virtual memory size increases but doesn't decrease" itself is very normal and usual behavior because heap is used in many internal processes.

Comment 11

[WADA:World Anti-bad-Duping Agency]

Phenomenon of comment #0 was observed with Tb trunk 2007/5/21 build, but apparent virtual memory increase couldn't be observed with trunk 2007/7/14 build.
Behaviour seems to have been drastically changed by progress of Bug 379070, and it looks that different problem from Bug 388353, although it has relation because Bug 388353 is leak problem while mail viewing.

Closing as WORKSFORME, and remove dependency to Bug 388353.