Memory leak up to 1 GB, pop accounts on linux. Databases open?

RESOLVED INCOMPLETE

Status

defect
--
major
RESOLVED INCOMPLETE
7 years ago
6 years ago

People

(Reporter: pablo, Unassigned)

Tracking

({memory-leak})

Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [not lightning])

Attachments

(25 attachments, 10 obsolete attachments)

7.66 KB, image/png
Details
8.30 KB, image/png
Details
10.22 KB, image/png
Details
7.37 KB, image/png
Details
7.29 KB, image/png
Details
6.64 KB, image/png
Details
7.50 KB, image/png
Details
7.47 KB, image/png
Details
7.76 KB, image/png
Details
6.53 KB, image/png
Details
7.22 KB, image/png
Details
5.97 KB, image/png
Details
10.54 KB, image/png
Details
6.57 KB, image/png
Details
5.92 KB, image/png
Details
6.16 KB, image/png
Details
6.34 KB, image/png
Details
39.54 KB, image/png
Details
7.18 KB, image/png
Details
10.23 KB, image/png
Details
11.04 KB, image/png
Details
9.71 KB, image/png
Details
93.30 KB, application/octet-stream
Details
9.77 KB, image/png
Details
9.12 KB, image/png
Details
Reporter

Description

7 years ago
User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0
Build ID: 2012082500

Steps to reproduce:

Hi,

I have four calendars but I believe I have replicated the problem with a single ICS calendar.  What I did was pretty dumb ... I set the refresh cycle to one minute on a Holiday calendar.  It was a mistake however what I found was Thunderbird performed extremely poorly when composing messages (new or replying) and more importantly, its memory continued to grow and grow.  After several days of running Thunderbird/Lightning 24x7, Linux killed the process for consuming too much memory.

My work-around is to set the refresh to 60 minutes (there's no need to refresh a published holiday calendar more than once a day even!  :).  When a calendar refresh happens, message composition is still affected.

I suspect even with a 60 minute refresh, the memory leak is still around, it just takes 60 times longer to become an issue.  :)

Prior to opening the bug, I looked at the existing bugs and didn't find anything current.

If you require additional information, please don't hesitate to ask.  

Thank you for your consideration.

::: Calendar Information :::
o http://www.mozilla.org/projects/calendar/caldata/USHolidays.ics
o Email:  None
o Refresh Calendar:  Every minute
o Check `Cache'

::: Environment :::
o Thunderbird x64, 15.0


Actual results:

Thunderbird consumes too much memory and performance is spotty when composing messages.


Expected results:

No memory leak and possibly, perform well during a calendar refresh.
Reporter

Comment 1

7 years ago
In reading other bugs and in my recent observations with the refresh cycle set to 60 minutes, is it possible that garbage collection wasn't able to run and that's why the process bloated? 

Below is some data collected over a period of time using KDE's `System Monitor' tool.  It's overnight data so it's not representative of my normal workload.  What the sampling shows though is occasional bursts of memory, but it does come down.  Nothing like the memory consumption from the bug report. 

I'll update the bug with my findings.  Especially once the week starts (holiday, ironic!, this Monday so Tuesday will be a good test).

Time    Mem (in M)
-----   ----------
22:33   207           <- initial start
08:16   266
08:19   310
08:21   265
08:31   305
08:54   288
08:01   316
09:19   301
09:36   296
Reporter

Comment 2

7 years ago
I installed some packages so I need to reboot.  Here's some data for nearly a 24h period.  Looks like we do see the memory usage creep upwards:

Time    MB
-----   ---
10:00   260
10:10   318
10:25   240
11:37   307
11:43   264
12:10   257
16:26   276
17:00   408
17:07   366
17:39   471
17:50   382
18:18   358
19:36   367
19:53   380
20:15   401
20:40   386
21:28   398
07:20   412
09:11   423
09:22   429
09:33   438
Reporter

Comment 3

7 years ago
Reporter

Comment 4

7 years ago
Howdy,

I thought I'd take a more methodical approach on tracking memory resident set size (RSS).  Using RRD, I'm now polling (using `ps') the thunderbird's RSS and place the information in an RRD.  I've configured RRD to average five samples per interval (5 minute interval, samples gathered once a minute).

Attached is a graph showing the RSS of Thunderbird.  It's interesting that at night, when I'm not around (22:00 - 8.30), we see the staircase spikes.  Once I'm in, memory usage doesn't spike but we can see we're currently at ~410 MiB.  

I'll continue to run the sampling to gather more information.
Reporter

Comment 5

7 years ago
I'm needing to restart Thunderbird because reading an email sends my CPU skyrocketing.  Have a look at the current memory.
Reporter

Comment 6

7 years ago
I've bumped the priority on this bug to `Major'  I might have a work-around that I'm going to prove out today.  The work-around may only be applicable for certain situations like mine.

I've set my four calendars to synchronize manually.  I believe this is fine because typically Thunderbird is the the source of my calendars (but not always).  I may have to set up a reminder (ironic!) to manually synchronize.  The work-around may not help anyway ...

Let's see what happens ...
Severity: normal → major
OS: All → Linux
Hardware: All → x86_64
Reporter

Comment 8

7 years ago
Switching all calendars to manual synchronization didn't help.  Please see the graph showing memory usage with and without lightning enabled.
Pablo, does one of your graphs show composite of refresh=1min, 60 min, no calendar (tbird only)?
Reporter

Comment 10

7 years ago
Hi Wayne,

I'm not sure I can create a composite graph with the different options requested because each would have a different x-axis.

I'll attach the latest graph I have for 60-minute refresh showing gradual memory grown until I restarted it this morning.

As I have a smart device, I could also run for a while with Calendaring disable and provide a graph with that data.  I'll start that now.
Reporter

Comment 12

7 years ago
Well, this is interesting.  I could have sworn disabling calendar had memory well under 230 MiB.  The new graph (to be attached) proves otherwise.  Part of the problem is since I first encountered the issue, I've continued to upgrade (my box, tbird, add-ons).

I'm going to disable all my other add-ons and see what gives.

Thank you for the suggestion Wayne.
Reporter

Comment 14

7 years ago
Switching all add-ons off, it appeared I was still getting a memory leak.  But, I think I might be on to something that's unique to my environment.

First off, we have this issue where Tbird is leaking memory.  Second, when a recent update to an add-on caused Tbird to crash (firetray), I opened a ticket with those developers.  They recommended that I replicate the issue in a VM.  I tried and I couldn't.  Aha!

So, my next test is to migrate my email over to the VM and once it's set up, copy the entire contents back to my desktop.

I alluded to my environment being somewhat unique.  It's unique in not such a good way.  :)  

A few months ago, I switched to using Google Applications.  During the transition period, I had Tbird pulling from my old email server and from the gmail server.  Once I was satisfied no more email was flowing through my old email server, I made changes to my Tbird Account settings.  I don't recall the exact details but default email accounts were set for many folders to the wrong email account.  Strange.

Soooooooooooooo, I'm wondering if my fiddling with accounts (all of it done via the user interface - that is, no hacking of files), caused Tbird to hit some edge-case, not frequently exercised.
Reporter

Comment 15

7 years ago
I thought I'd write an update ...

After upgrading one of my add-on's recently, Thunderbird would fail to start.  I worked around the issue and submitted a bug report to the developers.  They kindly suggested I attempt to emulate the issue in a VM and send them the VM.  What I found was I couldn't.  Hmmm!

To make a long story short, I ended up re-creating my Thunderbird environment.  I created new accounts and copied over extensions, mail folders, etc. from the old environment.  

As of yesterday afternoon, I'm now running a new-ish tbird installation.  I'm monitoring memory growth so let's see what happens in the next 24-48h.
Have you, can you, preserve the failing profile?
Reporter

Comment 17

7 years ago
Hi Wayne,

I have a full backup of ~/.thunderbird 

I just restarted tbird because the Global Indexing seemed to run and run and run.  The clock restarts!  :)
Reporter

Comment 18

7 years ago
I'm beginning to think that I need to re-run my test with all add-on's disabled for an extended period of time.  The attached .png shows that memory was very stable through out the night.  During the, I am receiving emails.

Is it possible that one of my add-on's is causing a memory leak?  Hmmm.

At the moment, I'm inclined to let the current execution run to see what happens.  Will my memory use level off?  Or will Linux eventually kill it ...
Reporter

Comment 19

7 years ago
Posted image 32.5 hours ...
Reporter

Comment 20

7 years ago
Running gkrellm (and top), I can see tbird occasionally burst writes while composing an email.  I'm stopping this experiment and I'll revert to testing without
Reporter

Comment 21

7 years ago
I did an in-place upgrade from openSUSE 12.1 to 12.2 and it subtly wreaked havoc so last weekend, I did a fresh install of openSUSE 12.2

During this execution, I noticed periods where thunderbird would busily write for bursts of three to five seconds on my SSD (obviously it'd be longer on an HDD).  

I rummaged around the web and set the compact threshold from (I believe it's the default of) 20 MB to the minimum of 1 MB.  My thinking is I'd rather write /less/, more frequently than /more/, less frequently.  :)

Attached is the graph starting roughly on Oct 16 midnight until this writing.

I'm curious whether I'll plateau or continue to use more memory.  I do have some large folders which at times, I search.
I don't believe that theory net you an overall improvement. But more to the point I dont believe the memory issue is tightly coupled to compact.  And the test I'd rather see to confirm that is you disable compact complete rather than tweak the size
Reporter

Comment 23

7 years ago
Hi Wayne,

I'm more than happy to disable compacting for testing purposes.  Compacting has now been disabled.  :)

Should I let the current run, run or restart Thunderbird?

Thx!
No need to restart
Reporter

Comment 25

7 years ago
Compacting ruled out ... :) 

My next test is to enable compacting and see if I can disable many of my add-ons while still being functional at work.
Hi, is this an problem with Lightning or not? In the latter case I'd suggest to move the bug into a Product/Component where it might be noticed by Thunderbird or Mozilla Toolkit developers. Did you use a clean Thunderbird profile using only calendaring features but no email, chat, other extension or anything else?
Reporter

Comment 27

7 years ago
Hi Stefan,

I haven't tried just a clean install without email or anything else.  I'll try that in a separate Unix account on the box.

Also, I know realize that the problem seems to take 24-48h to manifest itself and some of my earlier tests were false-positives.  At the moment, I'm running tbird with many of my add-on's disabled as well as Lightning.  

Part of my problem in testing is I use tbird at work so removing too many add-ons makes it less convenient to use ... :) ... but I should suck it up!  

Thanks.
Reporter

Comment 28

7 years ago
After 45 hours, the graph shows Thunderbird 16.x's memory use inching upward.  Is this expected or should, say, after five hours, Thunderbird should have settled in its memory use?  

This test is with all add-ons disabled and using the default theme.  I did not try safe-mode.

I think I can now clearly say it's not a Lightning issue.  I'm more than happy to re-classify the ticket.

What I'd like is some direction on how to proceed.  I have some flexibility with creating a bogus email account on my domain and setting up tbird in a VM (for potential remote access to debug?) ... or simply set up something on my Host.

Please advise.
Reporter

Comment 29

7 years ago
Unless I hear otherwise, I'm going to start with a fresh install of Thunderbird under a different user.  I'll set up a bogus email account so it can check email once a minute on one account (although I have six)

Please let me know whether I should do something different.

Thx!
Reporter

Comment 30

7 years ago
I've created a new user on my machine and six bogus email addresses in one of my domains.  In a wee script, I am emailing the bogus addresses /etc/passwd, once every six minutes.  All the while, I'm capturing memory information from this new instance.

tbird is very vanilla.  Aside from creating the six user accounts, I've disabled `When a new message arrive' both `Show an alert' and `Play a sound'

Let's see what happens after 48h.
Reporter

Comment 31

7 years ago
Around 3.30 am this morning, I exceeded how many emails I could send out for my IP.  Strange as it wasn't all that many ... thanks Google Apps!  :) .... I'll have to set up a local mail server for my testing.
Reporter

Comment 32

7 years ago
I have a VM set up with tbird and I'm using my rickety firewall box as an email server for my six, @example.com email addresses.  I just kicked off the test where I'm POP'ing once a minute and receiving zero to two email messages, per account.
Reporter

Comment 33

7 years ago
I'm attaching a checkpoint of my mock up test environment.  

Is this memory creep expected?
Reporter

Updated

7 years ago
Component: Provider: ICS/WebDAV → General
Product: Calendar → Thunderbird
Version: Lightning 1.7 → 16
a change of only 60k over 2 days is not significant. 

do you see the performance issue with compose while in *safe* mode?
Reporter

Comment 35

7 years ago
Hi Wayne,

The only slowness I notice with my existing environment (not the rig environment as I'm not using it for anything but to attempt to replicate the issue) during compose is when tbird is flushing to disk.

What do you suggest should be my next step in the VM?  Add all the folder structures from my existing environment or try to make the accounts as close to identical as possible.  For example, I use davmail for one of my existing accounts whereas the VM, everything is POP3.

Cheers,
-pablo
as far as slowness, what is size of popdate.dat for your accounts?
and what is autosave interval setting and interval for checking for new messages?

as far as memory - if comment 28 and comment 29 are both with calendar disabled, then I would say there is nothing to investigate in thunderbird - an increase of 100MB or so, over a 1-2 day period is nominal
Keywords: mlk
Summary: Memory leak → Memory leak up to 1gb
Reporter

Comment 37

7 years ago
Hi Wayne,

You tricked me, there's no `popdate' but there is `popstate' :)

$ find . -iname pop'*'.dat | xargs ls -lah
-rw------- 1 pablo users 5.3K Nov  5 16:21 ./hp27.hostpapa.com/popstate.dat
-rw------- 1 pablo users  315 Nov  5 16:22 ./localhost/popstate.dat
-rw------- 1 pablo users   61 Nov  5 16:22 ./mail.wakenet.ca/popstate.dat
-rw------- 1 pablo users   61 Nov  5 16:21 ./pop.googlemail-1.com/popstate.dat
-rw------- 1 pablo users   61 Nov  5 16:22 ./pop.googlemail-2.com/popstate.dat
-rw------- 1 pablo users   61 Nov  5 16:22 ./pop.googlemail.com/popstate.dat

As far as memory, yes, comment 29 is showing a marked increase in memory.  Recall that the original issue is after several days, I hit 1 GiB.  I could go back to that test.

At the moment, I have tbird running in a VM, POP'ing email from my local box.  I have safe-mode enabled.
Reporter

Comment 38

7 years ago
After a slight ... :) ... sabbatical, I'm trying to figure out this issue again.

At the moment, I've disabled all add-on's (I'm in safe mode).  I believe it takes at least four days (24h x 4) before the problem manifests itself.

I started the current test at 13:40 local time.
Reporter

Comment 39

7 years ago
48h later; safe-mode; I'll continue for at least another 48h
Reporter

Comment 40

7 years ago
Posted image 4d, 3h-ish
4 days, 3 hours-ish ... have we reached steady-state or will we continue to consume memory?

Comment 41

7 years ago
As this started as a Thunderbird/Lightning memory leak I'll describe what I am getting, I'm running WinXP 32 bit with Tbird 17 and Lightning 1.9 IMAP mail and ical remote calanders. I also saw the same in TBird 15.1 but upgraded to see if that would fix the problem. If I disable the Lightning extension Tbird uses 130 meg of memory just loading the Lightning plugin with 3 calendars configured takes that up to 616Meg. With the calendars 'viewed' (checked in the calendar pane) Tbird becomes so unresponsive that it takes between 5 and 10 minutes to draw the UI from start. With 1 calendar viewed the responsiveness is OK. When unresponsive the processor usage of Tbird is pretty much constantly 100%
Reporter

Comment 42

7 years ago
Hi Tim,

I believe your issue may be different because you're CPU bound.  The key to flushing out the issue(s) is to start thunderbird in safe-mode and see if the problem manifests itself.

It's a bit of a pain because you'll have to use other means to rely on your calendar.  Luckily for me, I can use my Smart Device.

If the problem doesn't manifest in safe-mode, I'd run a test where you disable all add-ons except for Lightning.  Either confirm the problem is the add-on or not, by enabling additional add-ons.

One thing to keep in mind is to not become impatient (like me!  :) and let your tests run long enough, to ensure you've either confirmed or not confirmed an issue.  Also, the issue may not be related to time either, it could be based on, for example, the number of emails processed per day.  In my case, I don't get as many emails on the weekend as I do during the week, with the holidays, etc.  All things to consider.

Cheers!
Reporter

Comment 43

7 years ago
Posted image 5d, ~4h - ~610 MiB
During the weekend, I don't get much email but during non-holiday, work days, I do.  The graph shows how today, Monday, memory jumped from roughly 500 MiB (AM) to 610 MiB (now).

I may let things run for another day ... tomorrow is a holiday so I don't expect much memory growth.  

What I've noticed though is tbird is more laggy (I'm running gkrellem and I can see CPU spikes) when I'm reading or composing new messages.

Can we definitely say we have a memory leak with tbird?  If so, I'd like to move to the next step.  

Dev help!  :)
Reporter

Comment 44

7 years ago
I'm on tbird 17.0 - updating the bug accordingly.
Version: 16 → 17
Reporter

Comment 45

7 years ago
I'm sorry about the frequent update.  I believe this is a P2 and while not a blocker, I'm marking this issue as "critical"  ... the work-around is, obviously, to restart tbird but I'd suggest memory leaks are "bad" [tm].

Please re-classify if you see fit.
Severity: major → critical
Priority: -- → P2
Pablo, did this start in version 15, or did it also happen with version 14?  Can you determine whether the majority of the increase is related to a specific function, for example creating new messages or checking for new messages.

priority setting is reserved for developers. importance=major is already appropriate. visit https://bugzilla.mozilla.org/page.cgi?id=fields.html#importance to see why critical is not
Severity: critical → major
Priority: P2 → --
Reporter

Comment 47

7 years ago
Hi Wayne,

Thank you for resetting the priority.  I don't have sufficient scope to set it properly so I swagged.  I got it wrong.  Ooopsie.  :)

I can't recall exactly what version of tbird the issue first cropped up because I like to keep my machine up to date with patches.  I usually take the lazy man's approach and simply reboot it rather than finding which binaries require a restart.

It's hard to say which activity might be the culprit.  Today being the Dec 31st, I didn't get as much email as regular work days.  Possibly 75 or so messages.  I sent a paltry 36 mail messages today.

I have six email accounts.  All but one are POP3S and one uses POP3/davmail.  Of the six, five check for new email once a minute and one checks every two minutes.

As tomorrow is a holiday, I'll continue to leave tbird run.  Its Virtual Size is nearly 2,000 MiB.  I don't know whether there's any value though so if you say stop, I will because it'd be ideal for me to re-enable my add-on's and lightning etc.

I'm committed to help find the root-cause of this issue so if I need to run in some sort of debug mode, I'm more than happy to do so.

Have a safe and Happy New Year and as always, thank you.

-pablo
Reporter

Comment 48

7 years ago
Posted image 6d, 5:40h: ~640 MiB
As today was a holiday, not many emails flowed in and I sent (as of this writing) two messages.

I'm halting the current test until I'm given direction on how to proceed to gather more data.
(first reported at TB15)

Pablo, did you see this problem in TB14?

some thoughts
- there is a compose issue where we see a loss of 2MB per compose
- your memory increase sounds like something else
- sometimes issues on linux have been determined to NOT be caused by thunderbird
- given that you run lightning, and this is a thunderbird bug report, it's somewhat important that all testing results be without lightning
-  attachment 696905 [details] is interesting - it shows increase of 100MB - 200MB within a few hours

What are the sizes of your largest active folders' .msf files in your thunderbird profile of six accounts?  How many are gmail accounts?
Flags: needinfo?(pablo)
Version: 17 → 15
Reporter

Comment 50

6 years ago
Hi Wayne,

Unfortunately I don't know which version of TB this became a problem.  Part of the reason is depending on the Distro's stability, there may be several patches which are easier done by rebooting.  I first noticed the problem last summer when I went on vacation and the kernel killed tbird due to memory use.

Regarding the comment "given that you run lightning, and this is a thunderbird bug report, it's somewhat important that all testing results be without lightning", recall that my last test was running TB in safe-mode - see comment 38.

As for the size of my six account's .msf files, they're all very small - more shortly:

1. Gmail #1 - 27K
2. Gamil #2 - 13K
3. Account #3 - 4K
4. Account #4 - davmail - 2.3K
5. Account #5 - 2.6K
6. Gmail #3 - 21K

My set up is such that I have many email filters which shunt my email from each account to other folders.  These folders all live under `Local Folders'  I have a whack of sub-folders but within the set, I access a small subset:  possibly 10 or so.  One of the largest is an .msf size of 38K.

Additionally, I have a `Received.msf' where I copy /all/ incoming email.  It is currently sized at 12M - more on this shortly!  :)  There's a companion `Sent.msf' whose sized at 2.2M.

I have an `archive' folder which I occasionally access where I have all email sent by me, by year, for 2006 - 2012 (and soon 2012).  I also have corresponding `receive' sub-folders by year as well.

What I do is once a month, peel off a month from the respective `Sent/Receive' folders and move the emails to the respective year archive.  For example, March 1st, I'll move all of January `sent' and `receive' to a new, respective, `2013' archive.

On the occasion, I might search year `2012' `send' or `receive'  It's rare that I ever go back to `2011' 

The `receive' 2012.msf is 23M and the `send' 2012.msf is 5.4 M.

You didn't ask about the corresponding root files (e.g. without the .msf extension).  I don't know if it's material but for send/receive archives, they can be quite big.  The 2012 sent is 287M (okay, that's not super big) and the 2012 receive is 1,006M

I hope the above isn't too confusing.  If need be, I can screen scrape and send attachments.

Cheers,
-pablo
Flags: needinfo?(pablo)

Updated

6 years ago
Depends on: 480843

Comment 51

6 years ago
Hi Pablo,

Maybe can be interesting check the size of others data files and number of items, as calendars databases and address book files.

Some time ago I had a similar problem, same symptom (more than 1GB of memory, and running very slow) and the cause, in my case, was that my main address book was very very large (if I remember well, more than 30,000 contacts!) because a buggy extension duplicate the contacts several times. The size of the file was not so large to that memory consumption but seems that when Thunderbird process the address book for thinks like address autocomplete, it takes a lot of more space for it in memory, increasing every time the compose window was opened.

Another try is to rebuild the index of bigger mail folders (repair the folder), maybe something corrupted?

The cause in your case seem different, but give it a try.

Hope it can help.
Reporter

Comment 52

6 years ago
Hi,

Thank you for the information ... my address book looks in order.  I do have many contacts but luckily there are no duplicates.  :)

I'll have a look on whether deleting the indexes will lose any information I have per message (like its tag).

Cheers!
-pablo
(In reply to Pablo from comment #48)
> Created attachment 696905 [details]
> 6d, 5:40h:  ~640 MiB

stupid question - why are all your charts and file sizes labeled in "K" ?
Flags: needinfo?(pablo)
Reporter

Comment 54

6 years ago
Hi Wayne,

I'm not quite sure what you're asking.  I'm going to assume when you wrote "file sizes", you mean "memory size"?  I'm using RRD to create the graphs.  

Below is a sample invocation of RRD to generate the graph:

rrdtool graph graph.png DEF:memory=tbird.rrd:rss:AVERAGE LINE1:memory#ff0000:'Thunderbird 17.x (tbird-test)' -t 'Memory Resident Set Size' -v 'Bytes' --units=si --start '19:56 12/26/12'
Flags: needinfo?(pablo)
(In reply to Tim from comment #41)
> As this started as a Thunderbird/Lightning memory leak I'll describe what I
> am getting, I'm running WinXP 32 bit with Tbird 17 and Lightning 1.9 IMAP
> mail and ical remote calanders. I also saw the same in TBird 15.1 but

For the record, Tim discovered "had corrupted the ics file and consequently it had become several hundred times the size that it should have been."
Pablo, is this reproducing with version 17, or version 22 beta?

xref bug 844937 and bug 796989 (both Mac)
Flags: needinfo?(pablo)
Reporter

Comment 57

6 years ago
Hi Wayne,

I'm currently on version 17.0.6 and two days ago (June 11th), I thought I'd re-test and unfortunately the problem remains.  As we had shown the problem exists without add-ons, I ran this test with the add-ons.  It's a bit trying to run the tests mid-week without add-ons because I make use of them.  :\

See the latest graph.png attachment (after this post).

I just downloaded v22, beta 1.  I'll give it a shot.
Reporter

Comment 58

6 years ago
Posted image June 11th, 2013
Reporter

Comment 59

6 years ago
Hi Wayne,

I currently have 64-bit t-bird and rummaging around the Mozilla site, I only found 32-bit beta versions.  Reading some of the online pages, they indicate one needs 32-bit plug-ins for 32-bit tbird.  

Can you provide a link where I can get my hands on a beta, 64-bit version?
my understanding is all the linux offerings at https://ftp.mozilla.org/pub/mozilla.org/thunderbird are for 64bit
Reporter

Comment 61

6 years ago
Hi Wayne,

There are two builds listed for 22:  build 1 and build 2.  I am testing with build 2.  I don't know if that matters but if I should test with build 1, let me know.  Otherwise, the test has started!  I'll let you know how it goes ... in 24 to 48h.
Flags: needinfo?(pablo)
Reporter

Updated

6 years ago
Summary: Memory leak up to 1gb → Memory leak up to 1 GB
Reporter

Comment 63

6 years ago
I see my previous graphs, why things were confusing.  I was pushing into RRD kilobytes rather than bytes.  This is why the y-axis wasn't labeled properly - my error.  The data is still correct, just the label is off on the older graphs.  :\

I've corrected the logic in my code for subsequent graphs.
Reporter

Comment 64

6 years ago
Well, betas are called betas for a reason.  Tbird crashed when I attempted to sort by "Subject" .. I won't do that again ... :) ... when I get a chance, I'll see if there's a bug reported for this issue.
Reporter

Comment 65

6 years ago
Reporter

Comment 66

6 years ago
Tbird 22 just crashed.  We can see in comment #65 memory is trending up.
Reporter

Comment 67

6 years ago
Way back when this issue first arose, I had gone on our yearly vacation and came back to Thunderbird being killed for excessive memory consumption.

This year, before going on vacation, I decided to ratchet back how often I check for new email from one minute to 15 minutes.  Upon my return, Thunderbird was still running.  Hmmm!!!

I've going to run a test where I run Tbird (17.0.7) with two minute frequency checks.  Let's see what happens.
Reporter

Comment 68

6 years ago
Bloating
(In reply to Pablo from comment #68)
> Created attachment 778428 [details]
> 17.0.7/64-bit:  2 minute new-mail frequency checking
> 
> Bloating

Were you using Thunderburd at the time - or is system unattended in the 4 day period?

summary - first two days nothing happens - 200MB growth over 2 days. Then 200MB growth in 2 hours immediately followed by 200MB more in the next ~8 hours.
Flags: needinfo?(pablo)
Reporter

Comment 70

6 years ago
Hi Wayne,

During the entire period, I was using Thunderbird.  Its my only mail client and I use it for work.  At times I get more email than other times (due to say a hot production issue).
Flags: needinfo?(pablo)
Reporter

Comment 71

6 years ago
I'm currently testing with a five minute new-mail frequency check.  It'll be somewhat invalid because I will be away this weekend and the last graph covered an entire work week.
Going forward, for consistency, I think you should stick to doing only one version - only version 17 or only beta (currently version 23).  I think using version 23 will be more helpful if you can use it.

Please permanently turn on msgdb:1,timestamp   log setting per https://wiki.mozilla.org/MailNews:Logging and attach log file to bug when providing charts/info.

Also please post help | troubleshooting information
and attach file with results from help | troubleshooting | about:memory | measure and save
Reporter

Comment 74

6 years ago
10-4 on enabling logging (done!) as well as sticking with one version (17.0.7 - I realize it's not ideal but I need stability for work).

I've also reset new-mail frequency checking back to what I had:  one minute.

Do you need an `about:memory' baseline?  If not, no need to answer and I'll assume no - save you some keystrokes.

As always, thank you for your help Wayne. I appreciate you sticking with me on this.
(In reply to Pablo from comment #74)
> 10-4 on enabling logging (done!) as well as sticking with one version
> (17.0.7 - I realize it's not ideal but I need stability for work).

what has been unstable with beta?
actually, I forgot that version 23 is rather essential to getting good information from about:memory


> I've also reset new-mail frequency checking back to what I had:  one minute.
> 
> Do you need an `about:memory' baseline?  If not, no need to answer and I'll
> assume no - save you some keystrokes.

not really
Reporter

Comment 76

6 years ago
(In reply to Wayne Mery (:wsmwk) from comment #75)
> (In reply to Pablo from comment #74)
> > 10-4 on enabling logging (done!) as well as sticking with one version
> > (17.0.7 - I realize it's not ideal but I need stability for work).
> 
> what has been unstable with beta?

When I'd sort a folder with over 25,000 messages by Subject, the beta would crash.

> actually, I forgot that version 23 is rather essential to getting good
> information from about:memory
> 

Well, there it is .... I'll see about trying 23.
(In reply to Pablo from comment #76)
> (In reply to Wayne Mery (:wsmwk) from comment #75)
> > (In reply to Pablo from comment #74)
> > > 10-4 on enabling logging (done!) as well as sticking with one version
> > > (17.0.7 - I realize it's not ideal but I need stability for work).
> > 
> > what has been unstable with beta?
> 
> When I'd sort a folder with over 25,000 messages by Subject, the beta would
> crash.

please paste the bp-XXXXX crash ID here in the bug.
see https://support.mozillamessaging.com/en-US/kb/mozilla-crash-reporter#w_viewing-crash-reports
Reporter

Comment 78

6 years ago
Checkpoint:  Roughly 49 hours.  The next 24-48 hours will be telling.

Let me know if need any information now on memory usage as reported by Thunderbird.
Reporter

Comment 79

6 years ago
I'll upload memory information shortly ... I'm going on vacation next week ...
Reporter

Comment 80

6 years ago
Reporter

Comment 81

6 years ago
Posted file Trace file (obsolete) —
Reporter

Comment 82

6 years ago
Hi Wayne,

I'll be on vacation next week .. please see the uploaded files:  a memory report as well as the trace file you requested.
(In reply to Pablo from comment #81)
> Created attachment 781908 [details]
> Trace file
unfortunately it's all binary zero

not sure what to make of the memory reports.
Reporter

Comment 84

6 years ago
It seems the o(In reply to Wayne Mery (:wsmwk) from comment #83)
> (In reply to Pablo from comment #81)
> > Created attachment 781908 [details]
> > Trace file
> unfortunately it's all binary zero
> 
> not sure what to make of the memory reports.

It seems there are some binary zeros but it's not all binary zeros.  You can clean it up with:

 tr -d '[\000]' < tbird_mem.log > tbird_mem_clean.log

I am back from vacation and I'll quickly upload some new information you may find interesting.
Reporter

Comment 85

6 years ago
I went on vacation Friday afternoon (July 26rd Eastern Time).   When I left, I switched new email-checking to 15 minute intervals rather than my preferred one minute checking.

The graph shows memory consumption slowing after switching to 15 minute checking.  It's possible because I wasn't using tbird while on vacation, that that is why memory consumption did not increase.
Reporter

Comment 86

6 years ago
Posted file Latest memory-report.json.gz (obsolete) —
Reporter

Comment 87

6 years ago
Posted file tbird_mem_log.gz-aa (obsolete) —
The compressed trace file is too large to attach so I've "split" the file into chunks.  To reconstitute the file, please download all the pieces and "cat tbird_mem*-a* > tbird_mem.log.gz" then "gunzip tbird_mem.log.gz"
Reporter

Comment 88

6 years ago
Posted file tbird_mem_log.gz-ab (obsolete) —
Reporter

Comment 89

6 years ago
Posted file tbird_mem_log.gz-ac (obsolete) —
Reporter

Comment 90

6 years ago
Posted file tbird_mem_log.gz-ad (obsolete) —
Reporter

Comment 91

6 years ago
Posted file tbird_mem_log.gz-ae (obsolete) —
Reporter

Comment 92

6 years ago
Posted file tbird_mem_log.gz-af (obsolete) —
Reporter

Comment 93

6 years ago
Posted file tbird_mem_log.gz-ag (obsolete) —
Reporter

Comment 94

6 years ago
Posted file tbird_mem_log.gz-ah (obsolete) —
This is the last piece.
Reporter

Comment 95

6 years ago
I have a whack of pending updates I need to apply to my O/S.  I regenerated the RRD - see attached.  What appears to be the case is the "New Mail" checking frequency is unrelated to memory consumption.  We can see memory climb today after relatively steady-state while on vacation.
Reporter

Updated

6 years ago
Attachment #784966 - Attachment is obsolete: true
Reporter

Updated

6 years ago
Attachment #784967 - Attachment is obsolete: true
Reporter

Updated

6 years ago
Attachment #781908 - Attachment is obsolete: true
Reporter

Updated

6 years ago
Attachment #784958 - Attachment is obsolete: true
Reporter

Updated

6 years ago
Attachment #784968 - Attachment is obsolete: true
Reporter

Updated

6 years ago
Attachment #784970 - Attachment is obsolete: true
Reporter

Updated

6 years ago
Attachment #784971 - Attachment is obsolete: true
Reporter

Updated

6 years ago
Attachment #784972 - Attachment is obsolete: true
Reporter

Updated

6 years ago
Attachment #784973 - Attachment is obsolete: true
Reporter

Updated

6 years ago
Attachment #784974 - Attachment is obsolete: true
Reporter

Comment 96

6 years ago
As I don't see any traction on this bug for nearly three months, I've marked it so `(never email me about this bug)'

My work-around:  restart Thunderbird every day or so ...

Eventually my MUA of choice will become stable and I'll swing back to it.
(In reply to Pablo from comment #96)
> As I don't see any traction on this bug for nearly three months, I've marked
> it so `(never email me about this bug)'

frustration stinks. Unfortunate that you "unsubscribe",  because 
a) most bug comments are yours, so you're not being spammed
b) you won't get notified when the bug does make progress

To level set here, in case you return:
1. We still need a good msgdb log
2. please, no more screen shots of memory usage :) - just text with couple numbers will do.
3. checking accounts for new messages every minute is reminiscent of a past fixed pop bug. Please change it to 10 minutes for every account, and tell us your results


(In reply to Pablo from comment #94)
> Created attachment 784974 [details]
> tbird_mem_log.gz-ah
> 
> This is the last piece.

I couldn't extract this file :(
Component: General → Backend
Flags: needinfo?(pablo)
Product: Thunderbird → MailNews Core
Summary: Memory leak up to 1 GB → Memory leak up to 1 GB, pop accounts on linux. Databases open?
Whiteboard: [closeme 2013-11-01][not lightning]
Version: 15 → 23
Reporter

Comment 99

6 years ago
(In reply to Wayne Mery (:wsmwk) from comment #97)
> (In reply to Pablo from comment #96)
> > As I don't see any traction on this bug for nearly three months, I've marked
> > it so `(never email me about this bug)'
> 
> frustration stinks. Unfortunate that you "unsubscribe",  because 
> a) most bug comments are yours, so you're not being spammed

You misinterpret why I marked `(never email me about this bug)' ... I'd have closed it had I had the opportunity.

> b) you won't get notified when the bug does make progress

Given nearly three months of inactivity, I think it's fair to say this bug is not a medium priority issue for you.  I understand OSS and I understand people contribute where and when they can.

> 2. please, no more screen shots of memory usage :) - just text with couple
> numbers will do.

Too bad you wait until now to say the above.  Also, the graphs are not screen shots.  They're RRD graphs.  Text numbers are so ... 1980's.

> 3. checking accounts for new messages every minute is reminiscent of a past
> fixed pop bug. 

The above is an incorrect assumption, my work style is to check email every minute.  If you suspect Thunderbird can't handle one minute polling, perhaps you should raise it the issue with other people within development.  This way it's either addressed or the option is not available.

> > 
> > This is the last piece.
> 
> I couldn't extract this file :(

... and three months later, I'm informed.  Feh.  I've long since removed the files from my computer.  You do realize that this means I'd have to re-gather all the information eh?  *sigh*

Wayne, I realize I sound ungrateful.  I'm not.  Thank you for your help.  It's too bad this issue fell off your radar.  I don't mind restarting Thunderbird once a day ... tbird is stop-gap for me.
Several reasons could be listed as to why this bug hasn't gotten much attention (that yours has gotten any attention frankly is better than average), but the most pertinent is that we don't have actionable material or anything promising. I reread almost the entire bug this morning and last night, and what I come up with for the path forward is comment 97. If you are unable to do any of these things then there is no hope for your specific case.


(In reply to Pablo from comment #99)
> (In reply to Wayne Mery (:wsmwk) from comment #97)
> > (In reply to Pablo from comment #96)
> > > As I don't see any traction on this bug for nearly three months, I've marked
> > > it so `(never email me about this bug)'
> > 
> > frustration stinks. Unfortunate that you "unsubscribe",  because 
> > a) most bug comments are yours, so you're not being spammed
> 
> You misinterpret why I marked `(never email me about this bug)' ... I'd have
> closed it had I had the opportunity.
> 
> > b) you won't get notified when the bug does make progress
> 
> Given nearly three months of inactivity, I think it's fair to say this bug
> is not a medium priority issue for you.  I understand OSS and I understand
> people contribute where and when they can.

in the grand scheme of all bugs I can work in, and and given the lack of promising material here, yes, that is correct.


> > 2. please, no more screen shots of memory usage :) - just text with couple
> > numbers will do.
> 
> Too bad you wait until now to say the above.  Also, the graphs are not
> screen shots.  They're RRD graphs.  Text numbers are so ... 1980's.

I don't care about the era.  Raw numbers that we can put in tabular format to compare the effect of different testcases is more useful and faster for the mind to process.  When you triage hundreds of bugs a month such things are important.

> > 3. checking accounts for new messages every minute is reminiscent of a past
> > fixed pop bug. 
> 
> The above is an incorrect assumption, my work style is to check email every
> minute.  If you suspect Thunderbird can't handle one minute polling, perhaps
> you should raise it the issue with other people within development.  This
> way it's either addressed or the option is not available.

It's nothing more than a theory and I'm not going to go to developers with a theory that isn't reproducible by me or anyone else. But if you change the settings and prove the theory THEN we have something useful to drive a solution

> ... and three months later, I'm informed.  Feh.  I've long since removed the
> files from my computer.  You do realize that this means I'd have to
> re-gather all the information eh?  *sigh*

I didn't ask you to regather. And not useful to progress unless it's a msgdb log.
p.s. I want to be clear, that your engagement is to be applauded. However, more than engagement is needed to advance a problem.
Reporter

Comment 102

6 years ago
Rather than having this thread devolve further, I'm not going to respond further.  Thanks again.
your welcome. feel free to add new information at any time.
Status: UNCONFIRMED → RESOLVED
Closed: 6 years ago
Flags: needinfo?(pablo)
Resolution: --- → INCOMPLETE
Whiteboard: [closeme 2013-11-01][not lightning] → [not lightning]
You need to log in before you can comment on or make changes to this bug.