753502 - memory usage spikes while selecting multiple messages with attachments and mail.operate_on_msgs_in_collapsed_threads=true

Status: RESOLVED DUPLICATE of bug 782899

(Thunderbird :: General, defect)

Description

[Jack Eidsness]

Attachment: Thunderbird memory.PNG

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:15.0) Gecko/15.0 Firefox/15.0a1
Build ID: 20120509030514

Steps to reproduce:

I had a problem when I selected several messages at once (click on first message, shift-click on another message). I can see that there is a very large spike in memory usage that scales up with the number of messages selected. It doesn't happen for all messages though. I have a folder full of 30-60 k HTML multipart messages, and it is very visible there, but in most of my other folders which just have normal, brief  messages, I have not been able to reproduce the problem. I have not been able to determine what, in general, these messages have in common to trigger this issue. I've had this folder and done similar group deletions in the past, without issue (and it's been bigger in the past - just last week I deleted a bunch of messages from this folder while using Daily, and saw no strange behavior)

I'm using this Daily build:
http://hg.mozilla.org/mozilla-central/rev/1df37b7c3247


Actual results:

When I select 2-5 of these messages, there is a brief pause before the messages (other than the first one I clicked) are highlighted; When I select 5 or more messages, the PC went pretty crazy with memory paging (the hard drive was audibly quite active), trying to allocate enough memory. During this time, the PC is barely responsive.

Sometimes the entire window (or other unrelated windows) disappears momentarily and re appears.

I watched Process Explorer while trying all this again, and I can see that there is a several seconds-long memory usage increase, but it goes back down to normal even while the messages are still selected. I've attached an image that shows the usage increases, corresponding to how many messages I selected (between 2 and 7 messages at once; I don't have a timeline for you but it did correspond to the size of the usage plateaus).

I got the same result in safe mode.


Expected results:

It seems clear that over 4 gigs of memory should not be necessary for selecting messages (which are not even being displayed yet) in a folder whose total size is only 30 Megs.

Comment 1

[Wayne Mery (:wsmwk)]

can you identify the specific build in which this behavior starts, by checking the old nightly builds?  
THey are available at ftp://ftp.mozilla.org/pub/thunderbird/nightly/
see http://garykwong.wordpress.com/2009/02/24/howto-find-regression-windows-through-manual-binary-search/

Comment 2

[Jack Eidsness]

Yesterday's build, http://hg.mozilla.org/mozilla-central/rev/e4f9e2eab6b1 does not have this behavior. It is a little sluggish (a few seconds, to highlight these messages) compared to release version of thunderbird, but there's no memory usage spike, blinking windows, or massive paging affecting system performance.

Comment 3

[Jack Eidsness]

I figure someone might want to see this

Explicit Allocations
2,475.12 MB (100.0%) -- explicit
├──2,427.10 MB (98.06%) -- js
│  ├──2,418.61 MB (97.72%) -- compartment([System Principal])
│  │  ├──2,384.84 MB (96.35%) ── string-chars [143]
│  │  └─────33.77 MB (01.36%) -- (7 tiny)
│  │        ├──21.22 MB (00.86%) ++ gc-heap
│  │        ├───5.75 MB (00.23%) ── analysis-temporary [35]
│  │        ├───3.04 MB (00.12%) ── script-data [210]
│  │        ├───1.75 MB (00.07%) ++ objects
│  │        ├───1.70 MB (00.07%) ++ shapes-extra
│  │        ├───0.27 MB (00.01%) ++ mjit
│  │        └───0.05 MB (00.00%) ++ type-inference
│  └──────8.49 MB (00.34%) ++ (13 tiny)
├─────30.26 MB (01.22%) ── heap-unclassified
└─────17.76 MB (00.72%) ++ (10 tiny)

Other Measurements
2,475.13 MB ── explicit
    0.00 MB ── gfx-d2d-surfacecache
    0.00 MB ── gfx-d2d-surfacevram
    0.21 MB ── gfx-surface-win32
          0 ── ghost-windows
2,446.68 MB ── heap-allocated
2,459.07 MB ── heap-committed
      0.50% ── heap-committed-unused-ratio
    0.79 MB ── heap-dirty
   17.31 MB ── heap-unused
    0.00 MB ── images-content-used-uncompressed
        224 ── js-compartments-system
         12 ── js-compartments-user
   26.00 MB ── js-gc-heap
    5.75 MB ── js-main-runtime-analysis-temporary
   12.55 MB ── js-main-runtime-gc-heap-allocated
   10.84 MB ── js-main-runtime-gc-heap-arena-unused
    0.00 MB ── js-main-runtime-gc-heap-chunk-clean-unused
    0.08 MB ── js-main-runtime-gc-heap-chunk-dirty-unused
   23.48 MB ── js-main-runtime-gc-heap-committed
   10.92 MB ── js-main-runtime-gc-heap-committed-unused
     87.00% ── js-main-runtime-gc-heap-committed-unused-ratio
    2.52 MB ── js-main-runtime-gc-heap-decommitted
    1.05 MB ── js-main-runtime-mjit
    6.75 MB ── js-main-runtime-objects
    4.16 MB ── js-main-runtime-scripts
    4.50 MB ── js-main-runtime-shapes
2,388.67 MB ── js-main-runtime-strings
    0.18 MB ── js-main-runtime-type-inference
          0 ── low-commit-space-events
          0 ── low-memory-events-physical
          0 ── low-memory-events-virtual
2,511.52 MB ── private
2,278.11 MB ── resident
    9.36 MB ── storage-sqlite
2,747.70 MB ── vsize
    0.33 MB ── window-objects-dom
    1.10 MB ── window-objects-layout-arenas
    1.30 MB ── window-objects-layout-style-sets
    0.02 MB ── window-objects-layout-text-runs
    0.81 MB ── window-objects-style-sheets

also confirmed similar behavior in today's mac build.

Comment 4

[Jack Eidsness]

Attachment: A mail spool with a bunch of copies of one message I can use to reproduce problem. shift click a bunch of them to see it.

The contents of this mail is weird, but I don't know what is weird about it that makes this memory problem manifest, so I just took something real and removed as much identifying information as I could find.

Comment 5

[Jack Eidsness]

I don't know exactly what changed, but this is no longer going as high. A CPU core is maxed for several seconds, and the memory usage increases by just 100-150 MB, regardless of how many messages are selected. Selecting more of these messages only changes how long the CPU usage is maxed. To me, it seems to be a noticeable but not particularly disruptive impact on system performance.

Comment 6

[Wayne Mery (:wsmwk)]

(In reply to Jack Eidsness from comment #5)
> I don't know exactly what changed, but this is no longer going as high.
Had you disabled global search/index?

what addons do you run?

Comment 7

[Jack Eidsness]

"Include messages in this folder in Global Search results" is checked for this folder, and I haven't changed it any time in recent memory.

Add-ons:
Add-on Compatibility Reporter 1.1
DOM Inspector 2.0.11
Extra Folder Columns 1.1.4
Growl/GNTP 1.2.3
Test Pilot or Thunderbird 1.3.9
ViewAbout 2.0.1

With today's Daily, I checked and it is still doing the CPU thing in safe-mode.

Comment 8

[Wayne Mery (:wsmwk)]

(In reply to Jack Eidsness from comment #4)
> Created attachment 623296 [details]
> A mail spool with a bunch of copies of one message I can use to reproduce
> problem. shift click a bunch of them to see it.

using the mbox I see this with only with mail.operate_on_msgs_in_collapsed_threads=true 

viewing any one of the 12 messages in the folder requires 3-7MB. But selecting all briefly drives memory up by as much as 150MB, with high CPU, for the time required to build and display the message summaries

Comment 9

[Wayne Mery (:wsmwk)]

Jack, can you track down EXACTLY which oldest daily build this problem first appears?

just to be clear, I didn't reproduce 4gb memory usage

Comment 10

[Jack Eidsness]

I think I can take a crack at that this weekend.

Comment 11

[Wayne Mery (:wsmwk)]

(In reply to Jack Eidsness from comment #10)
> I think I can take a crack at that this weekend.

Hope you had good results.  Someone else having similar problem but much more recent regression

Comment 12

[Wayne Mery (:wsmwk)]

(In reply to Jack Eidsness from comment #10)
> I think I can take a crack at that this weekend.

Jack, can you tell us what you found?