The default bug view has changed. See this FAQ.

HUGE memory consumption when downloading headers for large IMAP folder with old profille (problem when mailnews.customDBHeaders is customized)

RESOLVED FIXED in Thunderbird 20.0

Status

MailNews Core
Backend
--
major
RESOLVED FIXED
4 years ago
2 months ago

People

(Reporter: Jonathan Kamens, Assigned: Jonathan Kamens)

Tracking

(Blocks: 1 bug, {mlk, regression})

Thunderbird 20.0
x86_64
All
mlk, regression
Dependency tree / graph

Thunderbird Tracking Flags

(thunderbird18 unaffected, thunderbird19 fixed, thunderbird-esr17 unaffected)

Details

(Whiteboard: [regression:TB19])

Attachments

(1 attachment, 2 obsolete attachments)

(Assignee)

Description

4 years ago
I have an IMAP folder with 17,869 messages in it. This folder is _not_ selected for offline use, which mean that only headers get downloaded, except for messages that I choose to view.

Thunderbird 16.0.2 Linux x86_64 is able to download the headers for all of the messages in the folder, i.e., if I open the folder properties and select "Repair folder", without its memory usage growing hardly at all.

In contrast, when I attempt to repair the folder with Thunderbird trunk, its memory usage immediately starts to explode. Before it has downloaded even 3,000 of the headers, memory usage has tripled, to over 3GB. If I try to let it finish, it gets to 9GB or 10GB of memory usage, I think even before half the headers have been downloaded, and then it dies because I only have 12GB of RAM and no swap configured in my computer.

Note that this happens when "Enable Global Search and Indexer" is NOT enabled.

This is 100% reproducible -- Repair Folder in TB 16.0.2 works fine, Repair Folder in TB Trunk consumes huge amounts of RAM and eventually I need to kill it before it completely hoses my machine.

I am wondering if it's not actually something functional that has changed, but rather something lower level than that which is preventing JavaScript garbage collection happening while the headers are being downloaded. But that's just a guess; I don't have any hard evidence of it.

If there is anything I can do to help track down what's going on here, please let me know. It seems like a rather serious bug, perhaps even a critical or blocker bug.
what is size of your inbox folder's .msf file?
are you able to try daily build?
Keywords: mlk
(Assignee)

Comment 2

4 years ago
Just to be clear, it's not my Inbox I'm talking about, it's another user's Inbox underneath "Other Users". The Other Users/<user>.msf folder that's involved is 5,971,494 bytes when all headers are downloaded.

This is 100% reproducible with the 11/12 nightly for Linux x86_64.

Comment 3

4 years ago
Why do you Repair the folder? Is something wrong with it often or did you just find this problem by accident?
(Assignee)

Comment 4

4 years ago
The first time I noticed this happening, it just happened spontaneously when I tried to view the other user's inbox and there were many messages needing to be downloading from it. That time, there was no folder repair involved. After that, "Repair folder" was just a useful way to reproduce the issue.

The issue was also reproducible after completely removing Other Users* from my ImapMail folder.

However, the issue is NOT reproducible in a new Thunderbird profile. I.e., if I create a new profile, add this account to it, turn off offline synchronization to the account, and then open the inbox beneath Other Users, all of the message headers are loaded _without_ Thunderbird's memory getting bloated. I don't know what's different about my profile that is causing this issue to occur, but I do want to be clear that this is a regression -- it does not occur in 16.0.2 and does occur on Trunk and in nightly builds.

Another data point: It still occurs with my default profile when I run in safe mode, i.e., with all addons disabled.
(Assignee)

Comment 5

4 years ago
Is anything going to be done about this bug? Has anyone made any effort to confirm it? TB18 can't ship with this bug in it? It makes it literally impossible to download IMAP folders with lots of messages in them.
(In reply to Jonathan Kamens from comment #5)
> Is anything going to be done about this bug? Has anyone made any effort to
> confirm it? TB18 can't ship with this bug in it? It makes it literally
> impossible to download IMAP folders with lots of messages in them.

not seeing it (never have).  You mean it happens with TB18 (and trunk), and not with TB17, correct?

can you narrow the regression range pllease to a one day range of TB18 builds from https://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/2012/10/  You may need to cast a net wider than October.
Keywords: regression, regressionwindow-wanted
Summary: HUGE memory consumption when downloading headers for large IMAP folder → HUGE memory consumption when downloading headers for large IMAP folder with old profille
(Assignee)

Comment 7

4 years ago
I bisected the nightly comm-central builds, Linux x86_64. The 2012-10-22 build is fine, and the 2012-10-24 build has the issue. Unfortunately there is no 2012-10-23 build to test.
comm-central, Changes pushed after 2012-10-22 00:00:00, before 2012-10-24 06:00:00
> http://hg.mozilla.org/comm-central/pushloghtml?startdate=2012-10-22+00%3A00%3A00&enddate=2012-10-24+06%3A00%3A00
Bug 536324 is relevant?
> Bug 536324 Change nsIChannel to support 64-bit content-length
According to the change in nsIChannel, changes in protocol handlers/channel handlers is seen. Similar changes is needed for other protocol handler etc.?
Or affected by by following?
> Bug 787557 Thunderbird 15 inactive msg db close change prevents FolderLoaded events when folders are updated
comm-aurora, Changes pushed after 2012-10-22 00:00:00, before 2012-10-24 06:00:00
> http://hg.mozilla.org/releases/comm-aurora/pushloghtml?startdate=2012-10-22+00%3A00%3A00&enddate=2012-10-24+06%3A00%3A00
If aurora has same regression window, less suspects are obtained.
(Assignee)

Comment 11

4 years ago
I figured out the preference that's causing the issue:

user_pref("mailnews.customDBHeaders", "Received");

The memory issue occurs when this preference exists and does not occur when it does not. Also, I confirmed again that the issue starts happening in the 2012-10-24 build.
I think then one would suspect bug 787557
Assignee: nobody → kent
Blocks: 787557
Status: UNCONFIRMED → NEW
Component: Folder and Message Lists → Backend
Ever confirmed: true
Keywords: regressionwindow-wanted
OS: Linux → All
Product: Thunderbird → MailNews Core
Target Milestone: --- → Thunderbird 17.0
(darn drugs)
Whiteboard: [regression:TB17]
Target Milestone: Thunderbird 17.0 → ---
Version: Trunk → 17
(Assignee)

Comment 14

4 years ago
No, that's not it. It's bug 363238. The problem is that m_receivedValue isn't reset when starting to parse each message, so it gets larger and larger forever (and puts incorrect values for "received" into the database).
(Assignee)

Comment 15

4 years ago
nsParseMailMessageState::Clear in nsParseMailbox.cpp needs to be updated to truncate m_receivedValue.
urk. so http://hg.mozilla.org/comm-central/pushloghtml?startdate=2012-10-22+00%3A00%3A00&enddate=2012-10-24+06%3A00%3A00
Blocks: 363238
No longer blocks: 787557
Whiteboard: [regression:TB17] → [regression:TB19]
Version: 17 → 19
(Assignee)

Comment 17

4 years ago
Created attachment 694444 [details] [diff] [review]
Patch to clear m_receivedValue for each parsed message
Assignee: kent → jik
Status: NEW → ASSIGNED
Attachment #694444 - Flags: review?(kent)
(Assignee)

Comment 18

4 years ago
Can someone clarify for me how I can tell which TB releases this bug should be tracked for, or just set the appropriate flags if it's easy for you to figure out?
what are some common addons that might be setting customDBHeaders received?
tracking-thunderbird-esr17: --- → ?
(Assignee)

Comment 20

4 years ago
The "IMAP Received Date" add-on sets it (in fact, that's the whole point of the add-on).
Comment on attachment 694444 [details] [diff] [review]
Patch to clear m_receivedValue for each parsed message

>+  m_receivedValue.Truncate(0);
Note: assuming this patch is otherwise correct, preferred use is Truncate() without an explicit parameter.
(Assignee)

Comment 22

4 years ago
Created attachment 694539 [details] [diff] [review]
Patch to clear m_receivedValue for each parsed message

Thanks for the correction, Neil. Updated patch.
Attachment #694444 - Attachment is obsolete: true
Attachment #694444 - Flags: review?(kent)
Attachment #694539 - Flags: review?(kent)

Comment 23

4 years ago
Comment on attachment 694539 [details] [diff] [review]
Patch to clear m_receivedValue for each parsed message

Thanks for the patch, and ouch! for my regression.

One issue though: your patch has a bunch of stuff after "exporting patch" that includes a second copy of the patch. I don't know if that is an artifact of your particular usage of hg, but when I did my usual "hg qimport ..." followed by "hg qpush" your patch generated an error on the second hunk, which was the same as the first hunk.

but r=me for the change, just be careful when it is landed.
Attachment #694539 - Flags: review?(kent) → review+
(Assignee)

Comment 24

4 years ago
Not sure how that happened. I'll try to pay more attention the next time I submit a patch. I'm going to keep this one as-is though to preserve the review+. Whoever checks this in, please note the comment above!

Does this fix need to be checked in anywhere other than trunk?
Keywords: checkin-needed

Comment 25

4 years ago
Created attachment 695814 [details] [diff] [review]
remove extraneous export info

[Approval Request Comment]
Regression caused by (bug #): 363238 
User impact if declined: received custom header set incorrectly, excess memory use
Testing completed (on c-c, etc.): should bake on trunk
Risk to taking this patch (and alternatives if risky): very low
Attachment #694539 - Attachment is obsolete: true
Attachment #695814 - Flags: approval-comm-release?
Attachment #695814 - Flags: approval-comm-esr17?

Updated

4 years ago
Attachment #695814 - Flags: review+
Comment on attachment 695814 [details] [diff] [review]
remove extraneous export info

[Triage Comment]
If the regression is caused by bug 363238, which it seems to be, then the version on that bug is TB 19. I just checked that bug 363238 didn't land in 17 or 18, so this patch only needs landing for Aurora for the next TB beta and for SM release of the 19 equivalent.
Attachment #695814 - Flags: approval-comm-release?
Attachment #695814 - Flags: approval-comm-release-
Attachment #695814 - Flags: approval-comm-esr17?
Attachment #695814 - Flags: approval-comm-esr17-
Attachment #695814 - Flags: approval-comm-aurora+
Summary: HUGE memory consumption when downloading headers for large IMAP folder with old profille → HUGE memory consumption when downloading headers for large IMAP folder with old profille (problem when mailnews.customDBHeaders is customized)
https://hg.mozilla.org/comm-central/rev/cd8eeca7ee80
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Keywords: checkin-needed
Resolution: --- → FIXED
Target Milestone: --- → Thunderbird 20.0
https://hg.mozilla.org/releases/comm-aurora/rev/28f9a0fb4ea7
status-thunderbird18: --- → unaffected
status-thunderbird19: --- → fixed
status-thunderbird-esr17: --- → unaffected
tracking-thunderbird-esr17: ? → ---
Blocks: 1330872
You need to log in before you can comment on or make changes to this bug.