810637 - HUGE memory consumption when downloading headers for large IMAP folder with old profille (problem when mailnews.customDBHeaders is customized)

Status: RESOLVED FIXED

(MailNews Core :: Backend, defect)

Description

[Jonathan Kamens]

I have an IMAP folder with 17,869 messages in it. This folder is _not_ selected for offline use, which mean that only headers get downloaded, except for messages that I choose to view.

Thunderbird 16.0.2 Linux x86_64 is able to download the headers for all of the messages in the folder, i.e., if I open the folder properties and select "Repair folder", without its memory usage growing hardly at all.

In contrast, when I attempt to repair the folder with Thunderbird trunk, its memory usage immediately starts to explode. Before it has downloaded even 3,000 of the headers, memory usage has tripled, to over 3GB. If I try to let it finish, it gets to 9GB or 10GB of memory usage, I think even before half the headers have been downloaded, and then it dies because I only have 12GB of RAM and no swap configured in my computer.

Note that this happens when "Enable Global Search and Indexer" is NOT enabled.

This is 100% reproducible -- Repair Folder in TB 16.0.2 works fine, Repair Folder in TB Trunk consumes huge amounts of RAM and eventually I need to kill it before it completely hoses my machine.

I am wondering if it's not actually something functional that has changed, but rather something lower level than that which is preventing JavaScript garbage collection happening while the headers are being downloaded. But that's just a guess; I don't have any hard evidence of it.

If there is anything I can do to help track down what's going on here, please let me know. It seems like a rather serious bug, perhaps even a critical or blocker bug.

Comment 1

[Wayne Mery (:wsmwk)]

what is size of your inbox folder's .msf file?
are you able to try daily build?

Comment 2

[Jonathan Kamens]

Just to be clear, it's not my Inbox I'm talking about, it's another user's Inbox underneath "Other Users". The Other Users/<user>.msf folder that's involved is 5,971,494 bytes when all headers are downloaded.

This is 100% reproducible with the 11/12 nightly for Linux x86_64.

Comment 3

[:aceman]

Why do you Repair the folder? Is something wrong with it often or did you just find this problem by accident?

Comment 4

[Jonathan Kamens]

The first time I noticed this happening, it just happened spontaneously when I tried to view the other user's inbox and there were many messages needing to be downloading from it. That time, there was no folder repair involved. After that, "Repair folder" was just a useful way to reproduce the issue.

The issue was also reproducible after completely removing Other Users* from my ImapMail folder.

However, the issue is NOT reproducible in a new Thunderbird profile. I.e., if I create a new profile, add this account to it, turn off offline synchronization to the account, and then open the inbox beneath Other Users, all of the message headers are loaded _without_ Thunderbird's memory getting bloated. I don't know what's different about my profile that is causing this issue to occur, but I do want to be clear that this is a regression -- it does not occur in 16.0.2 and does occur on Trunk and in nightly builds.

Another data point: It still occurs with my default profile when I run in safe mode, i.e., with all addons disabled.

Comment 5

[Jonathan Kamens]

Is anything going to be done about this bug? Has anyone made any effort to confirm it? TB18 can't ship with this bug in it? It makes it literally impossible to download IMAP folders with lots of messages in them.

Comment 6

[Wayne Mery (:wsmwk)]

(In reply to Jonathan Kamens from comment #5)
> Is anything going to be done about this bug? Has anyone made any effort to
> confirm it? TB18 can't ship with this bug in it? It makes it literally
> impossible to download IMAP folders with lots of messages in them.

not seeing it (never have).  You mean it happens with TB18 (and trunk), and not with TB17, correct?

can you narrow the regression range pllease to a one day range of TB18 builds from https://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/2012/10/  You may need to cast a net wider than October.

Comment 7

[Jonathan Kamens]

I bisected the nightly comm-central builds, Linux x86_64. The 2012-10-22 build is fine, and the 2012-10-24 build has the issue. Unfortunately there is no 2012-10-23 build to test.

Comment 8

[WADA:World Anti-bad-Duping Agency]

comm-central, Changes pushed after 2012-10-22 00:00:00, before 2012-10-24 06:00:00
> http://hg.mozilla.org/comm-central/pushloghtml?startdate=2012-10-22+00%3A00%3A00&enddate=2012-10-24+06%3A00%3A00

Comment 9

[WADA:World Anti-bad-Duping Agency]

Bug 536324 is relevant?
> Bug 536324 Change nsIChannel to support 64-bit content-length
According to the change in nsIChannel, changes in protocol handlers/channel handlers is seen. Similar changes is needed for other protocol handler etc.?
Or affected by by following?
> Bug 787557 Thunderbird 15 inactive msg db close change prevents FolderLoaded events when folders are updated

Comment 10

[WADA:World Anti-bad-Duping Agency]

comm-aurora, Changes pushed after 2012-10-22 00:00:00, before 2012-10-24 06:00:00
> http://hg.mozilla.org/releases/comm-aurora/pushloghtml?startdate=2012-10-22+00%3A00%3A00&enddate=2012-10-24+06%3A00%3A00
If aurora has same regression window, less suspects are obtained.

Comment 11

[Jonathan Kamens]

I figured out the preference that's causing the issue:

user_pref("mailnews.customDBHeaders", "Received");

The memory issue occurs when this preference exists and does not occur when it does not. Also, I confirmed again that the issue starts happening in the 2012-10-24 build.

Comment 12

[Wayne Mery (:wsmwk)]

I think then one would suspect bug 787557

Comment 13

[Wayne Mery (:wsmwk)]

(darn drugs)

Comment 14

[Jonathan Kamens]

No, that's not it. It's bug 363238. The problem is that m_receivedValue isn't reset when starting to parse each message, so it gets larger and larger forever (and puts incorrect values for "received" into the database).

Comment 15

[Jonathan Kamens]

nsParseMailMessageState::Clear in nsParseMailbox.cpp needs to be updated to truncate m_receivedValue.

Comment 16

[Wayne Mery (:wsmwk)]

urk. so http://hg.mozilla.org/comm-central/pushloghtml?startdate=2012-10-22+00%3A00%3A00&enddate=2012-10-24+06%3A00%3A00

Comment 17

[Jonathan Kamens]

Attachment: Patch to clear m_receivedValue for each parsed message

Comment 18

[Jonathan Kamens]

Can someone clarify for me how I can tell which TB releases this bug should be tracked for, or just set the appropriate flags if it's easy for you to figure out?

Comment 19

[Wayne Mery (:wsmwk)]

what are some common addons that might be setting customDBHeaders received?

Comment 20

[Jonathan Kamens]

The "IMAP Received Date" add-on sets it (in fact, that's the whole point of the add-on).

Comment 21

[[e-mail]]

Comment on attachment 694444 [details] [diff] [review]
Patch to clear m_receivedValue for each parsed message

>+  m_receivedValue.Truncate(0);
Note: assuming this patch is otherwise correct, preferred use is Truncate() without an explicit parameter.

Comment 22

[Jonathan Kamens]

Attachment: Patch to clear m_receivedValue for each parsed message

Thanks for the correction, Neil. Updated patch.

Comment 23

[Kent James (:rkent)]

Comment on attachment 694539 [details] [diff] [review]
Patch to clear m_receivedValue for each parsed message

Thanks for the patch, and ouch! for my regression.

One issue though: your patch has a bunch of stuff after "exporting patch" that includes a second copy of the patch. I don't know if that is an artifact of your particular usage of hg, but when I did my usual "hg qimport ..." followed by "hg qpush" your patch generated an error on the second hunk, which was the same as the first hunk.

but r=me for the change, just be careful when it is landed.

Comment 24

[Jonathan Kamens]

Not sure how that happened. I'll try to pay more attention the next time I submit a patch. I'm going to keep this one as-is though to preserve the review+. Whoever checks this in, please note the comment above!

Does this fix need to be checked in anywhere other than trunk?

Comment 25

[Kent James (:rkent)]

Attachment: remove extraneous export info

[Approval Request Comment]
Regression caused by (bug #): 363238 
User impact if declined: received custom header set incorrectly, excess memory use
Testing completed (on c-c, etc.): should bake on trunk
Risk to taking this patch (and alternatives if risky): very low

Comment 26

[Mark Banner (:standard8)]

Comment on attachment 695814 [details] [diff] [review]
remove extraneous export info

[Triage Comment]
If the regression is caused by bug 363238, which it seems to be, then the version on that bug is TB 19. I just checked that bug 363238 didn't land in 17 or 18, so this patch only needs landing for Aurora for the next TB beta and for SM release of the 19 equivalent.

Comment 27

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/comm-central/rev/cd8eeca7ee80

Comment 28

[Mark Banner (:standard8)]

https://hg.mozilla.org/releases/comm-aurora/rev/28f9a0fb4ea7