what is size of your inbox folder's .msf file? are you able to try daily build?
Just to be clear, it's not my Inbox I'm talking about, it's another user's Inbox underneath "Other Users". The Other Users/<user>.msf folder that's involved is 5,971,494 bytes when all headers are downloaded. This is 100% reproducible with the 11/12 nightly for Linux x86_64.
Why do you Repair the folder? Is something wrong with it often or did you just find this problem by accident?
The first time I noticed this happening, it just happened spontaneously when I tried to view the other user's inbox and there were many messages needing to be downloading from it. That time, there was no folder repair involved. After that, "Repair folder" was just a useful way to reproduce the issue. The issue was also reproducible after completely removing Other Users* from my ImapMail folder. However, the issue is NOT reproducible in a new Thunderbird profile. I.e., if I create a new profile, add this account to it, turn off offline synchronization to the account, and then open the inbox beneath Other Users, all of the message headers are loaded _without_ Thunderbird's memory getting bloated. I don't know what's different about my profile that is causing this issue to occur, but I do want to be clear that this is a regression -- it does not occur in 16.0.2 and does occur on Trunk and in nightly builds. Another data point: It still occurs with my default profile when I run in safe mode, i.e., with all addons disabled.
Is anything going to be done about this bug? Has anyone made any effort to confirm it? TB18 can't ship with this bug in it? It makes it literally impossible to download IMAP folders with lots of messages in them.
(In reply to Jonathan Kamens from comment #5) > Is anything going to be done about this bug? Has anyone made any effort to > confirm it? TB18 can't ship with this bug in it? It makes it literally > impossible to download IMAP folders with lots of messages in them. not seeing it (never have). You mean it happens with TB18 (and trunk), and not with TB17, correct? can you narrow the regression range pllease to a one day range of TB18 builds from https://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/2012/10/ You may need to cast a net wider than October.
Keywords: regression, regressionwindow-wanted
Summary: HUGE memory consumption when downloading headers for large IMAP folder → HUGE memory consumption when downloading headers for large IMAP folder with old profille
I bisected the nightly comm-central builds, Linux x86_64. The 2012-10-22 build is fine, and the 2012-10-24 build has the issue. Unfortunately there is no 2012-10-23 build to test.
comm-central, Changes pushed after 2012-10-22 00:00:00, before 2012-10-24 06:00:00 > http://hg.mozilla.org/comm-central/pushloghtml?startdate=2012-10-22+00%3A00%3A00&enddate=2012-10-24+06%3A00%3A00
Bug 536324 is relevant? > Bug 536324 Change nsIChannel to support 64-bit content-length According to the change in nsIChannel, changes in protocol handlers/channel handlers is seen. Similar changes is needed for other protocol handler etc.? Or affected by by following? > Bug 787557 Thunderbird 15 inactive msg db close change prevents FolderLoaded events when folders are updated
comm-aurora, Changes pushed after 2012-10-22 00:00:00, before 2012-10-24 06:00:00 > http://hg.mozilla.org/releases/comm-aurora/pushloghtml?startdate=2012-10-22+00%3A00%3A00&enddate=2012-10-24+06%3A00%3A00 If aurora has same regression window, less suspects are obtained.
I figured out the preference that's causing the issue: user_pref("mailnews.customDBHeaders", "Received"); The memory issue occurs when this preference exists and does not occur when it does not. Also, I confirmed again that the issue starts happening in the 2012-10-24 build.
I think then one would suspect bug 787557
Assignee: nobody → kent
Status: UNCONFIRMED → NEW
Component: Folder and Message Lists → Backend
Ever confirmed: true
OS: Linux → All
Product: Thunderbird → MailNews Core
Target Milestone: --- → Thunderbird 17.0
Target Milestone: Thunderbird 17.0 → ---
Version: Trunk → 17
No, that's not it. It's bug 363238. The problem is that m_receivedValue isn't reset when starting to parse each message, so it gets larger and larger forever (and puts incorrect values for "received" into the database).
nsParseMailMessageState::Clear in nsParseMailbox.cpp needs to be updated to truncate m_receivedValue.
Created attachment 694444 [details] [diff] [review] Patch to clear m_receivedValue for each parsed message
Assignee: kent → jik
Status: NEW → ASSIGNED
Attachment #694444 - Flags: review?(kent)
Can someone clarify for me how I can tell which TB releases this bug should be tracked for, or just set the appropriate flags if it's easy for you to figure out?
what are some common addons that might be setting customDBHeaders received?
tracking-thunderbird-esr17: --- → ?
The "IMAP Received Date" add-on sets it (in fact, that's the whole point of the add-on).
Comment on attachment 694444 [details] [diff] [review] Patch to clear m_receivedValue for each parsed message >+ m_receivedValue.Truncate(0); Note: assuming this patch is otherwise correct, preferred use is Truncate() without an explicit parameter.
Created attachment 694539 [details] [diff] [review] Patch to clear m_receivedValue for each parsed message Thanks for the correction, Neil. Updated patch.
Comment on attachment 694539 [details] [diff] [review] Patch to clear m_receivedValue for each parsed message Thanks for the patch, and ouch! for my regression. One issue though: your patch has a bunch of stuff after "exporting patch" that includes a second copy of the patch. I don't know if that is an artifact of your particular usage of hg, but when I did my usual "hg qimport ..." followed by "hg qpush" your patch generated an error on the second hunk, which was the same as the first hunk. but r=me for the change, just be careful when it is landed.
Attachment #694539 - Flags: review?(kent) → review+
Not sure how that happened. I'll try to pay more attention the next time I submit a patch. I'm going to keep this one as-is though to preserve the review+. Whoever checks this in, please note the comment above! Does this fix need to be checked in anywhere other than trunk?
Created attachment 695814 [details] [diff] [review] remove extraneous export info [Approval Request Comment] Regression caused by (bug #): 363238 User impact if declined: received custom header set incorrectly, excess memory use Testing completed (on c-c, etc.): should bake on trunk Risk to taking this patch (and alternatives if risky): very low
Comment on attachment 695814 [details] [diff] [review] remove extraneous export info [Triage Comment] If the regression is caused by bug 363238, which it seems to be, then the version on that bug is TB 19. I just checked that bug 363238 didn't land in 17 or 18, so this patch only needs landing for Aurora for the next TB beta and for SM release of the 19 equivalent.
Summary: HUGE memory consumption when downloading headers for large IMAP folder with old profille → HUGE memory consumption when downloading headers for large IMAP folder with old profille (problem when mailnews.customDBHeaders is customized)
Status: ASSIGNED → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Target Milestone: --- → Thunderbird 20.0
status-thunderbird18: --- → unaffected
status-thunderbird19: --- → fixed
status-thunderbird-esr17: --- → unaffected
tracking-thunderbird-esr17: ? → ---
You need to log in before you can comment on or make changes to this bug.