Closed Bug 952976 Opened 11 years ago Closed 11 years ago

Memory leak in idle state: memory usage grows at approx 1Gb/hour rate, when empty folder with very large .msf, in Trash, is found

Categories

(MailNews Core :: Backend, defect)

Product:
MailNews Core
Component:
Backend
Platform:
x86_64
Windows 7
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: elenst, Unassigned)

References

(Blocks 1 open bug)

Details

(Keywords: memory-leak, perf)

Attachments

(8 files, 1 obsolete file)

Thunerbird 24.2.0 safe mode memory usage
12.24 KB, image/png
Details
20.0 memory growth over 3 hours
18.07 KB, text/plain
Details
28.0 about:memory and Windows perf monitor for 15 and 45 min
104.24 KB, text/plain
Details
Memory log, about:memory snapshots and NSPR log
69.01 KB, application/x-zip-compressed
Details
feeds.rdf
8.67 KB, text/plain
Details
nspr-msgdb5.log
6.67 KB, text/plain
Details
procmon_thunderbird_JIRA.zip
88.94 KB, application/x-zip-compressed
Details
nspr3.zip
41.42 KB, application/x-zip-compressed
Details
After upgrade to 24.1.1 my Thunderbird started consuming a lot of memory and crashing every few hours with OOM. The previous version was 17.0.7. Upgrade to 24.2.0, which I am on now, did not help. All I do to get the problem is start Thunderbird. The memory usage is normal initially, but it keeps growing steadily, on average, at about 1 Gb per hour, grows up to ~4 Gb which is max that my system can give it, then it crashes with OOM. There is nothing extreme in my configuration, it has 4 IMAP accounts and probably a dozen of RSS subscriptions. I've disabled all plugins, turned off "check for new messages" both on startup and on a time interval, restarted TB in the safe mode, and let it be for 1 hour (was not sending any emails either). Snapshots from the taskman for the startup values and 1-hour values are attached. Sorry if it is a duplicate of something. The closest I found in open bugs is #944376, but I'm not sure it is the same problem, since in my case there is no high CPU usage and no activity is necessary to trigger the erroneous behavior. If it's not a duplicate, please let me know what additional information you need. I've been sending crash reports dutifully for a while, but apparently they didn't help. The problem is rather painful, so I'm willing to do whatever I can to get it resolved.
please - backup your full Thunderbird profile directory, then - try https://ftp.mozilla.org/pub/mozilla.org/thunderbird/releases/20.0b1/win32/en-US/Thunderbird%20Setup%2020.0b1.exe also, from help | troubleshooting | about:crashes please copy and paste the text format list of all your crash IDs.
Severity: normal → major
Flags: needinfo?(elenst)
Keywords: mlk, perf
Below is the list of crash IDs. Please don't be misled by the relatively low number of them, I normally try to restart TB before it goes OOM. bp-d4d1a259-e259-4ae6-95b2-ac7b62131223 12/23/2013 3:10 PM bp-8855eb5f-1422-4ea0-89bf-c3e692131223 12/23/2013 11:41 AM bp-c01eb509-486b-4201-908c-066a82131222 12/22/2013 4:47 PM bp-356997eb-6561-41ae-ae9e-eed7e2131221 12/21/2013 12:13 PM bp-96c5b343-52f5-40d0-b82c-7eec42131220 12/20/2013 8:32 PM c43a9ea8-78a3-4731-b943-c4870c8f42e3 12/20/2013 2:08 PM bp-28a41745-7bbd-4cc2-836b-5f5052131219 12/19/2013 11:37 AM bp-53d1f7b2-c966-4e7f-8bbb-dc6fe2131218 12/18/2013 12:29 PM bp-bb59f091-1cfd-457e-9e04-bc64c2131217 12/17/2013 12:40 PM bp-8d4f76fe-16e7-424b-a506-bd2d32131213 12/13/2013 12:19 PM bp-646cf695-9161-42ea-8c74-7f1e02131211 12/11/2013 5:57 PM aebe274f-5d04-455d-8462-88a40d60a593 12/5/2013 6:20 PM bp-233bab5a-bae1-4678-8c66-ada0e2131130 11/30/2013 7:39 PM bp-46a8af10-413f-4687-b4a8-94dd82131129 11/30/2013 12:48 AM bp-27794e2a-1909-448f-bfad-2414f2131129 11/29/2013 9:24 AM bp-760e2db7-c6f7-4146-be33-3b0cf2131128 11/28/2013 6:16 PM I installed the 20.0 beta, but I will give it a chance to work a while before I jump into any conclusions. I will also try to rollback to 17.0.7 to compare, and will publish the results in a day.
Thanks. All of the crashes are "EMPTY", which is not unexpected for your situation. Instead of version 17, suggest you install Earilybird from http://www.mozilla.org/en-US/thunderbird/channel/ because it has some fixes to the dumping process. In addition please capture every hour or two a memory snapshots using help | troubleshooting | about:memory | measure and save, and attach them to this bug report
> memory usage grows at approx 1Gb/hour rate Get memory related trace data by Win's monitor tool like "Performance Log/Counter Log"(name in XP). If 1Gb/hour rate, "15 minutes to 30 minutes interval" is perhaps sufficient for first check. Constantly grows? Suddenly grows?
I will do all the suggested (in process now, details below), but first I have an important addition/correction to my story. I did rollback to 17.0.7 before having read the alternative suggestion, and left it work overnight (this and further experiments in safe mode, all plugins disabled, all mail/feed reads turned off). And it also failed. The behavior was somewhat different -- it grew slower (approx twice slower, although it's not the exact science), in ~8 hours it grew up to ~3.5 Gb and was still running. When I started looking at it, it displayed various "broken GUI" artefacts (low resolution, disappearing lines etc.) and eventually crashed. I sent the report as well, although it's just as empty. This reminded me how I got to upgrade from 17.0.7 at the first place. One day out of sudden my Thunderbird which worked fine till then displayed the very same GUI problems. I restarted it, and it happened again. I never associated it with OOM back then, I just assumed it's one of the cases when the installed version gets "too old" and Thunderbird forces your hand to upgrade by refusing to work further. So I did. After that I started getting OOM that described earlier, which made me watch the memory closely, which made me realize it's a permanent problem. So, after all, it wasn't the upgrade that caused the problem, it just made it more apparent. I have no idea what could have caused it, I don't remember doing anything specific around the time, but can't swear there was nothing at all. Sorry I didn't remember it before. Now back to the experiments. I tried 20.0-beta; seemingly it has the same dynamics as 17.0.7 (that is, grows slower than 24.x). Another difference from 24.x is that the growth appears to be less steady, more jumpy; but I wouldn't guarantee that, I don't have enough statistics for either version. I haven't waited till it crashes, but I captured memory snapshots from about:memory for the first 3 hours -- 4 snapshots total, 0/1/2/3 hour after start. The file 20.0-memory.txt is attached. Now I'm installing the Earlybird. If it also shows the problem, I will collect the memory data from it. Otherwise, I will return to the last released version and will do it there.
Also, I'm not particularly fluent at the Windows performance monitor, so I would appreciate some quick hints if possible related to this case. I enabled it, and it has 30+ counters in the Memory category. But it seems to be system-wide, and so far I can't find how to tune it per process. I suppose the system-wide information is not that much useful?
Flags: needinfo?(elenst)
I installed Earlybird and captured about:memory data and performance monitor memory summary for 15 and 45 min marks after the startup (everything is still disabled, safe mode). about:memory in 28.0 clearly shows that the growth relates to two particular folders in RSS feeds, even though the feeds updates are currently disabled. I presume the folders are somehow corrupted, and probably if I get rid of them completely (which I can do), the problem will be gone. I will do it, but before that, if you want to get any additional information, please let me know, and I will collect it for you while I still have the problem easily reproducible. Thank you for help.
Obfuscated the original data a bit. Nothing secret in there, just a little concealer.
Attachment #8351367 - Attachment is obsolete: true
Hi all, I realize it's slow due to holidays and all that, but I really need to know whether I can clean up all the mess and live happily ever after, or I should wait. Please let me know whether you are going to need any information from me regarding the problem. Thanks.
Flags: needinfo?
(In reply to elenst from comment #6) > But it seems to be system-wide, and so far I can't find how to tune it per process. Following is needed. (1) Start Tb (2) Select Tb's process, task/thread as "monitor target" at "Counter" setting of Win, and save the "monitor target". Because of "momory size related problem", not all measurement data(column) is not needed. (3) IIRC, as long as "thread-name like one of Tb" is not changed, saved option setting is usable for other Tb run on the PC. You don't need to check "Task Manager Disply" manually/periodically. You can see "Virtual memory size" transition as Graph, if you save as CSV file. You need to do manual "about:memory" capturing only. A suspect is "periodical update check of RSS feed". So, following may help your trouble shooting in order to provide required data for problem analysis by developers. (a) NSPR log for Tb's HTTP access. (b) Monitoring of Tb's RSS Folder access(open etc.) and MsgDBHdr(mail meta data) access(add/delete/change etc.). (b) can be achieved by utilizing "Folder Listener" and "MsgDB Listener" interface in Tb. I can provide you "add on for testing" to watch such events using the Listners.
Flags: needinfo?
Summary: Memory leak in idle state: memory usage grows at approx 1Gb/hour rate → Memory leak in idle state: memory usage grows at approx 1Gb/hour rate, when particular folders in RSS feeds are used, even though the feeds updates are currently disabled
Hi, Sorry, I failed to decipher your (2)-(3) instruction, but I finally found per-process configuration, hope the data is somewhat close to what you requested. If not, let me know and I will recollect it. The attached archive contains 3 files which relate to the same session of TB 24.2.0. The duration of the session was ~1 hour. All startup and periodic updates, including the feeds, were turned off. - TB_perfmon.csv - several memory-related counters for thunderbird process, from start to end; - TB_nspr.log - the log with NSPR_LOG_MODULES=nsHttp:5,timestamp. It was only updated on startup and shutdown, and does not contain any connections to my feeds. I checked later that if I enable the feeds, the log created with the same options is updated periodically and shows connection to the feed sources; - TB_about-memory.txt - several about:memory snapshots during the session. The separator is the line ################### Timestamp: ~HH:mm ################### . It shows the actual timestamp in local time (same TZ as in perfmon log). In regard to (b), I can run your add-on if you provide one, but please only do so if you think there is a reasonable chance that it will help, I suppose none of us wants to spend time just for the sake of it.
(In reply to elenst from comment #11) > In regard to (b), (snip) It's my add-on to obtain following data for event of "move 3 mails from Sub-1 to Inbox" while I'm testing Tb, by "1. enable tracing for Inbox, Sub-1, 2. do move 3 mails from Sub-1 to Inbox, 3. disable tracing and print log". > DateTime=2013/12/30 04:07:59.348 GMT, MsgDB=mailbox://nobody@Local%20Folders/Inbox/Sub-1, EventType=DBOpened > DateTime=2013/12/30 04:07:59.405 GMT, FolderURI=mailbox://nobody@Local%20Folders/Inbox/Sub-1, EventType=FolderLoaded > DateTime=2013/12/30 04:08:16.891 GMT, MsgDB=mailbox://nobody@Local%20Folders/Inbox, EventType=DBOpened > DateTime=2013/12/30 04:08:16.896 GMT, FolderURI=mailbox://nobody@Local%20Folders/Inbox, EventType=MRMTimeChanged > DateTime=2013/12/30 04:08:17.267 GMT, MsgDB=mailbox://nobody@Local%20Folders/Inbox, EventType=MsgDBHdrAdd, MessageKey=113632754, messageOffset=113632754 > DateTime=2013/12/30 04:08:17.604 GMT, MsgDB=mailbox://nobody@Local%20Folders/Inbox, EventType=MsgDBHdrAdd, MessageKey=114996141, messageOffset=114996141 > DateTime=2013/12/30 04:08:17.903 GMT, MsgDB=mailbox://nobody@Local%20Folders/Inbox, EventType=MsgDBHdrAdd, MessageKey=116359528, messageOffset=116359528 > DateTime=2013/12/30 04:08:17.910 GMT, MsgDB=mailbox://nobody@Local%20Folders/Inbox/Sub-1, EventType=MsgDBHdrDeleted, MessageKey=427626016, messageOffset=427626016 > DateTime=2013/12/30 04:08:17.911 GMT, MsgDB=mailbox://nobody@Local%20Folders/Inbox/Sub-1, EventType=MsgDBHdrDeleted, MessageKey=428995001, messageOffset=428995001 > DateTime=2013/12/30 04:08:17.911 GMT, MsgDB=mailbox://nobody@Local%20Folders/Inbox/Sub-1, EventType=MsgDBHdrDeleted, MessageKey=430358388, messageOffset=430358388 > DateTime=2013/12/30 04:08:17.961 GMT, FolderURI=mailbox://nobody@Local%20Folders/Inbox/Sub-1, EventType=DeleteOrMoveMsgCompleted (1) MsgDBHdrAdd event and MsgDBHdrDeleted event on some folders is hooked and logged, to know when an message in a mail folder is added or deleted, for problem analysis of "mail download/filter move" relevant problem. (2) For ease of understanding log by (1), folder event, such as FolderLoaded, DBOpened, MRMTimeChanged, DeleteOrMoveMsgCompleted, on the some folders are hooked and logged. Please ask me to provide you the my add-on, if and only if you actually need such trace data for your problem analysis and you actually want to get such data by the add-on. See bug 866951 and bug 948082 and attached trace data to these bugs. That is trace data for "Drag&Drop relevant events at threadpane/folderpane". Above is merely "Drag&Drop=>MsgDBHdrAdd/MsgDBHdrDeleted, threadpane/folderpane=>Some folders" version.
> Please ask me to provide you the my add-on, if and only if you actually need > such trace data for your problem analysis and you actually want to get such > data by the add-on. Thanks, but no, I have no plans to investigate this problem by myself, I have enough issues in my product to keep me busy. I am willing to assist TB developers by providing additional information if they want to debug and try to fix it, but that's as far as I can go.
(In reply to elenst from comment #13) > > Please ask me to provide you the my add-on, if and only if you actually need > > such trace data for your problem analysis and you actually want to get such > > data by the add-on. > > Thanks, but no, I have no plans to investigate this problem by myself, I > have enough issues in my product to keep me busy. I am willing to assist TB > developers by providing additional information if they want to debug and try > to fix it, but that's as far as I can go. elenst, wada asks questions and does work that aids developers in fixing bugs. So, you would not want to wait for a developer to comment if you wish to have this bug investigated further. Suggest you collected the data for comment 12 and then reply here. Thanks. (let's assume for now this is backend)
Component: Untriaged → Feed Reader
Flags: needinfo?(elenst)
Keywords: regression, regressionwindow-wanted
Product: Thunderbird → MailNews Core
Whiteboard: [closeme 2014-02-01][regression:TB22?]
> elenst, wada asks questions and does work that aids developers in fixing > bugs. I figured that much, which is why I pay actual attention to what wada said: > > > Please ask me to provide you the my add-on, if and only if you actually need > > > such trace data for your problem analysis and you actually want to get such > > > data by the add-on. I already explained that I do not meet this "if and only if" clause because *I* do not need the trace data. I *can* collect it if somebody else needs it for their analysis. If somebody does, please do say so, in which case I hope wada will provide the add-on and I will collect data; but I'm not going to jump through a hoop until there is an actual audience for the show (and I presume wada doesn't want to waste time either, hence the clause). > So, you would not want to wait for a developer to comment if you wish > to have this bug investigated further. I expect that the wish to have the bug to be investigated should come from development, not from me. I'm ready to help them in doing so, but if it's only me who wants it, and the developers don't care, then please feel to ignore it. I'll give it two more days, if nothing happens till Jan 5th I will drop the guilty folders, and I suppose the problem will stop being reproducible, which I will comment about and it will allow you to happily close the bug as "not reproducible" or whatever equivalent status/resolution there is.
Flags: needinfo?(elenst)
(In reply to elenst from comment #15) > I expect that the wish to have the bug to be investigated should come from > development, not from me. No, that's generally not how we work here. We try to have someone reproduce the problem or collect sufficient data for a developer to come to create a patch. > I'll give it two more days, if nothing happens till Jan 5th I will drop the > guilty folders, and I suppose the problem will stop being reproducible, > which I will comment about and it will allow you to happily close the bug as > "not reproducible" or whatever equivalent status/resolution there is. with the holidays it's unlikely anyone will reply within your two days time frame. indeed most paid mozilla developers are still off. But perhaps alta88 is around at least keep a copy of the profile as it is, so if a developer does come around they can ask for this data. Without profile I suspect no one will be able to reproduce and the bug will be closed incomplete and all your effort and wada's will have been wasted. That would be sad.
(In reply to Wayne Mery (:wsmwk) from comment #16) > > No, that's generally not how we work here. We try to have someone reproduce > the problem or collect sufficient data for a developer to come to create a > patch. That's my point precisely. You have the "someone" -- I can reproduce and am willing to collect the data, so now it's time to hear from a developer who will say what data is needed next and when it becomes "sufficient". From my experience, collecting piles of data without a specific request from an actual person who is going to analyze it is majorly useless, the data goes to trash and time is wasted. From the comment by wada (who obviously knows more about how things are here), it seems that this observation is applicable here as well. > at least keep a copy of the profile as it is I will backup the profile.
so it seems the problem is with a (one) feed folder moved to trash (the folder and its .msf file). could you describe what is in that trash folder? folder names, sizes. if they're not too big or sensitive, perhaps they can be zipped and attached. it's unlikely to be anything to do with feed processing per se, but on a lower level, such as .msf file needing to be rebuilt and a process gone wild as a result.
(In reply to elenst from comment #11) > Memory log, (snip) Peek VM size looks to increment XX MB per approximately MM minutes. Average VM size during a "MM minites period" looks to increment YY MB per each "MM minutes". VM size duaring a "MM minites period" doesn't look to vary so large. "VM size inrement per each MM minites period" looks to continue forever. MM looks around 3 minutes. Default of msg purge interval=5 minutes. So, It doesn't look msg pure event. What kind of activity is invoked on the RSS feed periodically? Note: Update check of RSS is disabled and "HTTP access doesn't occur" is already confirmed. "Log for Tb's file access" may help to know what happens. If Win, Process Monitor of Sysnternal can be used for it. By Counter log with short interval(eg. 1nmin), when big increment happened can be known. By Process Monitor log, "fles accessed by Tb duribg a interval" can be known. > http://technet.microsoft.com/en-us/sysinternals > http://technet.microsoft.com/en-us/sysinternals/bb896645 See following page for my add-on. > http://www.h2.dion.ne.jp/~radon/mozilla/tb-addon/WinBackMyTrash-0-1-0-004-MsDBHdrAddDelete.html > http://www.h2.dion.ne.jp/~radon/mozilla/tb-addon/WinBackMyTrash-0-1-0-004-ViewAsHTMLtemporary.html Main purpose of this add-on is to dump Global object properties(window, gDBView etc.), XPCOM object oroperties(MsgDBHdr, msgFolder, msgDatabase etc.) to Error Console for ease of my testing of Tb. Event tracing of HdrAdded/HdrDeleted, View as HTML temporary menu etc. are functions for testing of other bugs and problem analysis of other bugs. "Why not opened to public" is simply "it can easily produce crash/loop/memory leak etc. if dagnerous test case is blindly executed" :-) "tracing of HdrAdded/HdrDeleted" can be used to confirm that "msgDBHdr add/msgDBHdr delete" doesn't occur on relevant msgFoler for RSS when the memory leak happend.
FYI. "Rebuild index happened or not" can be known by NSPR log with MSGDB:5. Following log of %temp%MozillaMailnews\<FolderName>.msf is written if rebuld-index happened on <FolderName>. > (Win-XP example) > nsMsgDatabase::Open(C:\DOCUME~1\<UserName>\LOCALS~1\Temp\MozillaMailnews\<FolderName>.msf, FALSE, ..., TRUE) See bug 905576 for it.
(In reply to elenst from comment #11) > about:memory snapshots (snip) Heap area looks to increase infinitely. > ################### Timestamp: ~00:01 ################### > 128,104,638 B -- heap-allocated > 130,461,696 B -- heap-committed > 2,352,242 B -- heap-committed-unused > 1.82% -- heap-committed-unused-ratio > 512,000 B -- heap-dirty > 4,006,098 B -- heap-unused > > ################### Timestamp: ~00:12 ################### > 208,748,300 B -- heap-allocated > 215,826,432 B -- heap-committed > 7,032,500 B -- heap-committed-unused > 3.36% -- heap-committed-unused-ratio > 2,097,152 B -- heap-dirty > 8,256,372 B -- heap-unused > > ################### Timestamp: ~00:35 ################### > 419,423,622 B -- heap-allocated > 425,525,248 B -- heap-committed > 6,093,818 B -- heap-committed-unused > 1.45% -- heap-committed-unused-ratio > > ################### Timestamp: ~00:55 ################### > 580,840,364 B -- heap-allocated > 590,217,216 B -- heap-committed > 9,329,300 B -- heap-committed-unused > 1.60% -- heap-committed-unused-ratio > 1,908,736 B -- heap-dirty > 10,502,564 B -- heap-unused This is usually, object is never destroyed after creation/use, or destroyed object won't be freemained by garbage collector due to, for example, never-cleared reference between C++ object and XPCOM object.
To elenst(bug opener): Do you see your problem with newly created Tb profile, with problematic RSS feed only is defined? If you start Tb with '-no-remote -p "he_new_profile"', you can run mulriple Tb instances on one PC, one with Tb24/daily-use-prof, another with Tb-trunk/newly-created-profile. If you will do so, please change skin or color/font setting of Tb-trunk, in order to avoid your confusios.
(In reply to alta88 from comment #18) > could you describe what is in that trash folder? folder names, sizes. if > they're not too big or sensitive, perhaps they can be zipped and attached. The name of the folder in question is 'JIRA'. It sits in the trash of Feeds. The complete path according to about:memory is mailbox:///C:/Users/<username>/AppData/Roaming/Thunderbird/Profiles/<profile name>.default/Mail/Feeds/Trash.sbd/JIRA Disk size of the corresponding JIRA.msf is 11,396 KB. I cannot say how many messages are in there, because it seems I am not able to open it. Any attempts to do so causes much faster memory consumption, and the proper message list never shows up. As far as I remember, it was exactly why it ended up in trash -- it started misbehaving and I tried to get rid of it. It did not help though, the new Feeds/JIRA folder does not work either. There are no subfolders in Feeds/Trash/JIRA -- at least there were none when it was outside Trash and still working. There is a single subscription which is supposed to store articles in Feeds/Jira (which this folder used to be before it was sent to trash): https://mariadb.atlassian.net/activity?maxResults=5&os_authType=basic&title=undefined It is a subscription of a medium activity, I would estimate it as a few dozen messages a day, but my mileage may vary. The subscription requires authentication. It includes all updates of JIRA entries in several JIRA projects associated with MariaDB. While I, personally, don't think there is anything particularly sensitive in there, I'm not at liberty to publicize the contents of the entire folder, since apart from the normal open JIRA contents it contains comments that our users chose to restrict the access to, and also some administrative stuff. If it comes to the point when you cannot proceed without the actual folder contents, I'll see what I can do -- maybe I can edit it, or something (would be easy to try if I could actually open it, but see above about opening). I will look into trying WADA's add-on tomorrow, when I'm back to the machine in question.
(In reply to WADA from comment #22) > To elenst(bug opener): > Do you see your problem with newly created Tb profile, with problematic RSS > feed only is defined? > If you start Tb with '-no-remote -p "he_new_profile"', you can run mulriple > Tb instances on one PC, one with Tb24/daily-use-prof, another with > Tb-trunk/newly-created-profile. If you will do so, please change skin or > color/font setting of Tb-trunk, in order to avoid your confusios. I will try it as soon as I am back to the machine. I suspect that the feed is not working at the moment, it happened to break before, and now I can't get it work on another machine. Also, as a correction to my previous statement > There is a single subscription which is supposed to store articles in Feeds/Jira > (which this folder used to be before it was sent to trash): > > https://mariadb.atlassian.net/activity?maxResults=5&os_authType=basic&title=undefined That's the link that the *current* Feeds/JIRA folder attempts to use; the older link could have been somewhat different, particularly in the option set (e.g. maxResults could be omitted). The source was the same.
You don't mention how large the message folder is, and it's not clear if you're using the eu , or the .msf is really 11M in size. If the latter, that's enourmous and the problem - something is filling it and as long as the folder's msgDb is open it will blow up memory. A folder might be 200M for a .msf of 11M - but this can be highly variable. Using https://mariadb.atlassian.net/activity one can subscribe to the feed (or changing 'basic' auth to 'none'); it has 10 entries for a file size of 22K and .msf of 13K. There isn't anything unusual about it and subscribing/updating works fine in Tb24 (and a new feeds account). Since you've backed up the profile (really just the trash folders are needed), how about doing a Properties-> Repair Folder, in trash.
(In reply to alta88 from comment #25) > You don't mention how large the message folder is, and it's not clear if > you're using the eu , or the .msf is really 11M in size. If the latter, > that's enourmous and the problem - something is filling it and as long as > the folder's msgDb is open it will blow up memory. A folder might be 200M > for a .msf of 11M - but this can be highly variable. My bad, I thought the whole contents is packed into an *.msf file, and did not look further. I believe I checked specifically the msf file size, but I will make sure tomorrow. > Using https://mariadb.atlassian.net/activity one can subscribe to the feed > (or changing 'basic' auth to 'none'); it has 10 entries for a file size of > 22K and .msf of 13K. There isn't anything unusual about it and > subscribing/updating works fine in Tb24 (and a new feeds account). I could not use it without authentication in real life, because as I said there is also restricted contents and I need it all. But I can surely try it as an experiment. Also, if the size is anyhow close to linear, and 10 messages make an msf of 13K, it's entirely possible that I had 10K+ messages in there by the time it was sent to trash. > Since you've backed up the profile (really just the trash folders are > needed), how about doing a Properties-> Repair Folder, in trash. Okay.
(In reply to elenst from comment #23) > I cannot say how many messages are in there, because it seems I am not able to open it. > Any attempts to do so causes much faster memory consumption, > and the proper message list never shows up. Could yu please explain details about the "I am not able to open it"? We assumed problem with "normal/ordinal RSS feed". But it sounds problem with "RSS feed which can not be accessed normally any more".
> Could yu please explain details about the "I am not able to open it"? > We assumed problem with "normal/ordinal RSS feed". > But it sounds problem with "RSS feed which can not be accessed normally any > more". I think these are two separate problems, not working folder and not working feed, sorry for the confusion, I only discovered the latter one recently. I'll try to explain. About not working *folder*: I have a Windows machine where I observe the memory problem which is the subject of this report. It used to have Feeds/JIRA folder as described above, where the feed from mariadb.atlassian.net stored the messages. The feed and the folder worked all right, it got updated, I could see the messages, etc. One day I tried to open the folder as usual, but the TB was not displaying messages in there properly, but kept pretending that it was loading something ("waiting" cursor, etc.). Eventually, instead of scrolling through the folder, I would only see GUI glitches -- messages disappearing from the screen, etc. Restart did not help. Upgrading from TB 17 to 24 did not help. Then I dropped the folder (moved it to Trash) and created a new one with the same name. It's possible that I also dropped the subscription and created a new one instead, but I'm not 100% sure about it. It did not help, now I cannot open either the old folder Feeds/Trash/JIRA, or the new one Feeds/JIRA, with the same result -- "loading/waiting" cursor, GUI glithces. And, as I only noticed after TB upgrade, the memory consumption issue. About not working *subscription*: I tried to set up the same feed recently on another machine, openSUSE 13.1 with TB 24.1.0, but it does not pass the validation, it says "The feed URL cannot be found" (the URL is from the atlassian page, https://mariadb.atlassian.net/activity?maxResults=5&os_authType=basic&title=undefined, I also tried some variations of it). I'm trying to find out what's wrong with the URL, but everything is slow due to the holidays, so it takes long. However, I'm not sure that the issues are related, since the memory problem is observed even when all updates are disabled, and TB does not go to mariadb.atlassian.net at all, hence cannot know whether the URL is valid or not. Hope it's clearer now.
FYI. Because you are Win user, you can get NSPR log ad-hoc if you use DebugView of SysInternal. > http://technet.microsoft.com/en-us/sysinternals/bb896647 See bug 402793 comment #6 for it.
(In reply to elenst from comment #28) > I think these are two separate problems, not working folder and not working > feed, sorry for the confusion, I only discovered the latter one recently. > I'll try to explain. > > About not working *folder*: > > I have a Windows machine where I observe the memory problem which is the > subject of this report. > It used to have Feeds/JIRA folder as described above, where the feed from > mariadb.atlassian.net stored the messages. The feed and the folder worked > all right, it got updated, I could see the messages, etc. > One day I tried to open the folder as usual, but the TB was not displaying > messages in there properly, but kept pretending that it was loading > something ("waiting" cursor, etc.). Eventually, instead of scrolling through > the folder, I would only see GUI glitches -- messages disappearing from the > screen, etc. > Restart did not help. > Upgrading from TB 17 to 24 did not help. > Then I dropped the folder (moved it to Trash) and created a new one with the > same name. It's possible that I also dropped the subscription and created a > new one instead, but I'm not 100% sure about it. > It did not help, now I cannot open either the old folder Feeds/Trash/JIRA, > or the new one Feeds/JIRA, with the same result -- "loading/waiting" cursor, > GUI glithces. And, as I only noticed after TB upgrade, the memory > consumption issue. Does it mean you have two "folders" and "files"? ...\Mail\Feed\Trash.sbd\<Title_of_JIRA>.msf & ...\Mail\Feed\Trash.sbd\<Title_of_JIRA> ...\Mail\Feed\<Title_of_JIRA>.msf & ...\Mail\Feed\<Title_of_JIRA> Was both Feed\Trash.sbd\<Title_of_JIRA> and ...\Mail\Feed\<Title_of_JIRA> updated by GetMsg? I can't think so, if "Delete RSS feed" is done on Tb 24. Feed\feeds.rdf dc:identifier="http://<path of RSS feed>.xml" => <fz:destFolder RDF:resource="mailbox://nobody@Feeds<Title_of_the_RSS_feed>"/> This entry is removed by "Delete of RSS feed". Even if update of Feed\feeds.rdf is interfered, the Feed\feeds.rdf won&t point ...\Mail\Feed\Trash.sbd\<Title_of_JIRA>, so feeditem won't be writen to ...\Mail\Feed\Trash.sbd\<Title_of_JIRA>. At Feeds/Trash/JIRA, I think item in ...\Mail\Feed\Trash.sbd\<Title_of_JIRA> is shown using ...\Mail\Feed\Trash.sbd\<Title_of_JIRA>.msf without server access, as done on local mail folder. Is RSS feed content shown when "Work Offline" mode? Is Trash\<Title_of_JIRA> registered in ...\Mail\Feed\feeds.rdf? If following entries are seen in ...\Mail\Feed\feeds.rdf, update of Trash/JIRA may occur. dc:identifier="http://<path of RSS feed>.xml" => <fz:destFolder RDF:resource="mailbox://nobody@Feeds<Title_of_the_RSS_feed>"/> dc:identifier="http://<path of RSS feed>.xml" => <fz:destFolder RDF:resource="mailbox://nobody@Feeds/Trash/<Title_of_the_RSS_feed>"/> If so, "not accessible to the RSS feed on the PC" is perhaps same as "About not working *folder* on other machine". If so, memory leak in recent Tb may be caused by duplicated entry for same RSS feed in feed.rdf. > > About not working *subscription*: > > I tried to set up the same feed recently on another machine, openSUSE 13.1 > with TB 24.1.0, but it does not pass the validation, it says "The feed URL > cannot be found" (the URL is from the atlassian page, > https://mariadb.atlassian.net/ > activity?maxResults=5&os_authType=basic&title=undefined, I also tried some > variations of it). I'm trying to find out what's wrong with the URL, but > everything is slow due to the holidays, so it takes long. > > However, I'm not sure that the issues are related, since the memory problem > is observed even when all updates are disabled, and TB does not go to > mariadb.atlassian.net at all, hence cannot know whether the URL is valid or > not. > > Hope it's clearer now.
Attached file feeds.rdf —
Attached file nspr-msgdb5.log —
I tried to find and reply to all questions to me from above. If I missed something, please let me know. Also, I didn't use WADA's add-on yet; if you still need the data it produces, please let me know and I will run it. (In reply to WADA from comment #19) > "Log for Tb's file access" may help to know what happens. > If Win, Process Monitor of Sysnternal can be used for it. Attached CSV file in procmon_thunderbird_JIRA.zip. The events are filtered by process name, File system event type, and the presence of JIRA feed folder in the path; if a wider log is needed, please let me know. (In reply to WADA from comment #20) > "Rebuild index happened or not" can be known by NSPR log with MSGDB:5. Attached nspr-msgdb-5.log, created with NSPR_LOG_MODULES=nsHttp:5,timestamp,MSGDB:5. TB was running for ~30 min and then was shut down, totally idle, all updates disabled, GUI not touched (Local folders is in focus upon startup). It gained ~200mb over this time. (In reply to WADA from comment #22) > Do you see your problem with newly created Tb profile, with problematic RSS > feed only is defined? Could not define the feed in a new profile using https://mariadb.atlassian.net/activity?maxResults=5&os_authType=basic&title=undefined (which I suppose the one associated with the JIRA folder used to be), due to the feed problem described earlier -- it says that the URL could not be found. I can revisit this later when I find out what's wrong with the address. Tried https://mariadb.atlassian.net/activity?streams= (which I also found in feeds.rdf), it seems to be working all right in a new profile. (In reply to alta88 from comment #25) > You don't mention how large the message folder is, and it's not clear if > you're using the eu , or the .msf is really 11M in size. If the latter, > that's enourmous and the problem - something is filling it and as long as > the folder's msgDb is open it will blow up memory. A folder might be 200M > for a .msf of 11M - but this can be highly variable. I have a file Feeds/Trash.sbd/JIRA.msf (11,396 Kb, recent timestamp) and a file Feeds/Trash.sbd/JIRA (0 Kb, old timestamp): 0 2013-11-28 00:04 JIRA 11669316 2014-01-04 22:39 JIRA.msf So, I presume the JIRA file is supposed to be the "message folder", but something is wrong with it? (In reply to alta88 from comment #25) > Since you've backed up the profile (really just the trash folders are > needed), how about doing a Properties-> Repair Folder, in trash. When I press 'Repair Folder', nothing visible happens, the timestamp or the size of 'JIRA.msf' and 'JIRA' do not change. Also: when I open Properties on the folder, the screen shows Name (JIRA), but Location is empty. Consequently, the OK button does not work. (In reply to WADA from comment #29) > FYI > Because you are Win user, you can get NSPR log ad-hoc if you use DebugView > of SysInternal. Thanks. I'm a user, but not a big fan; so please lets see first whether the NSPR log I created by the usual means provides the information you wanted to get. If it doesn't, I'll try the Windows tool. (In reply to WADA from comment #30) > Does it mean you have two "folders" and "files"? > ...\Mail\Feed\Trash.sbd\<Title_of_JIRA>.msf & > ...\Mail\Feed\Trash.sbd\<Title_of_JIRA> > ...\Mail\Feed\<Title_of_JIRA>.msf & ...\Mail\Feed\<Title_of_JIRA> That's exactly right. I have: 0 2013-12-17 04:41 JIRA 11650272 2014-01-04 22:19 JIRA.msf 0 2013-11-28 00:04 Trash.sbd\JIRA 11669375 2014-01-05 00:12 Trash.sbd\JIRA.msf > Was both Feed\Trash.sbd\<Title_of_JIRA> and ...\Mail\Feed\<Title_of_JIRA> > updated by GetMsg? As you can see, none was updated for a long time. 2013-11-28 (timestamp of Trash.sbd\JIRA) might well be when the folder was moved to trash. I can't remember what 2013-12-17 (timestamp of JIRA) stands for, my best guess is that's when the feed stopped working. > Is RSS feed content shown when "Work Offline" mode? No, same story: even if I switch to Offline mode and then try to open the folder, nothing of its contents is displayed, TB stays in "thinking" mode, GUI glitches. > Is Trash\<Title_of_JIRA> registered in ...\Mail\Feed\feeds.rdf? > If following entries are seen in ...\Mail\Feed\feeds.rdf, update of > Trash/JIRA may occur. > dc:identifier="http://<path of RSS feed>.xml" > => <fz:destFolder > RDF:resource="mailbox://nobody@Feeds<Title_of_the_RSS_feed>"/> > dc:identifier="http://<path of RSS feed>.xml" > => <fz:destFolder > RDF:resource="mailbox://nobody@Feeds/Trash/<Title_of_the_RSS_feed>"/> > > If so, "not accessible to the RSS feed on the PC" is perhaps same as "About > not working *folder* on other machine". > If so, memory leak in recent Tb may be caused by duplicated entry for same > RSS feed in feed.rdf. There is no 'Trash' anywhere in the feeds.rdf; but there might still be something that TB considers a duplicate and chokes upon, I see different feed links for atlassian.net in different places of feeds.rdf. I attached the entire file for you to review, it's small and there is nothing secret in there.
Attachment #8355817 - Attachment mime type: application/rdf+xml → text/plain
Attachment #8355818 - Attachment mime type: text/x-log → text/plain
(In reply to elenst from comment #32) > Created attachment 8355818 [details] > nspr-msgdb5.log MsgDatabase Open request is logged multiple times. > nsMsgDatabase::Open(C:\Users\ ... \Mail\Feeds\JIRA.msf, FALSE, 600bbd0, TRUE) > nsMsgDatabase::Open(C:\Users\ ... \Mail\Feeds\Trash.sbd\JIRA.msf, FALSE, 600baa0, TRUE) However, report of "currently opened MsgDatabase" is always for "Local Folders\Unsent Messages" only. > 1 open DB's > C:\Users\ ... \Mail\Local Folders\Unsent Messages.msf - 0 hdrs in use If MsgDatabase Open failure is normally detected by Tb's MsgDatabase Open code, "error openin db ..." log should be written. If MsgDatabase Open is successful, the MsgDatabase should be contained in "nn open DB's" log. (In reply to elenst from comment #31) > Created attachment 8355817 [details] > feeds.rdf Definition of RSS feeds for "Feeds/JIRA" folder. > <RDF:Description RDF:about="http://www.planetmysql.org/" fz:quickMode="false" /> > <fz:feed RDF:about="https://mariadb.atlassian.net/activity?maxResults=5&amp;os_authType=basic&amp;title=undefined" > dc:identifier="https://mariadb.atlassian.net/activity?maxResults=5&amp;os_authType=basic&amp;title=undefined" > fz:quickMode="false"> > <fz:destFolder RDF:resource="mailbox://nobody@Feeds/JIRA"/> > </fz:feed> It looks for me; (a) 1. MsgDatabase open of Feeds/JIRA folder failed due to server access failure. 2. Open failure is not detected by MsgDatabase open code, then waits for open-success/open-failre notification. 3. Because Feeds/JIRA is not opned state, MsgDatabase open, then goto step 1. This is "not working *subscription*" part, which was observed with new profile, and on other machine, and after delete/re-define the RSS feed. (b) Even if Work Offline mode, open of Feeds/Trash/JIRA reuires RSS feed relevant data, for example, "certicicate is not expired". Due to such error, open of Feeds/Trash/JIRA also waits forever. This is "not working *folder*" part. os_authType=basic is relevant to problem? But if such issue or cerificate relevant problem, log by nsHttp:5 should be writtem at same time... "RSS feed" approximately consists of; (i) Feed item fetch from "RSS feed URI defined in feeds.rdf" into feeditems.rdf. (ii) Write feed item data into ...\Mail\Feeds\<Folder_Name> file as if mail data, and create MsgDBHdr(Offset in file etc.) in ...\Mail\Feeds\<Folder_Name>.msf. Do you see content by next? - Copy ...\Mail\Feeds\Trash.sbd\JIRA and JIRA.msf files to ...\Mail\Local Folders\Trash.sbd\JIRA and JIRA.msf files. - Restart Tb. How did you subscribe the https://mariadb.atlassian.net/activity? Do you see your problem with subscibing via RSS feeds definitions at latest https://mariadb.atlassian.net/activity site? If added to "Live Bookmarks" at browser(checked with SeaMonkey), following was generated. Feed Location: https://mariadb.atlassian.net/activity Site Location: https://mariadb.atlassian.net/activity When https://mariadb.atlassian.net/activity was pasted to Feed URL: field of Tb, following was generated in feeds.rdg. > <fz:feed RDF:about="https://mariadb.atlassian.net/activity" > dc:title="Activity Streams" > dc:identifier="https://mariadb.atlassian.net/activity" > fz:quickMode="false"> > <fz:destFolder RDF:resource="mailbox://nobody@Feeds/Activity%20Streams"/> > </fz:feed> > Could not define the feed in a new profile using https://mariadb.atlassian.net/activity?maxResults=5&os_authType=basic&title=undefined Does it mean that you pasted following string to Feed URL: field of Tb? https://mariadb.atlassian.net/activity?maxResults=5&os_authType=basic&title=undefined
If https://mariadb.atlassian.net/activity?maxResults=5&os_authType=basic&title=undefined was requested by Firefox, following is logged at Browser Console(or Web Console). > 14:20:16.868 GET https://mariadb.atlassian.net/activity [HTTP/1.1 401 Unauthorized 1079ms] > 14:20:17.546 POST http://ocsp.digicert.com/ [HTTP/1.1 200 OK 93ms] > 14:20:17.672 POST http://ocsp.digicert.com/ [HTTP/1.1 200 OK 47ms] And followig dialog was shown. > Authentication is requested. > ? A username and password are being requested by https://mariadb.atlassian.net. > The site says: "protected-area" > User Name: [ ] > Password: [ ] Why this HTTP GET request to mariadb.atlassian.net/activity and "401 Unauthorized" response is not logged by Tb in your environment?
When accessed https://mariadb.atlassian.net by Firefox, redirected to secure/Dashboard.jspa. > > 16:40:29.249 GET https://mariadb.atlassian.net/ [HTTP/1.1 302 Moved Temporarily 1000ms] > 16:40:30.255 GET https://mariadb.atlassian.net/secure/MyJiraHome.jspa [HTTP/1.1 302 Moved Temporarily 219ms] > 16:40:30.469 GET https://mariadb.atlassian.net/secure/Dashboard.jspa [HTTP/1.1 200 OK 766ms] RSS feed link for Activity Stram/JRA at https://mariadb.atlassian.net/secure/Dashboard.jspa was; https://mariadb.atlassian.net/activity?maxResults=5&title=undefined If you go https://mariadb.atlassian.net/activity using SeaMonkey after login, via link in the mariadb.atlassian.net site, following menu is shown. Subscribe this feed using [ News & Blogs ] When "Subscribe using News & Blogs" is executed by SeaMonkey who has both Browser and MailNews, RSS feed is added to SeaMonkey's MailNews, and following is generated in feeds.rdf of SeaMonkey. dc:identifier="https://mariadb.atlassian.net/activity" I used this URI for the RSS feed on Thunderbird. Because URI in "Website: https://mariadb.atlassian.net/..." is protected by basic authentication, user need to do Login in order to view the Website: content using Browser. > (a) If the RSS feed is subscribed by Browser's "Live Bookmark", following is usable. > dc:identifier="https://mariadb.atlassian.net/activity?maxResults=5&amp;os_authType=basic&amp;title=undefined" > (b) If the RSS feed is subscribed by SeaMonkey's "News & Blog", following may not be so convenient, because Sm has "Open in Browser", "Bookmark this link", "Copy link location" menu only for "Website: https://...". > dc:identifier="https://mariadb.atlassian.net/activity?maxResults=5&amp;os_authType=basic&amp;title=undefined" > in any case, "Login at Browser" will be required sooner or later in order to see content > of "Website: https://..." using Browser. > (c) If the RSS feed is subscribed by Thunderbird's "News & Blog", following is usseless, > because Tb has "Copy link location" menu only for "Website: https://...". > dc:identifier="https://mariadb.atlassian.net/activity?maxResults=5&amp;os_authType=basic&amp;title=undefined" > "Login at Browser" will be always required to see content of "Website: https://..." using Browser. If "Subscribe by News&Blog of Seamonkey via https://mariadb.atlassian.net/activity page" is not securiti violation of mariadb.atlassian.net, following is perhaps simplest workaround. Use https://mariadb.atlassian.net/activity as Feed URL: in Thunderbird.
To elenst(bug opener): Was you requested User Name/Password by Thunderbird with following Feed URL:? > dc:identifier="https://mariadb.atlassian.net/activity?maxResults=5&amp;os_authType=basic&amp;title=undefined" Do you use HTTP proxy or "HTTP access scanner" like one of anti-virus software? Do you use add-on relevant to basic authentication? "Easy to read HTTP flow log" doesn't look written when HTTPS; and/or basic authentcation. When you get NSPR log next time, use following parameter, please. timestamp,MSGDB:5,nsHttp:5,nsHostResolver:5,negotiateauth:5,pipnss:5 DNS access is logged by nsHostResolver. So "finding access log for mariadb.atlassian.net" is easier. I don't know about negotiateauth/pipnss well. But if something is logged, it may help problem analysis. And, do "GetMsgs" for the RSS feed folder, in order to force Web site access to know about "newer feed items".
Attached file nspr3.zip —
(In reply to WADA from comment #35) > Do you see content by next? > - Copy ...\Mail\Feeds\Trash.sbd\JIRA and JIRA.msf files > to ...\Mail\Local Folders\Trash.sbd\JIRA and JIRA.msf files. > - Restart Tb. The behavior is the same as with Trash/JIRA (no content, GUI glitches, faster memory consumption). > How did you subscribe the https://mariadb.atlassian.net/activity? Feeds -> Subscribe -> put the link into Feed URL, press Add. > Do you see your problem with subscibing via RSS feeds definitions at latest > https://mariadb.atlassian.net/activity site? I added the subscription to a new profile, let it run for a short while, did not see any problems so far: the contents is shown, the updates are received. > > Could not define the feed in a new profile using https://mariadb.atlassian.net/activity?maxResults=5&os_authType=basic&title=undefined > > Does it mean that you pasted following string to Feed URL: field of Tb? > > https://mariadb.atlassian.net/activity?maxResults=5&os_authType=basic&title=undefined Yes, that's what I did (in the new profile). (In reply to WADA from comment #36) > Why this HTTP GET request to mariadb.atlassian.net/activity and "401 > Unauthorized" response is not logged by Tb in your environment? I did not collect/attach any logs from the *new* profile. If you need them, please let me know. (In reply to WADA from comment #38) > To elenst(bug opener): > Was you requested User Name/Password by Thunderbird with following Feed > URL:? > > dc:identifier="https://mariadb.atlassian.net/activity?maxResults=5&amp;os_authType=basic&amp;title=undefined" When I tried to add it to a new profile, yes, I was. I entered the credentials and *after that* I got "The URL could not be found". When I start Thunderbird with the old profile, I'm not asked for credentials anymore (I used to, back in days). > > Do you use HTTP proxy or "HTTP access scanner" like one of anti-virus > software? I don't use a proxy. I tried to switch off for a short while whatever the anti-virus has, didn't notice any change in behavior, neither with the new profile trying to subscribe, nor with the old one trying to show feeds contents. The attached log was written during this downtime of the protection. > Do you use add-on relevant to basic authentication? I don't use any add-ons, whatever is there on the list of Plugins is 'Never activate', and the only add-on 'Test pilot' on Extensions list is disabled. > When you get NSPR log next time, use following parameter, please. > timestamp,MSGDB:5,nsHttp:5,nsHostResolver:5,negotiateauth:5,pipnss:5 > DNS access is logged by nsHostResolver. So "finding access log for > mariadb.atlassian.net" is easier. I don't know about negotiateauth/pipnss > well. But if something is logged, it may help problem analysis. > And, do "GetMsgs" for the RSS feed folder, in order to force Web site access > to know about "newer feed items". The new log is attached as npsr3.zip. I don't see there any access to atlassian.net, though. Timestamps in the log: 16:05:41 - TB is started, 'Local folders' is in focus; 16:13:18 - focus moved to 'Feeds'; 16:16:37 - 'Get messages' executed on Feeds; 16:28:59 - TB is shut down. There was no other voluntary activity from my side.
(In reply to elenst from comment #39) > > Do you see content by next? > > - Copy ...\Mail\Feeds\Trash.sbd\JIRA and JIRA.msf files > > to ...\Mail\Local Folders\Trash.sbd\JIRA and JIRA.msf files. > The behavior is the same as with Trash/JIRA (no content, GUI glitches, > faster memory consumption). Trash/JIRA seems culprit of memory leak. How about following? Copy ...<ProblematicDailyUseProfile>\Mail\Feeds\Trash.sbd\JIRA and JIRA.msf to ...<NewlyCreateProblemaWith_JIRA_only\Mail\Local Folders\Trash.sbd\JIRA and JIRA.msf "Nothing is shown for Trash/JIRA at Thread pane" may be "all feed item data has EXPUNGED bit=On in X-Mozilla-Status: header". Check ...\Feeds\Trash.sbd\JIRA(not JIRA.msf) content by text editor, and see X-Mozilla-Status: headers, please. - Each feed item data(==same as mail) is separated by "From - <date-timestamp>" line. - Each mail has X-Mozilla-Status; header, which has hexadecimal string of 2bytes data. Expunge bit is first/left-most bit of 4-th digit for last 4bits of the 2 bytes data. So, if 4-th digit is one of "8, 9, A, B, C, D, E, F", the mail is marked as "deleted", and is removed permanently by "Compact" of the folder. > > Do you see your problem with subscibing via RSS feeds definitions at latest > > https://mariadb.atlassian.net/activity site? > I added the subscription to a new profile, let it run for a short while, did > not see any problems so far: the contents is shown, the updates are received. Anyway, as alta88 said in Comment #25, you have backup of profile which produces memory leak problem. And, as alta88 said in Comment #25 and you already saw, "no basic authentication" is simplest/easiest workaround in JIRA RSS feed case. You now can do following on daily-use profile. - "Enpty Trash" at Feeds account. - Define JIRA RSS feed with Feed URL = https://mariadb.atlassian.net/activity Prolem determination work can be continued with newly created Tb's profile only. (a) JIRA RSS feed definition with basic authentication. > dc:identifier="https://mariadb.atlassian.net/activity?maxResults=5&amp;os_authType=basic&amp;title=undefined" (b) Copy of ...\Feeds\Trash.sbd\JIRA & JIRA.msf under ...\LocalFolders\JIRA & JIRA.msf. > 16:13:18 - focus moved to 'Feeds'; > 16:16:37 - 'Get messages' executed on Feeds; When you get NSPR log with newly created profile, do following, please. - focus moved to 'Feeds/JIRA' (force access to JIRA only) - 'Get messages'
Summary: Memory leak in idle state: memory usage grows at approx 1Gb/hour rate, when particular folders in RSS feeds are used, even though the feeds updates are currently disabled → Memory leak in idle state: memory usage grows at approx 1Gb/hour rate, when "RSS feed wth basic authentication which is held in Feeds/Trash" is used, even though the feeds updates are currently disabled
Summary: Memory leak in idle state: memory usage grows at approx 1Gb/hour rate, when "RSS feed wth basic authentication which is held in Feeds/Trash" is used, even though the feeds updates are currently disabled → Memory leak in idle state: memory usage grows at approx 1Gb/hour rate, when "RSS feed with basic authentication which is held in Feeds/Trash" is used, even though the feeds updates are currently disabled
Sorry, I missed following in your comment #34. > That's exactly right. I have: > 0 2013-12-17 04:41 JIRA > 11650272 2014-01-04 22:19 JIRA.msf > 0 2013-11-28 00:04 Trash.sbd\JIRA > 11669375 2014-01-05 00:12 Trash.sbd\JIRA.msf What happened was perhaps following; 1. JIRA RSS feed was accessed normally by Tb 17, then JIRA.msf was increased to 11 MB. 2. In Tb 24, RSS feed access failed due to basic authentication, then all feed item data in file named JIRA was removed. 3. After it, you deleted JIRA RSS feed, then files are moved to Feeds/Trash, and rss feed definition was removd from feeds.rdf. 4. And you re-defined the JIRA RSS feed with basic authentication on Tb 24, but feed access failed due to basic authentication. 5. You used Tb profile by Tb 17, then all feed items are normally stored, and JIRA.msf size was increated to 11MB, as done at step 2. 6. JIRA.msf and Trash.sbd\JIRA.msf was broken by step 2 and step 5. 7. Because of problem in feed access with basic authentication, data in JIRA.msf won't be updated. Because Trash.sbd\JIRA.msf and JIRA is already removed from feeds.rdf, server acess won't happen, then data in Trash.sbd\JIRA.msf won't be updated. Because JIRA.msf and Trash.sbd\JIRA.msf is broken, MsgDatabase Open can't be executed norally, then "MsgDatabase Open" process won't destory generated object. There are 3 problems. (1) "not working *subscription*" in Tb 24 due to basic authentication. (2) If problem (1) happens in Tb 24 on already accessed RSS feed with basic authentication, .msf file is broken. (3) If problem (2) happens, memory leak occurs. Both problem (2) and problem (3) is "problem after problem (1)" or "problem after problem ... after problem (1)". I don't think "detailed analysis of problem (2) and problem (3)" is fruitful. I believe we are better to focus on problem (1) only. elenst(bug opener), can you open separate bug for problem (1)? :
(In reply to elenst from comment #34) > Also: when I open Properties on the folder, the screen shows Name (JIRA), > but Location is empty. Consequently, the OK button does not work. "Folder" is internally set of "<folder> & <folder>.msf which is deterined by Server Settings etc. in prefs" and "objects which holds folder's meta data". Folder related objects which can be accessed via JavaScript are; > msgFolder : [xpconnect wrapped nsIMsgFolder] > msgFolder.filePath : [xpconnect wrapped nsIFile] > msgFolder.msgDatabase : [xpconnect wrapped nsIMsgDatabase] > msgFolder.msgDatabase.dBFolderInfo : [xpconnect wrapped nsIDBFolderInfo] > msgFolder.downloadSettings : [xpconnect wrapped nsIMsgDownloadSettings] > msgFolder.msgDatabase.msgRetentionSettings : [xpconnect wrapped nsIMsgRetentionSettings] msgDatabase corresponds to data saved in .msf file. "Location:", "Retention Policy" etc. in "Folder Properties" is shown from data saved in these objects. "button-1/Folder Info of Selected Folder" of my add-on dumps these object's properties to Error Console. "Where something wrong happens" may be guessed by following. Comparison of such data with normal folder, What property is not shown as expected, On which property exception occurs, Where exception occurs, Tb chaches or not, etc. If you are innterested in these objects, see it by the add-on. Please don't forget to uninstall after testing. "OK button does not work" may be similar phenomenon to "MsgDatabase Open doesn't complete( then memory leak happens)", because "View Folder Properties" also invokes internal "MsgDatabase Open" process.
Confirming per bug opener's observations and provided data.
Status: UNCONFIRMED → NEW
Ever confirmed: true
(In reply to WADA from comment #40) > How about following? > Copy ...<ProblematicDailyUseProfile>\Mail\Feeds\Trash.sbd\JIRA and JIRA.msf > to ...<NewlyCreateProblemaWith_JIRA_only\Mail\Local > Folders\Trash.sbd\JIRA and JIRA.msf Yes, there is still the problem, both with displaying the contents and memory usage (although memory usage grows slower than with the main profile, still it has grown 2+ Gb overnight). > Check ...\Feeds\Trash.sbd\JIRA(not JIRA.msf) content by text editor, and see > X-Mozilla-Status: headers, please. As mentioned before, the JIRA file has size 0 according to the file system, and text editors concur -- it's empty. > Anyway, as alta88 said in Comment #25, you have backup of profile which > produces memory leak problem. And, as alta88 said in Comment #25 and you > already saw, "no basic authentication" is simplest/easiest workaround in > JIRA RSS feed case. > You now can do following on daily-use profile. > - "Enpty Trash" at Feeds account. Done. Purging went a bit bumpy, but it worked at the end. There seems to be no memory leak anymore, at the first glance. > - Define JIRA RSS feed with Feed URL = https://mariadb.atlassian.net/activity (In reply to WADA from comment #41) > (1) "not working *subscription*" in Tb 24 due to basic authentication. > ... > elenst(bug opener), can you open separate bug for problem (1)? While setting up the new subscription, I figured out the reason of the authentication problem, at least the user side of it. Apparently, at some point authentication for the RSS feed stopped accepting an email address as a login (for the web interface it still works), so the whole 'Cannot find URL' thing was just an obscure indication of unrecognized credentials. And once it happens in the TB session, modifying the link does not help; it does not ask for credentials anymore, just throws the error right away, even if I remove authentication parameter from the URL. But if I restart TB, I can start from scratch -- that's how I was able to create the plain activity feed in the previous experiments, and now, when I was asked for credentials on the first attempt, I entered a user name only, it went through all right, and the feed was created. I tried it twice to confirm that it's not an accident. I don't know if it happens on the atlassian/ondemand side, or on the TB side, and whether it's a bug or a feature. Given that, do you still want a separate report for the problem? > (a) JIRA RSS feed definition with basic authentication. > > dc:identifier="https://mariadb.atlassian.net/activity?maxResults=5&amp;os_authType=basic&amp;title=undefined" > (b) Copy of ...\Feeds\Trash.sbd\JIRA & JIRA.msf under ...\LocalFolders\JIRA > & JIRA.msf. > > > 16:13:18 - focus moved to 'Feeds'; > > 16:16:37 - 'Get messages' executed on Feeds; > > When you get NSPR log with newly created profile, do following, please. > - focus moved to 'Feeds/JIRA' (force access to JIRA only) > - 'Get messages' Do you mean to run 'get messages' on the new RSS feed from (a), or on the copy of the broken folder from (b)? If it's (a), do you still want to see it, considering the above note about authentication? If it's (b), do I also need to edit feeds.rdf somehow? (If I only copy the files to the new profile, there will be no subscription associated with them, right)?
Please note the following, regarding the feeds part: 1. Move to trash/delete unsubscribes the url from the folder and removes all traces in feeditems.rdf and feeds.rdf. 2. Feed urls a)https://mariadb.atlassian.net/activity?maxResults=5&amp;os_authType=basic&amp;title=undefined and b)https://mariadb.atlassian.net/activity are 2 separate unique subscriptions. 3. Yes, if you enter a bad auth, the 401 return code is treated the same as a 404; on first use and auth dialog you know this and it won't subscribe. Later rejects of formerly successful subscribes are silent (though logged), like a 404. 4. The auth url pwd is probably found in Options->Security->Passwords->Saved Passwords. Page GET requests to an auth url are handled by core Fx code. When it went bad, there was no user feedback. There are some old feeds auth bugs and undoubtedly feed auth url processing could be improved. 5. Feed urls are easily updated in the Subscribe dialog and not by hacking feeds.rdf. 6. Feeds have extreme logging; set Feeds.logging.console to debug or trace (and restart). Regarding the memory issue, the bug is that a 0 length file with erroneous .msf gets loaded. If you have a legit very large folder and an 11M .msf, that's either not an issue or an issue of a different kind (design). I recall a change a while ago that closed the msgDb (.msf) after a period of non access to a folder. However, if a folder gets out of sync, the fix is Repair Folder (this is not a judgement on whether that's correct practice in client app design). The root cause of the 0 length file and its huge .msf will unlikely be discovered and may have non Tb reasons.
Component: Feed Reader → Backend
(In reply to alta88 from comment #45) Just for clarification -- do I understand correctly that there are no further requests for information from me in the above comment, and the whole thing has been classified as - regarding memory leak: a bug in how TB broken files (0-sized, in particular); - regarding getting the 0-sized file: unknown origin, possibly external reasons; - regarding not being able to connect with the email: not a TB problem; - regarding weird behavior after a bad connect: questionable design, not worth bothering about? If so, it sounds reasonable to me. I might try to reproduce the 0-sized file later, but I agree that external reasons are more likely (a disk problem or whatever). I also tend to think that the problem with authentication via email is on the atlassian side, they constantly change something. Regarding getting out of sync and handling of a 0-sized file, if it's of any importance for fixing, there was yet another related problem that I had to deal with while removing the broken folder on the previous step. When I did so (either by sending it to Trash or by emptying Trash), after TB restart the folder appeared to have gone; but it turned out that only the JIRA file got removed, while JIRA.msf was left behind. So when I created a folder with the same name in the same place again, the problem re-appeared. Eventually I had to remove the .msf file manually after purging the folder, only then the problem was gone for good. It wasn't a big deal since I already knew where to look from your previous investigation, but if I didn't, it would have been very confusing. It might also be that the root cause of getting out-of-synced files and 0-sized folders hides here somewhere, but I wasn't able to quickly reproduce it with "normal" (unbroken) folders, so I will stick to the HW theory for now. > 5. Feed urls are easily updated in the Subscribe dialog and not by hacking > feeds.rdf. FWIW, I've never done any hacking of feeds.rdf, or folder files, or *msf files. I didn't even know they existed until we started looking into this. They could be a subject of some unnoticed HW issue, though. > However, if a folder gets out of sync, the fix is Repair Folder > (this is not a judgement on whether that's correct practice in client app design). As you might remember, I wasn't able to do 'Repair Folder' in my case, it just did nothing.
FYI. Followng is Today's Browser log of Firefox 26. mariadb.atlassian.net looks to stop returning 401 for &os_authType=basic. > Request URL: https://mariadb.atlassian.net/activity?maxResults=5&amp;os_authType=basic&amp;title=undefined > Request Method: GET > Status Code: HTTP/1.1 200 OK > > Request Headers 08:23:02.000 > > User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:26.0) Gecko/20100101 Firefox/26.0 > Host: mariadb.atlassian.net > Connection: keep-alive > Accept-Language: ja,en-us;q=0.7,en;q=0.3 > Accept-Encoding: gzip, deflate > Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 > > Response Headers Δ1047ms > > X-Content-Type-Options: nosniff > X-ASEN: SEN-2153519 > X-AREQUESTID: 83x18255x1 > Vary: Accept-Encoding > Strict-Transport-Security: max-age=315360000;includeSubdomains > Server: nginx > Pragma: no-cache > Expires: Wed, 31 Dec 1969 23:59:59 GMT > Date: Mon, 06 Jan 2014 23:23:03 GMT > Content-Type: application/atom+xml;charset=UTF-8 > Content-Length: 2429 > Content-Encoding: gzip > Connection: keep-alive > Cache-Control: no-cache, no-store, must-revalidate 401 was not returned to Today's RSS feed access by Firefox. > Request URL: https://mariadb.atlassian.net/browse/CONC-64 > Request Method: GET > Status Code: HTTP/1.1 200 OK
Sorry. I pasted wrong URI for RSS feed(&amp; instead of &). 401 was still returned to &os_authType=basic.
(In reply to elenst from comment #46) > (In reply to alta88 from comment #45) > > Just for clarification -- do I understand correctly that there are no > further requests for information from me in the above comment, and the whole > thing has been classified as > - regarding memory leak: a bug in how TB broken files (0-sized, in > particular); > - regarding getting the 0-sized file: unknown origin, possibly external > reasons; > - regarding not being able to connect with the email: not a TB problem; > - regarding weird behavior after a bad connect: questionable design, not > worth bothering about? > in solely my opinion, that's pretty much it. the behavior after a failed auth (on a previously successful auth) could be changed by adding a different log error message, in the absence of bug 261841. as for authed feeds, Tb uses a different system from Firefox and its doorhanger offering to save site passwords, so there isn't any Saved Passwords management for sites like there is for other accounts. bug 267203. > > > However, if a folder gets out of sync, the fix is Repair Folder > > (this is not a judgement on whether that's correct practice in client app design). > > As you might remember, I wasn't able to do 'Repair Folder' in my case, it > just did nothing. it's possible things in Trash are handled differently. it's possible the code detected an empty folder without messages and returned. using the Tb UI is preferable to going to the filesystem. Empty Trash will ensure both folder and .msf are removed. Repair Folder is better than the ancient meme of deleting .msf (it auto rebuilds on folder select), and is worse for feeds than other account types due to losing the key folder-feedUrl relationship and falling out of sync in the edge case of moving/renaming a folder (if done before a resync on biff, new in Tb28). also, i don't think there's a regression here.
Keywords: regression, regressionwindow-wanted
Whiteboard: [closeme 2014-02-01][regression:TB22?] → [closeme 2014-02-01]
Summary: Memory leak in idle state: memory usage grows at approx 1Gb/hour rate, when "RSS feed with basic authentication which is held in Feeds/Trash" is used, even though the feeds updates are currently disabled → Memory leak in idle state: memory usage grows at approx 1Gb/hour rate, when empty folder with very large .msf, in Trash, is found
FYI. (In reply to alta88 from comment #45) > 2. Feed urls > a) https://mariadb.atlassian.net/activity?maxResults=5&os_authType=basic&title=undefined > and > b) https://mariadb.atlassian.net/activity > are 2 separate unique subscriptions. They are different URLs in feeds.rdf and feeditems.rdf. However, these are same site and a part of same feed in this bug's case. (case-1) Because (a) and (b) is same URL, password manager entry is shared. In Firefox, if URL (a) was accessed and username/password was not provided at prompt, access to (b) was done with "Authorization: Basic Og==", and 401 was returned from server. This may be Cache relivant phenomenon. > Request URL: https://mariadb.atlassian.net/activity > Request Method: GET > Status Code: HTTP/1.1 401 Unauthorized > > Request Headers 10:38:24.000 > > User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:26.0) Gecko/20100101 Firefox/26.0 > Host: mariadb.atlassian.net > Connection: keep-alive > Authorization: Basic Og== > Accept-Language: ja,en-us;q=0.7,en;q=0.3 > Accept-Encoding: gzip, deflate > Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 (case-2) If following RSS folders were defined in one Tb, second folder was not updated, and Feed URL field was cleared after a while. (c) FolderA: Feed URL = https://mariadb.atlassian.net/activity?title=Hello&maxResults=50 (d) FolderB: Feed URL = https://mariadb.atlassian.net/activity?maxResults=80&title=GoodBy (case-3) If following Feed URLs was defined in one RSS folder in Tb, duplicate entries appeared, because same RSS feed item is loaded as feeditem of different RSS feed. (e) FolderX: Feed URL = https://mariadb.atlassian.net/activity?title=Hello&maxResults=50 (f) FolderX: Feed URL = https://mariadb.atlassian.net/activity?maxResults=80&title=GoodBy Is "A_feed_site?title=Hello" like one different RSS feed? Or a part of a same RSS feed? (case-3) is similar to "cross posts in new is not marked as Read by Tb". Note-1: Because this bug's case is "one deleted RSS folder under Trash and one active RSS folder with basic auth", case-2/case-3 are irreleant to this bug's problem.
(In reply to alta88 from comment #49) > also, i don't think there's a regression here. I think following part is new problem in Tb 24. > (1) "not working *subscription*" in Tb 24 due to basic authentication. I think, if "subscription with basic auth" is successfull in Tb, "Repair Folder" etc. works even if "file sze=0" was somehow generated, and .msf file, feeditems etc. is normlly cleand up, and subsequent memory leak won't occur because subsequent "MsgDatabase Open" is normally executed. Is it right?
About "in Trash" in bug summary? Trash(deleted RSS feed) only problem? As seen in MSGDB:4 log and file size data, "MsgDatabase Open won't complete" occurred also on active RSS feed with basic auth(access fails, file size=0 and large .msf was already generated.) Trash(deleted RSS feed) unique issue is; There is no way to recover from memory leak such as Feed URL: change, except "Empty Trash".
(In reply to WADA from comment #51) > (In reply to alta88 from comment #49) > > also, i don't think there's a regression here. > > I think following part is new problem in Tb 24. > > (1) "not working *subscription*" in Tb 24 due to basic authentication. > I think, if "subscription with basic auth" is successfull in Tb, "Repair > Folder" etc. works even if "file sze=0" was somehow generated, and .msf > file, feeditems etc. is normlly cleand up, and subsequent memory leak won't > occur because subsequent "MsgDatabase Open" is normally executed. > > Is it right? I don't think whether auth succeeds or not has anything to do with the the 0 file large .msf. It's no different from a 404 Not Found response. (In reply to WADA from comment #52) > About "in Trash" in bug summary? > Trash(deleted RSS feed) only problem? > As seen in MSGDB:4 log and file size data, "MsgDatabase Open won't complete" > occurred also on active RSS feed with basic auth(access fails, file size=0 > and large .msf was already generated.) > Trash(deleted RSS feed) unique issue is; > There is no way to recover from memory leak such as Feed URL: change, > except "Empty Trash". The feeds system, if it encounters an unavailable database, will schedule a reparse/rebuild, and process on the next cycle. If a user selects a folder that has gone stale, the same thing happens immediately. I also don't think this is a 'memory leak'. A large .msf will certainly be loaded as part of normal operation. The case here is an incorrectly large .msf which isn't being corrected. If you have a very large folder and also large .msf, then delete everything in the folder file (empty the file via an editor, not Tb delete), selecting the folder will rebuild .msf to a small size.
elenst, 1. Earlybird has some improvements in crash handling and folder/file handling that may make it worth using - available from http://www.mozilla.org/en-US/thunderbird/channel/ 2. Do you continue to have this problem after empty trash/deleting bad .msf files? 3. What antivirus software do you use, and is the Thunderbird profile directory explicitly excluded from scanning?
Whiteboard: [closeme 2014-02-01] → [closeme 2014-02-15]
(In reply to Wayne Mery (:wsmwk) from comment #54) > elenst, > 1. Earlybird has some improvements in crash handling and folder/file > handling that may make it worth using - available from > http://www.mozilla.org/en-US/thunderbird/channel/ Thanks, I will keep it in mind, although generally I try to stay away from non-stable products unless I am to test them. > 2. Do you continue to have this problem after empty trash/deleting bad .msf > files? I kept having the problem when I just emptied the trash, but it disappeared when I removed related files via the file system. I have not had the problem since then. > 3. What antivirus software do you use, and is the Thunderbird profile > directory explicitly excluded from scanning? I had been using AVG over the whole period since before the problem started and till after I removed the files and confirmed that the problem disappeared. The Thunderbird profile directory was not explicitly excluded from scanning.
Resolved per whiteboard
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → WORKSFORME
Whiteboard: [closeme 2014-02-15]
Blocks: 1330872
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: