Closed Bug 1261357 Opened 10 years ago Closed 10 years ago

too many objects in local folder? folder apparently full not taking in any more pop3 fetched messages via filter

Categories

(Thunderbird :: Folder and Message Lists, defect)

Product:
Thunderbird
Component:
Folder and Message Lists
Version:
38 Branch
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: abittner, Unassigned)

Details

User Agent: Mozilla/5.0 (Windows NT 5.1; rv:45.0) Gecko/20100101 Firefox/45.0 Build ID: 20160315153207 Steps to reproduce: too many objects in local folder? folder apparently full not taking in any more pop3 fetched messages via filter very long-term thunderbird user using pop3/smtp only having local folder structure have one folder where mailing-list message arrive for many years today three or four mails arrived and an error dialog came up on thunderbird 38.7.1 telling me that there was something wrong with that local folder while writing it or so, I dont exactly recall the error dialog any more. I noticed that three or four mailing-list messages stayed in the inbox folder of thunderbird. I looked at my mailing-list folders properties inside the thunderbird tree and it displayed roughly 107675 mails, 625megabytes total for that folder. Now these figures dont ring a bell to me as to maybe some 32bit counter limit or storage space limit or anything related, so I was wondering if there is some object count limit or so hidden somewhere in the source code or what wall I was hitting? I eventually removed a few thousand mails from that mailing-list folder to a newly created mailing-list-old or similar, and then via drag-drop I was able to move those mailing-list messages from the inbox folder of Thunderbird to then slightly cleaned up mailing-list folder again afterwards. I dont know if there are any detailed logs kept anywhere of the current issue, later on I removed like 70thousand mails from the original folder to the newly created one, and talking about Thunderbird bugs here, anyone using Thunderbird with a lot of objects is going nuts as the gui becomes highly locked and nonresponsive and it is no fun at all to deal with a lot of objects in folders in Thunderbird, so this is yet another huge field of software engineering that could need lots of enhancements and speed-ups :( Anyways, any way to help with this bug, or is this a known limitation and such? Thanks.
I can maybe even move back those tens of thousands of mails again and re-create the situation to see if there is a limit imposed again, and maybe even supply the content/file of the mailing-list folder as its mostly(?) public mailing-lists' content but I dont know if I should and it takes really a long time to mess with those multiple tens of thousands of objects with Thunderbird so I would be happy if somebody could maybe automate or script a testing to generate some tens of thousands of objects for local folders or something instead. Thanks.
What was the exact error message? There is e.g. a limitation for the size of a single folder and that is 4GB. That includes messages that you have deleted after the last Compact of that folder. Can you look at that?
no there was a simple popup created by the thunderbird gui itself only with okay button or so, which said that it could not write to that mailbox or it could not be modified or similar to that. i am not sure any more. all pop3 fetches during that state or from that moment on which would have landed messages in my mailinglist folder would fail with that popup. as i said i am not aware of any limits either which i could think of as a reason for this error, but as soon as i had originally moved away only some thousand messages to another folder my original mailinglist folder would again take messages in via the filter and i have given the number of messages that the folder had during that state when it was apparently filled or blocked and its size. thats all i can give as in terms of raw numbers. what now?
update: yesterday I moved back 70thousand something mailinglist mails from the secondary folder i created back to my original main mailinglist folder. Eventually all mails moved back normally without errors. Meanwhile i have more mails in the original mailinglist folder than when this weird bug started, and i have had mails arriving via filter into the original mailinglist folder as well normally. Maybe some other component caused the bug or some extension, I deactivated some extensions in thunderbird, such as dkim and some others. Maybe those cannot handle the number of mails in a folder normally, or we have race conditions or something. If this stuff comes up again I will add to this bug.
Yes, thanks for reporting. The limit of number of messages in a folder is 2^32 (32bit counter), which is 4 billion, so that is probably not what you hit. Also you say you did not hit the 4GB folder size limit. You could try to enable the "Order received" column in the message list. You observe the numbers in that column. If those are nearing 2^32 (4 billion), that may be the problem.
I have that column and it currenty has 655million+x as a value. I always had thought that it was actually maybe some kind of a unix timestamp or similar. What is the value exactly supposed to be and how does it get constructed? Some messages have way different values from that 655million. Some three messages from a week ago or so have all 240million+x in that column.
It an internal message identifier (key), that is just unique and always increasing. Each new message gets a new key. The key is not changed when the folder is compacted. For legacy reasons, some operations set the key equal to the offset of the message in the storage file. But some operations (e.g. move by filter), do not tie it to the offset, just increment it by 1. We are slowly moving to the "always increment by 1" scenario in all cases. IN TB45 (that should be release in days), this should actually be finished.
still no repeat of the error message? (looking for exact wording)
Flags: needinfo?(abittner)
No I have not had this bug/behavior ever since I have "fixed" it by moving out a few tens of thousands of messages to another subfolder and back again, sorry.
Flags: needinfo?(abittner)
Moving messages back and forth does affect the key (comment 5) of them so that could have solved the issue. Please reopen the bug when the problem reappears and then please paste the exact message you get. The message today should make clear whether the folder is full (due to the 4GB limit), or if the disk is full (out of free space). Also please use TB45+ which manages the key by incrementing by one in all cases.
Status: UNCONFIRMED → RESOLVED
Closed: 10 years ago
Resolution: --- → WORKSFORME

It was mentioned before that the limit of number of messages in a folder is 4 billion.
It this the same for both mbox and maildir storage formats?
And was the limit same in earlier versions of TB (particularly in 45.5.1)?
Thanks.

Yes, the limit of number of messages is still there and is because of some internal counters. Both mbox and maildir format could support more messages, but they can't be referenced due to the TB counters/keys. However, in maildir your filesystem would struggle if you approached 4 billion files in a single folder (directory). But I doubt you would have so many real messages.

However, in contrast to older TB versions, the 4GB folder size of mbox format was already removed in current versions.
vinist, you haven't commented in this bug before, what is your particular problem?

Thank you for the answer!
I don't have any problem now, while I had a problem once earlier when unexpectedly reached maximum size of a folder without knowing at that time that the limit was 4 GB.
So I wanted to make sure I know the limit of number of messages in a folder too - in order not to run once into overflow here. And the only place I found this limit announced was your comment #5 in this discussion, and so I wanted to clarify if this was for both mbox and maildir.
Now it's clear.

There is a wiki page conveniently listing the interesting size limits in Thunderbird:
http://kb.mozillazine.org/Limits_%28Thunderbird%29

Thanks for confirming you do not see this bug reappear.

Thank you.
Yes, I know this page, I've found it before, when searching for the info, but when it says on the limit of messages in a folder it references with a direct link to your comment #5 in this discussion :-)
So again, the comment in this discussion seems to be the only available source on the subject.

You need to log in before you can comment on or make changes to this bug.