497554 - A folder can be grown over 4GB without any warning at all (and results in problems.)

Status: RESOLVED DUPLICATE of bug 387502

(Thunderbird :: General, defect)

Description

[ISHIKAWA, Chiaki]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; ja; rv:1.9.0.10) Gecko/2009042315 Firefox/3.0.10
Build Identifier: version 2.0.0.21 (20090302)

I am reporting this problem  which was origianlly experienced
by a colleague of mine using thunderbird on Windows XP 32bit platform, but
I could re-produce this on linux i86 32bit platform under controlled environment.

(There is now a bug called 387502, which is similar. But it was marked "unconfirmed" and
never followed up. It happened under windows with an earlier version.
Mine is on a different platform with a recent version and I am attaching a
test file for this.)

Fact: 
It seems that there is a 4GB size limit of folder.
Unforatunately, TB doesn't seem to properly keep this limit under control and
allows the operations that make folders grow over this limit. Eventually, the
user experiences problems (not being able to read messages, losing messages.)

My action below is by copying a message into a folder, but
my colleague's case was a large Sent folder.
He had no way of knowing that sending a message (with an attachment that
is largish) will go over the 4GB limit of folder, in this case, Sent folder,
and was quite perplexed and as a result lost a day's worth of sent messages.

I am attaching a file to re-create the problem.
unpack Seed-for-big-folder
And run the following command with the other script file (your directory
portion of the pathname shold be changed)

gawk  -f /extra/ishikawa/download/create-dummy-mails.awk Seed-for-big-folder

A large file (close to but less than 4GB ) is created.
(The script creates a file consisting of slightly changed copies of a single largish message
that contains an attachment. Each message is about 3.x MB.)
Then follow the steps below.
Seed-for-big-folder.gz
 



Reproducible: Always

Steps to Reproduce:
1. Place the "dummy-file" in your Mail directory (where Sent, Inbox, et al are located.)
   Also place "Seed-for-big-folder".

2. Start thunderbird and see it rebuilds the summary for this new folder files.
   It can take an hour on a slow computer (1GB, 512MB PC built in circa 2002.)
   especially for "dummy-file".

3. Notice that "dummy-file" holder it has 1342 messages.

4. Repeat this process and notice that TB doesn't show any warning.
   (repeat three times. I did this three times.)

   4-a: Pick a message from Seed-for-big-folder, and move or copy it to
        dummy-file folder.
   4-b: You may wanto to check that dummy-file grows over 4GB during this oepration.
   4-c: Check the number of messages indeed increased by one.
   Quitting and restarting TB.

5. Now after three such operations, I find that
The message counter of dummy-file showed 1345. 
But upon closer inspection, the messages shown 
for the 1343th, 1344th, and 1345th,
are all garbled. 
1343th seems to be more or less OK.
1344th showed a portion (near the header) of the second message in
the original folder file. (I think lseek() or friends is wrapping the
search position module 4GB).
Ditto for 1345th file. (This shows the partial header portion of
the THIRD message in the original folder file. Again search position is wrapped module 4GB)

Actual Results:  
As mentioned, the addition of messages to grow a folder over 4GB succeeded without
any warnings.

But as a result, the messages thus added are not quite accessible any more.
(If such messages are created for sending and this copying occurs during
send operation and Sent grows over 4GB, the user loses precious messages!)


Expected Results:  
Either I should 
 - get the warning that the said copy/move operatoin grows the folder
   over 4GB and is kept from performing such operation.
   (I wonder though, what should happen when one sends out the message
    and copying takes place AFTER the message is sent anyway.
    Maybe we should create Sent-2 or some such folder and keeps the
    just sent folder there? But showing the WARNING/ERROR message
    is clearly required, and TB fails this.)

2. - be able to use a folder larger than 4GB (but this probably is a wish list.)

But then again, TB needs to force 4GB limit everywhere and should not
permit an operation that happens WITHIN TB to limit such 
problems.

Also, since the 4GB limit seems to be very difficult to enformce from what I had
read in bug 387502 report,
I would suggest that on systems that permits large than 4GB file,
TB should check for filesize >= 4GB in strategic places,
and warn the user of potential problems after the fact.


IMHO, the current failure of TB to enforce 4GB and
failure to preserve messages is a fatal error.

But more perplexing is the FAILURE to offer any warning/error message at all when that happens.


We need to make two efforts
(1)  - to prevent and WARN the user when folder size is going to
   be over 4GB, and

(2) - to detect that a folder has a filesize >= 4GB after the fact,
      and WARN the user to take remedical action.

Comment 1

[ISHIKAWA, Chiaki]

Attachment: A gawk script to create a large mail folder based on an input file (attached separately).

Test script to create "dummy-file", a large (close to, but less than 4GB)
mail folder from command line.
It takes as its input as a file that contains a largish mozilla message. 

gawk  -f create-dummy-mails.awk Seed-for-big-folder

Seed-for-big-folder is attached separately.
The script is adjusted for the first message in Seed-for-big-folder to create a
large (< 4GB) folder file, which grows over 4GB.

Comment 2

[ISHIKAWA, Chiaki]

Attachment: An input file for the script to create a large (close to, but less than 4GB) file.

use this as the first argument to the script attached separately after
unpacking (bzip2 -d ).
gawk  -f create-dummy-mails.awk Seed-for-big-folder

Comment 3

[ISHIKAWA, Chiaki]

I had to trim Seed-for-big-folder due to the upload limit size.
It now has only two messages. 
So you should "copy" (instead of "move") the message in the
said folder (mentioned in 4-(a) above)
since there are only two messages now.

TIA

Comment 4

[ISHIKAWA, Chiaki]

If thunderbird has a checking code for the growing size of a folder like this,
it may have a problem on a platform where C/C++ compiler doesn't support
truly 64 bit size item.

unsigned csize --- current size of the folder
unsigned msize --- message size (to be added)
unsigned nsize --- new size of the folder

nsize = csize + msize;

To avoid potential 32bit only arithmetic problem,
we need to add the check for overflow of the addition, etc..

if ( (nsize < csize) || (nsize < mszie) ) {
   /* overflow. 2^32 bit is reached. */
   
   warn the user and takes care of the 4GB limit.
}

The above check works even if there is no 64 bit arithemtic
available on the platform.

Well, if TB lacks an internal API that doesn't 
tell the folder (or the routine to handle
the copy/move of a message),  the size of the incoming message,
then tough lack...

Comment 5

[Joshua Cranmer [:jcranmer]]

(In reply to comment #0)
> (There is now a bug called 387502, which is similar. But it was marked
> "unconfirmed" and
> never followed up. It happened under windows with an earlier version.
> Mine is on a different platform with a recent version and I am attaching a
> test file for this.)

You *should* have added your comments to bug 387502 itself if it is in fact the same bug. Differing versions and platforms almost never make it a separate bug.


> We need to make two efforts
> (1)  - to prevent and WARN the user when folder size is going to
>    be over 4GB, and

According to bug 387502, we do warn the user. Reading the code, we will throw a warning if there is less than 1 MB of free room left.

> (2) - to detect that a folder has a filesize >= 4GB after the fact,
>       and WARN the user to take remedical action.

That would be bug 387502, then.

Comment 6

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #0)
> User-Agent:       Mozilla/5.0 (X11; U; Linux i686; ja; rv:1.9.0.10)
> Build Identifier: version 2.0.0.21 (20090302)

> gawk  -f /extra/ishikawa/download/create-dummy-mails.awk Seed-for-big-folder
> A large file (close to but less than 4GB ) is created.
> (The script creates a file consisting of slightly changed copies of a single
> largish message that contains an attachment. Each message is about 3.x MB.)

Mail offset in mail folder file is 32bits signed integer on Linux and Mac OS X.
(4GB support, 32bits unsigned integer use, is still implemented on MS Win only.)
So, if you externally create mail folder file greater than 2GB, I guess rebuild-index generates incorrect offset value(wraparound) for mails placed at larger than 2GB. It's same when "mail folder file greater than 2GB by MS Win" is mounted on Linux. It's current limitation/restriction.

Does Tb 2 on Linux create local mail folder file greater than 2GB by mail copy of Tb between mail folders?

Comment 7

[ISHIKAWA, Chiaki]

(In reply to comment #5)
> (In reply to comment #0)
> > (There is now a bug called 387502, which is similar. But it was marked
> > "unconfirmed" and
> > never followed up. It happened under windows with an earlier version.
> > Mine is on a different platform with a recent version and I am attaching a
> > test file for this.)
> 
> You *should* have added your comments to bug 387502 itself if it is in fact the
> same bug. Differing versions and platforms almost never make it a separate bug.
> 
> 
> > We need to make two efforts
> > (1)  - to prevent and WARN the user when folder size is going to
> >    be over 4GB, and
> 
> According to bug 387502, we do warn the user. Reading the code, we will throw a
> warning if there is less than 1 MB of free room left.
> 

I will switch t bug 387502 after this post.

But from what I learned from the colleague who has a Sent folder reaching and
going beyond 4GB, TB didn't seem to have thrown the warning/error at all.
If it did, I think he noticed and took a remedial measure, but
he didn't.

My creating a large file externally is to try to re-create my 
colleague's condition.

I suspect that the warning code is buggy somewhere 
since I can pleasantly create a slightly less than 2GB file
and copy messages into it without any ill-effect at all
to go beyond the 2GB limit on Linux.

Please set the number of messages variable in the BEGIN section of the awk script by half,
(that will create close to, but less than 2GB file)
and try copying messsages into that folder (after placing it
in the Mail subdirectory),
TB happily (without any warning/error) accepts messages until it goes beyond 2GB.

Then from what I read here, it is also a buggy behavior and
TB should have issued warnings, etc..



> > (2) - to detect that a folder has a filesize >= 4GB after the fact,
> >       and WARN the user to take remedical action.
> 
> That would be bug 387502, then.
> 
> *** This bug has been marked as a duplicate of bug 387502 ***

OK, I will follow up with 387502, but one reason I didn't follow it up was
that the copying done by sending (or receiving presumabley) requires quite a different
error recovery in terms of UI and thus I felt that my colleague's
situation where Sent grew larger than TB can handle without HIS being aware of it
requires special attention IMHO.

Anyway, I will go to 387502 hereafter.

Oh, to the final question in #6, TB creates larger than 2GB mail folder by
copying messages happily without any error/warning from what I saw.

> Reading the code, we will throw a
> warning if there is less than 1 MB of free room left.

There must be a subtle bug in there. (comparison of signed and unsigned,
going over 2^32, etc..)

TIA.

Comment 8

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #7)
> Oh, to the final question in #6, TB creates larger than 2GB mail folder by
> copying messages happily without any error/warning from what I saw.

I saw it(larger than 4GB if MS Win) a few times several months ago during my test, but I could't see it these several months.
I could see it again on MS Win by next test at last.
(1) File-A: (2559) mails * (32KB per a mail) => File-Size=80MB,     no File-A.msf 
    File-B: (64-1) mails * (64MB per a mail) => File-Size=4GB-64MB, no File-B.msf 
(2) Restart Tb, click File-A to force Rebuild-Index.
(3) Never touch File-B, so File-B.msf is not created.
    - emulation of "delete of msf" due to "outdated msf condition"
    - emulation of Save in Drafts/Sent, Junk move, Filter Copy/Move etc.
      without explicit mail folder open of Copy/Move target folder.
(4) Create Message filter : If subject contains xxx, Copy to File-B.
    It copies all mails in File-A to File-B.
(5) Run Filter on File-A
    => File size of File-B increased to 4,311,718,368 bytes(4.1GB).
       No warning of "exceeds 4GB".
(6) Run Filter on Folder-A again.
    Warning of "exceeds 4GB", and copy was not executed.