280793 - Warn if local mail folders get close to 4GB

Status: RESOLVED FIXED

(MailNews Core :: Backend, defect)

Description

[Jungshik Shin]

I've seen a few reports of Korean mozilla users that thunderbird can't handle
local folders larger than 2GB even on NTFS. This is likely to be a dupe, but my
search failed.

Comment 1

[David :Bienvenu]

I've heard reports of this. I had it working at one point, but it seems like it
has regressed. I need to recreate this. I doubt it has anything to do with
Korean users, but has to do with a signed/unsigned issue to do with message keys.

Comment 2

[Jungshik Shin]

adding cbie because he recently worked on large file handling. Perhaps, his fix
for bug 264599 has already fixed this problem unless msgDB/RDF has some more
fixes to go beyond 2/4GB size limit (basically, need to use 64bit int)

as for Korean users, I just mentioned it because that's where I heard of it.
Needless to say, it cannot possibly anything to do with Korean users :-)

Comment 3

[Jungshik Shin]

sorry for bug spam. if it's something to do with message keys, this must be an
issue on all platforms. 

Comment 4

[David :Bienvenu]

yes, it has to do with message keys, and even with my fix, we only go to 4GB,
not 2GB. Message keys are 32 bit, and converting them to 64 bits would be
challenging...at the time that I fixed things to work with 4GB files, the
underlying necko interfaces didn't work with 64 bit values so I didn't take that
challenge :-)

Comment 5

[Christian :Biesinger (don't email me, ping me on IRC)]

note bug 207400

I'd be happy to fix all necko interfaces that do not support 64-bit ints,
especially if they are unfrozen

Comment 6

[David :Bienvenu]

Jungshik, were these users using 1.0, do you know? I tried this on the trunk,
and it worked OK - see bug 207400. This should be working in 1.0 as well.

Comment 7

[Jungshik Shin]

They said they had problems with 1.0, but they may have meant 4GB. I asked them
to clarify and will get back here to report.

Comment 8

[Christopher W. Curtis]

I'm having a similar problem on Linux.  My 'About->Mozilla' string is:

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.8) Gecko/20050513 Debian/1.7.8-1

And my filters are failing.  The filesizes are:
 2147512157 May 25 14:37 Mail
          0 May 25 14:56 Mail.msf

It's not a filesystem issue, I also have a procmail'd version (unfiltered):
 3571697366 May 25 14:48 mbox

This is from the suite, not thunderbird specifically.

Comment 9

[timeless]

[e-mail]: can you try mozilla trunk instead of the 1.7 branch?

Comment 10

[Dan Mosedale (:dmosedale, :dmose)]

If a user's Inbox hits the 2G limit, will there be dataloss?

Comment 11

[Benjamin Smedberg]

In my experience on the 1.7 branch, yes: the mail file ends up being corrupted
to the point that rebuilding the index file kills everything.

Comment 12

[Dan Mosedale (:dmosedale, :dmose)]

Unless this code doesn't have the dataloss problem on the trunk that it has on the branch, I'd suggest this 
block 1.1.

Comment 13

[Asa Dotzler [:asa]]

transferring nomination to 1.8b4. If we don't get it there, we don't get it for 1.1

Comment 14

[David :Bienvenu]

on the trunk, the limit is 4GB. I have a patch that warns the user when they're
within 1MB of that limit for local folders.

Comment 15

[David :Bienvenu]

Attachment: warn if local folders get with 1MB of 4GB

Comment 16

[Benjamin Smedberg]

Comment on attachment 191372 [details] [diff] [review]
warn if local folders get with 1MB of 4GB

The wording of the dialog sounded awkward, so I asked MikeB to suggest some
alternatives:

<beltzner> I got two suggestions:
<beltzner> friendly-but-less-specific:
<beltzner> The folder %S is full, and can't hold any more messages. To make
more room, delete any old or unwanted mail and compact the folder.
<beltzner> tecchie-but-exact:
<beltzner> The folder %S has reached the maximum possible size (4GB) and can't
hold any more messages. To reduce the size,  delete any old or unwanted mail
and compact the folder.

Comment 17

[David :Bienvenu]

I prefer the non-techie one, with one tweak - instead of "To make more room",
I'd prefer "To make room for more messages", since there currently isn't any
room, so the phrase "more room" isn't quite right.

Comment 18

[David :Bienvenu]

fixed (in the sense that we warn the user and don't let the get new mail or msg
copy run).

if and when we support a single msg per file message store, then this won't be
an issue.

Comment 19

[Alex Ihrig [:Thunderbird_Mail_DE]]

We could suggest to create subfolders and move old messages to these subfolders
instead of deleting them.

Comment 20

[Christian Schmidt]

I haven't tested the patch myself, but I can see that the filesize check is done
_before_ the message is received or copied, i.e. it does not take the size of
the message being received/copied into account.

If I receive/copy a message larger than 1 MB (the warning threshold), wont that
cause the file to grow beyond 4GB without warning the user?

BTW I suggest leaving this bug open until the problem described in the summary
is eventually fixed.

Comment 21

[David :Bienvenu]

yes, that's true - however, time was short for 1.5 and I wanted to get the new
string in before we stop taking new strings. I think that's an edge case we can
worry about in beta - things should be much better than they were before. Also,
what's important is where the message starts, so as long as it starts before the
4GB mark, it should be OK.