1078367 - allow maildir backed mail folders to exceed 4GB in size [meta]

Status: RESOLVED FIXED

(MailNews Core :: Database, enhancement)

Description

[:aceman]

I think maildir backed folders can get to support >4GB folders sooner than mbox backed ones (because they hopefully not use internal 32bit file offsets as mbox uses).

It seems we just need to:
- allow sizeOnDisk to be 64bit: bug 1032360 or bug 789679
- allow the 64bit value of size on disk to be properly reported to JS consumers, e.g. via OnItemIntPropertyChanged() - bug 813459

After the above is done we need to do here:
- remove the 4GB limit inside nsMsgMaildirStore::HasSpaceAvailable()
- adapt/clone the test mailnews/local/test/unit/test_over4GBMailboxes.js to work on maildir (at least the parts that apply to maildir). Currently it is very mbox specific.

Comment 1

[:aceman]

This bug only covers the 4GB+ feature of maildir folders. It does not mean that when this bug is done, maildir can be suddenly exposed. The remaining bugs in maildir still need to be fixed and those are tracked in bug 845952.

Comment 2

[Andrew McGlashan]

I've got a folder that, on the NTFS file system which has compressed flag, has size on disk at 5.65GB with a storage size of 8.58GB.

Have I misunderstood this issue?

Comment 3

[:aceman]

And that folder is (In reply to :aceman from comment #0)
> It seems we just need to:
> - allow sizeOnDisk to be 64bit:  bug 789679
Done.

> - allow the 64bit value of size on disk to be properly reported to JS
> consumers, e.g. via OnItemIntPropertyChanged() - bug 813459
Done.

Bug 1032360 is open yet, to properly report the folder size inside of TB.

After that one, we can proceed onto updating the tests.

Comment 4

[:aceman]

(In reply to Andrew McGlashan from comment #2)
> I've got a folder that, on the NTFS file system which has compressed flag,
> has size on disk at 5.65GB with a storage size of 8.58GB.
> 
> Have I misunderstood this issue?

Is that folder used as maildir backend for a mail folder inside TB? I assume you created it in TB31.x . Does it still work in TB38? I think the checks were tightened in TB38 so it may report folder full. Or not, depending on what size the folder reports (bug 1032360).

Comment 5

[Andrew McGlashan]

I first started to use maildir with TB38.0.1 -- I've been wanting it forever it seems.

Those size are directly from the file system folder inside the TB profile for the ImapMail account.

That is  %profiledir%\ImapMail\mailserver_domain\INBOX\cur

Inside TB, it reports as having size 5.0 GB in properties for "size on disk".

Actually the sizes have changed on disk (perhaps properties before wasn't complete) -- as in select folder and then properties from context menu in Explorer.

I see 11.2 GB of files using 7.57 GB of storage (compressed).  I've got another serious bug to report though as there is actually 156K of emails (count), but over 350K of files in the "cur" directory right now after a "rebuild" action.  And it seems that number keeps growing.  TB itself is reporting the correct count, not sure if the 5.0 GB wold represent the actual storage of the actual emails without the duplication of files that is taking place.

Comment 6

[Andrew McGlashan]

Okay, repairing the folder left what was there before (200K) and created new files for the 156K of emails it re-downloaded from the server.  That's a bad repair problem.....

Comment 7

[Andrew McGlashan]

I started with a fresh profile in order to use maildir storage, set everything up fresh.  So far I've only loaded one email account, unfortunately there still seems to be lots of issues, even though it has been released in 38.0.1 :(

Comment 8

[:aceman]

The inconsistent number of files in the folder is a bug reported elsewhere.

I think the Properties of the folder inside TB shows the size on server, regardless of what is in the local filesystem folder.

So if you say your local size is 11GB that may be because due to bug 1032360 TB thinks the size is some tiny number so that it does not exceed 4GB. But when we fix bug 1032360 TB may start to report the folder being full. But that probably won't happen until TB45 :) And since then, we may officially remove the 4GB limit for maildir.

But I speak for POP3. The situation on IMAP is slightly more complicated.

I just wanted to say, that I do not think TB38 officially supports 4GB+ folders, regardless of backing store. It may incidentally work but is not guaranteed yet.
Have you come across any official sources claiming folders larger than 4GB to be supported in TB38?

Comment 9

[:aceman]

(In reply to Andrew McGlashan from comment #7)
> I started with a fresh profile in order to use maildir storage, set
> everything up fresh.  So far I've only loaded one email account,
> unfortunately there still seems to be lots of issues, even though it has
> been released in 38.0.1 :(

It is a pity no such problems were reported while TB38 was in development.

Comment 10

[Andrew McGlashan]

Okay, the Linux server has 5.3 GB according to "du -sch cur".

Call me an ass, but I assumed that storage would be limited by filesystem limitations and not by anything in TB.  I know of no reference claiming to support any size for a folder.

Why would you want to allow a false "full" situation to occur?  My entire Maildir folder on the server is about 46 GB.

Comment 11

[:aceman]

(In reply to Andrew McGlashan from comment #10)
> Call me an ass, but I assumed that storage would be limited by filesystem
> limitations and not by anything in TB.  I know of no reference claiming to
> support any size for a folder.
> Why would you want to allow a false "full" situation to occur?  My entire
http://kb.mozillazine.org/Limits_-_Thunderbird
Sure, everybody would love there to be no limits. But there is some old code that assumed some limits and hardcoded variable sizes. So we slowly work through that legacy code and update it.
In coding there always are some limits and the coder must choose them beforehand. Sometimes the world outgrows those limits and program must be updated.

Comment 12

[Andrew McGlashan]

Okay, so reading that limits reference, maildir isn't a problem for the 4GB issue.

"Maildir stores each message as an individual files, i.e. does not combine messages into a single file, and thus bypasses the 4GB folder size as well as eliminates any need to compact a folder."

Comment 13

[Andrew McGlashan]

Given the extreme slowness and other problems, I've reverted back to not saving messages locally.  This is so disappointing.  I gave up on mbox due to the troubles, now I'm giving up on maildir.  Perhaps the common problem of how .msf is handled might be the reason for the slowness; but I have nothing definitive.  Just a very poorly performing Intel Xeon X3360 (I know a bit old, but the machine shouldn't run like a slug) box with 8GB of RAM, sometimes of which TB took close to 3GB while trying to get messages sorted in the local store or whatever it was doing.  I'm using a couple of 2TB WD blacks (RAID1) with an LSI controller, so that shouldn't be a problem either.

With maildir I couldn't get folders to properly [sync] reflect the count of emails on the server.  Sometimes there were significantly more orphaned files than the count on the server; the server count was the same as being reported by TB though.

Comment 14

[:aceman]

We actually allowed maildir and also mbox folders to grow above 4GB folders in bug 789679 now.
The bugs blocking this bug aren't completely closed yet, but the partial problems needed to get folders above 4GB are solved in them.
So the description of this bug as filed is solved now.