Bug 845952 (maildirblockers)

finish "maildir" message storage [meta]

NEW
Unassigned

Status

MailNews Core
Database
5 years ago
2 months ago

People

(Reporter: wsmwk, Unassigned)

Tracking

(Depends on: 15 bugs, Blocks: 2 bugs, {feature, meta, user-doc-needed})

Trunk
feature, meta, user-doc-needed
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [maildir])

(Reporter)

Description

5 years ago
This meta bug is to organize the work to finish "maildir" message storage. Blocking are bugs from [1] and from bug 58308 (which is qmail and won't block this) I include fixed bug 402392 and bug 738651. The end goal is to enable maildir as default, but the meta bug could certainly live on to finish straggling issues after maildir enabled by default.

To do list:
- Someone to file a bug for "enable maildir as default message store"
- check to see if I have missed anything in blockers
- remove anything in blockers that doesn't exactly belong

[1] https://bugzilla.mozilla.org/buglist.cgi?list_id=5785777;field0-0-0=short_desc;type0-0-1=substring;field0-0-1=status_whiteboard;resolution=---;query_format=advanced;value0-0-1=maildir;type0-0-0=anywordssubstr;value0-0-0=maildir%20mail-dir;product=MailNews%20Core;product=Thunderbird

Comment 1

5 years ago
Add to the to-do another task:

- Tool to optimize mail storage (convert mbox to maildir in existing installations)
No longer depends on: 537626
(Reporter)

Updated

5 years ago
No longer depends on: 584695
Depends on: 58308, 852080, 855950, 855954
Whiteboard: [maildir]
No longer depends on: 789679
Depends on: 856087
"maildir" in bug summary of Bug 534135 is pure "Maildir" used by IMAP server instead of Tb's maildir-like, Maildir Lite, .... Removing from depndency.
No longer depends on: 534135
Depends on: 856286
Depends on: 856288

Comment 3

5 years ago
One day (after the >4GB mbox work) I'd like to try out the maildir-lite support and see if I can polish some of the bugs reported here.
Depends on: 856385
Depends on: 856387
Depends on: 856396
Depends on: 856407
Depends on: 856519
Depends on: 856532
FYI.

Quick summary of observed phenomena in some basic/simple test with recent trunk nightly(8Tb 22.0a0).

Current maildir store looks;
- RETR to LocalMbox/tmp/nnnn, move to LocalMbox/cur/nnnn, works.
- fetch body[] to IMAPMbox/tmp/nnnn, move to IMAPMbox/cur/nnnn, works.
- Copy/Move mail(s) is always same as
  Copy/Move from LocalSource/cur/nnnn to LocalTargetMbox/cur/nnnn.
- No care for Copy/Move Source/Target folder == berkleystore
    No Mbox/cur/nnnn, so misbehave.  
- No care for Copy/Move Source/Target folder == maildirstore/IMAP
  - Offline-Use=Off :
    No offline-store file(Mbox/cur/nnnn) => misbehave  
  - Offline-Use=On :
    Copy/Move from/to IMAP_maildir is;
       same Copy/Move as Copy/Move from/to Local_maildir
     + re-synchronizasion with server because of IMAP, if Move.
    So, if move from IMAP folder, IMAP_maildir/cur/nnnn is
    simply moved to maildir_Target_folder/cur/nnnn
    as done in move from maildir_local_mbox to maildir_local_mbox.
    then, when IMAP Mbox open after move, mail is fetched again.
- It looks that return code, status etc. is not checked many places.
  So, empty directory of Mbox/cur/nnnn, Mbox/cur/nnnn of file size=0
  is easily created in many many cases.
- Unique filename generation has holes, so same cur/nnnn file can be
  used by multiple mails.
- If directory for Mbox is suffixed,
    Mbox name is case sensitive in IMAP server.
    File name in client file syatem is case insensitive.
  bc/cur for abc, Abc-1/cur for Abc, ABC-2/cur for ABC, are used.
  In this case, association between Mbox name and Directory name
  is easily broken by rename of folder, by unsubscribe/subscribe(due to
  known bug).
Funny phenomena was also observed.
- If auto-sync of IMAP account is disabled,
  <server_name>.sbd is created, and <server_name>.sbd\INBOX, INBOX.msf
  is created and used, and <server_name>.sbd\INBOX/cur is created
  and used. Directory/file for sub folders under INBOX is created
  in <server_name>.sbd\INBOX.sbd.
  Directory/file for all other folder is normally created under
  <server_name> directory(and <server_name>.msf is created, as usual).
  This may be caused by /INBOX/Inbox folder what is intensionally
  created for bug testing.
 .
Depends on: 857003

Updated

5 years ago
Component: General → Database
Product: Thunderbird → MailNews Core
Hardware: x86 → All
Version: unspecified → Trunk
Depends on: 857436
Depends on: 857443
I've opened meta bug 859011 for many currently known problems around "Copy/Move mails with MaildirStore".
So moving such bugs from this bug's dependency tree to that bug, to keep this bug as root meta bug for "followups after MaildirStore implementation".
(Reporter)

Updated

5 years ago
Depends on: 797710

Updated

5 years ago
Depends on: 890742
(Reporter)

Updated

4 years ago
Depends on: 906469
(Reporter)

Updated

4 years ago
Blocks: 476239

Updated

4 years ago
Depends on: 816304

Updated

4 years ago
Depends on: 1011399
(Reporter)

Updated

3 years ago
Alias: maildirblockers

Updated

3 years ago
Keywords: feature

Updated

3 years ago
Depends on: 1078367
Depends on: 1124948
Depends on: 1124015

Updated

3 years ago
No longer depends on: 58308

Updated

3 years ago
No longer depends on: 753147
Blocks: 1135309
No longer blocks: 1135309
Depends on: 1135309
Depends on: 1143241
No longer depends on: 1124948, 1135309
Keywords: user-doc-needed

Comment 6

3 years ago
Is it possible to have the maildir files have the same filename [or at least the date/time part], sans the flags as they do on the server?

Comment 7

3 years ago
Deleting a folder with Thunderbird will delete the .msf file, but not the actual folder that messages are stored in.

Comment 8

3 years ago
My inbox folder has about 156k emails in it, but the corresponding file system folder has 200k files.  How can this be reconciled without downloading all the emails again?  It's taken close to 40 hours to download my whole mail store so far and it isn't quite finished yet.  The sluggishness and lock ups during this time are painful too.

I've had crashes and had to restart Thunderbird, this may be part of the problem.  I haven't deleted 40k emails either (from the inbox).

The inbox is only one of my folders, some other folders have huge numbers too.  All up I should have about 2M emails (not quite), which includes mailing list data amongst other emails going back over 10 years.

It would be much better to be able to rsync the Maildir folder and have a process adjust file names and remove tag information from the file name to the lines at the top of each email.  Then build the .msf files from the actual content ... so, basically an offline build.

Comment 9

3 years ago
Did you guys even test this?

I've initiated a repair on my Inbox and now I have 16K extra files in the OS with what looks like another 120K to go.

Total emails in the Inbox (reported by TB) is identical to the server's OS directory, that is around 156K; right now the client's OS directory has 217K of email files.

Of course the problems are amplified with large mail storage, but even small test folders exhibit problems.
(Reporter)

Comment 10

3 years ago
Andrew, AFAIK no one using maildir has experienced this problem. Please file a separate bug report describing your issue https://bugzilla.mozilla.org/enter_bug.cgi?product=Thunderbird and answering the questions posed here and make it block this bug - because this bug is a meta for overall tracking, not for fixing specific bugs. Thanks

(In reply to Andrew McGlashan from comment #8)
> My inbox folder has about 156k emails in it, but the corresponding file
> system folder has 200k files.  How can this be reconciled without
> downloading all the emails again?  It's taken close to 40 hours to download
> my whole mail store so far and it isn't quite finished yet.  The
> sluggishness and lock ups during this time are painful too.


> I've had crashes and had to restart Thunderbird, this may be part of the
> problem.  I haven't deleted 40k emails either (from the inbox).

Please post your crash IDs in the new bug report
https://support.mozilla.org/en-US/kb/mozilla-crash-reporter#w_viewing-crash-reports

> The inbox is only one of my folders, some other folders have huge numbers
> too.  All up I should have about 2M emails (not quite), which includes
> mailing list data amongst other emails going back over 10 years.
> 
> It would be much better to be able to rsync the Maildir folder and have a
> process adjust file names and remove tag information from the file name to
> the lines at the top of each email.  Then build the .msf files from the
> actual content ... so, basically an offline build.

I believe imap would not like that.
Please direct all future comments to the new bug(s) you file.

Updated

3 years ago
Depends on: 1176675
(Reporter)

Updated

2 years ago
Depends on: 1182686
(Reporter)

Updated

2 years ago
Depends on: 1044456
(Reporter)

Updated

2 years ago
Duplicate of this bug: 749983

Updated

2 years ago
Depends on: 1259035

Updated

2 years ago
Depends on: 1259040

Updated

2 years ago
Depends on: 1261633
(Reporter)

Updated

a year ago
Blocks: 1306254
(Reporter)

Updated

a year ago
Depends on: 1275948, 1264673
(Reporter)

Updated

a year ago
Depends on: 1307017
(Reporter)

Updated

a year ago
Depends on: 1317066
(Reporter)

Updated

a year ago
Depends on: 1293770
(Reporter)

Updated

a year ago
Depends on: 1203570
(Reporter)

Updated

a year ago
Depends on: 1317117

Comment 12

5 months ago
Is there a progress on this issue?

Thomas
You need to log in before you can comment on or make changes to this bug.