1512080 - MailDir Repair folder in IMAP causes folder contents to be deleted

Status: NEW

(MailNews Core :: Database, defect)

Description

[Matt]

Attachment: Activity manager..png

I noticed that my Gmail Maildir account was showing duplicate messages in the folder pane.  It also showed some 133,000 messages in the folder.

I right click the the folder,  selected properties and then the repair button.

Result:
Considerable disk noise as I/O ramped up on my slow disk and in due course Thunderbird reported the repair window as unresponsive.
Eventually the process complete with a report that the folder contained no messages.
Examination of the user profile shows no inbox folder in the folders for the account. 

On next get mail (just regular, not requested) the entire inbox folder reappeared in the profile folder and Thunderbird reported it was downloading some 286 thousand messages in the inbox folder.

At the time of filing the download has not yet completed.

Expected Result:
Re-index of the folder.

Attachment show the huge disparity in what is reported inthe status bar and the activity manager for this event.

Comment 1

[Matt]

Attachment: status line

Comment 2

[Matt]

Interestingly as the download continues the number total in the status line continues to increase. 301,000+ and rising. And apparently some 50,000 in the all mail folder as well.

Comment 3

[Matt]

After some hours both the activity manager and status bar have settled on 133652 messages to download.  Having downloaded less than 10,000 it is going to take some time to recover from this repair folder.

Comment 4

[Jorg K (CEST = GMT+2)]

Maybe Ben can take a look.

Comment 5

[Wayne Mery (:wsmwk)]

related to bug 1412779 ?

Is this fundamentlaly a maildir issue?
Or just more general?

Comment 6

[Matt]

AFAIK it is maildir specific, I use repair on my local folders without any issue, but you are correct, I just tried repairing an IMAP folder with GMX and it downloading messages 1 to 102. SO I think it is a general IMAP issue, not maildir specific as I had assumed.

Comment 7

[gene smith]

(In reply to Matt from comment #6)

AFAIK it is maildir specific, I use repair on my local folders without any issue, but you are correct, I just tried repairing an IMAP folder with GMX and it downloading messages 1 to 102. SO I think it is a general IMAP issue, not maildir specific as I had assumed.

My observation is that "repair" of an imap folder either re-downloads just the headers (if folder has no offline store) or the full messages (if folder does have offline store). So repairing an imap folder with 100k message and maildir or mbox offline store may take a while. (But not sure if this is a part of this issue.)

Comment 8

[Ben Campbell]

Picking through the code on this one, I the observations in comment #6 and comment #7 are right - I don't think it's a maildir-specific issue. Does anyone have any observations to the contrary?

I think the overall procedure is driven by rebuildSummary() (https://searchfox.org/comm-central/source/mail/base/content/folderPane.js#3385).
The gist of it is:

1. if it's an offline folder, delete the mbox file or maildir
2. force-close the database (summary file)
3. call folder.updateFolder() to perform the rebuild

In step 3 I think the IMAP updateFolder() grabs headers from the server, so it'll be slow. Local folders rebuild the database by reparsing the mbox/maildir, which should be way quicker.
step 1 is another ugly little instance of folder code directly fiddling the filesystem - it should be handled in the mbox/maildir specific plugablemailstore.
With 2, I don't think this actually deletes the msf file. It just marks it invalid. I tried running "repair folder" on a disconnected IMAP folder, with mbox, and no local message downloading. I'd have expected to see all the messages disappear, but they popped right back up. So I think it just picked up the old msf file again... needs a little more investigation.

I kind of think there are enough issues with folder repair that it warrants stepping back a little and figuring out exactly what it should be doing...
https://bugzilla.mozilla.org/buglist.cgi?quicksearch=folder%20repair

Comment 9

[gene smith]

Picking through the code on this one, I the observations in comment #6 and comment #7 are right - I don't think it's a maildir-specific issue. Does anyone have any observations to the contrary?

As a test, I tried to repair the maildir folder I've been working with in bug 1702692. What I see is when I repair Inbox is that the .eml files under Inbox/cur are all removed and don't come back and the message summary goes empty until you move to another folder and then back. But the .eml files never come back. It seems to be behaving like the folder is configured to not use offline store after the repair.

The only way I could get the .eml files back is to remove Inbox folder and Inbox.msf and restart tb. Then when an Inbox message is accessed the .eml files all come back after, I assume, they are finally all downloaded. (Haven't looked at code of even imap logs for this.)

I've never seen this problem with repair of mbox. However, to get rid of any redundant messages in mbox you have to at delete the mbox file and corresponding .msf to do a complete repair. Otherwise mbox repair leaves the redundant messages in the file. (By "redundant" I mean messages on server with "bad" headers that cause the file to grow as we see in bug 1702692 and friends.)

So typically if I really want to "repair" a folder, I just delete the storage file and the .msf file and let tb re-download and rebuild them rather than relying on the repair button.

Comment 10

[gene smith]

Another observation that is probably off-topic and maybe deserving its own bug:
With maildir, if I configure "just mark message as deleted" instead of moving deleted to a folder, when I delete a message, as expected, the message is crossed out and the red X is put in front. However, there doesn't seem to be a way to expunge the deleted messages from that folder since maildir doesn't provide a right-click "compact" option.
There is a "compact folders" option under "File" menu but that appears to apply to multiple folders and not to a specific folder and not sure if it even works for maildir folders.

Comment 11

[Wayne Mery (:wsmwk)]

Other repair issues https://mzl.la/3AcGIi8

Comment 12

[Magnus Melin [:mkmelin]]

Maybe this was bug 1683714