1099275 - Thunderbird should better inform / issue a warning to user before running Repair Folder, because this deletes the local copy/cache of it, which then can take days to re-download for imap

Status: RESOLVED FIXED

(Thunderbird :: Folder and Message Lists, defect)

Description

[Aaron Marcuse-Kubitza]

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:33.0) Gecko/20100101 Firefox/33.0
Build ID: 20141013200408

Steps to reproduce:

1) before trying this, back up "All Mail" and "All Mail.msf" in "~/.thunderbird/__.default/ImapMail/imap.googlemail.com/[Gmail].sbd/"
2) right-click the All Mail folder > Properties > Synchronization tab, check "Select this folder for offline use", and click Download Now
3) wait for it to finish downloading your e-mails
4) right-click the All Mail folder > Properties > General Information tab and click Repair Folder


Actual results:

"All Mail" and "All Mail.msf" in "~/.thunderbird/__.default/ImapMail/imap.googlemail.com/[Gmail].sbd/" are deleted without warning, and Thunderbird then attempts to re-download all your e-mails (which can take days if you have a lot of e-mails)


Expected results:

Thunderbird should display a warning that Repair Folder will delete the local copy of the folder and re-download the e-mails in it

Comment 1

[WADA:World Anti-bad-Duping Agency]

"Statements shown at left of Repair Folder button at Folder Properties/General panel" is same in all of "local mail folder", "imap offline-use=On folder", "imap offline-use=Off folder", newsgroup, RSS Feed.
Current statements is for local mail folder(especially for traditional msgstore/berkleystore case). 
This statements is better changed according to folder's characteristics.
Because room for the statements is enough, longer statements can be written at there.

If imap folder, major purpose of "Repair Folder" is not "Re-parse all mail's Unix Mbox mail separator" and "Re-parse all message headers"(This is cllled Rebuild-Index). Rather, major purpose of "Repair Folder" in imap is "re-fetch header data and message body data(if Offline-Use=On) of all mails from scratch". This kind of difference is better described by the statements.

Comment 2

[Aaron Marcuse-Kubitza]

> Current statements is for local mail folder(especially for traditional msgstore/berkleystore case). 
> This statements is better changed according to folder's characteristics.

Does that mean that for local folders, Repair Folder actually does what you would expect (just repairs the .msf, rather than deleting the stored e-mails)?

Comment 3

[WADA:World Anti-bad-Duping Agency]

(In reply to Aaron Marcuse-Kubitza from comment #2)
> Does that mean that for local folders, Repair Folder actually does what you
> would expect (just repairs the .msf, rather than deleting the stored e-mails)?

Yes. 
And, "Repair Folder of imap" also does do expected actions as designed, as implemented.
In any Repair Folder, what is done by Repair Folder of Mbox named FolderX = Clear FolderX.msf content + Open FolderX again.
Because FolderX.msf content is discarded(Repair Folder request by you == Discard current .msf file content),
-  if pop3, FolderX.msf content is re-created from mail data held in MsgStore file(file named FolderX ),
-  if imap, all data is always held at server, so FolderX.msf content is re-created from mail data held at imap server.
   Content of Offline-Store file for imap(file named FolderX) is managed by FolderX.msf.
   Because FolderX.msf content is cleared once according to your "Repair Folder" request,
   Offline-Store file for imap(file named FolderX) is also re-created from mail data held at imap server.

i.e. Phenomenon you saw != Offline-Store file is deleted.
      Phenomenon you saw == Re-creation of Offline-Store file is started according to your "Repair Folder" request.
      Offline-Store file for imap in Tb != "local copy of imap folder", even if "local file on local HDD" is used for it by Tb
      as Cache file != permanent data file, as Swap file != permanent data file,
      as files on mirror server is re-created from scratch if sync with main server is lost.
      If you need "local copy of imap folder", "Copy imap mail to folder of Local Folders" is sufficient and is always available.
Because no actual dataloss is involved(Offline-Store file is re-created sooner or later), I don't think "Confirmation dialog for Repair Folder" is mandatory.
Note: "Confirmation dialog for Shift+Delete" is shown by Tb, because "Undo delete" is impossible when "Shift+Delete" then it may produce "a kind of dataloss for user".

Comment 4

[Aaron Marcuse-Kubitza]

> Because no actual dataloss is involved(Offline-Store file is re-created sooner or later), I don't think "Confirmation dialog for Repair Folder" is mandatory.

But "sooner or later" can be *days* later if the user has a lot of e-mails (2+ GB). Most users would want to know that they now need to leave Thunderbird open, and their computer on, for several days to re-download all their e-mails.

Comment 5

[WADA:World Anti-bad-Duping Agency]

(In reply to Aaron Marcuse-Kubitza from comment #4)
> But "sooner or later" can be *days* later (snip)

I agree with you.

Reason why "data in berkleystore Offline-Store file" is not accessible if xxx.msf file is cleared  :
   "Offset of mail data held in Offline-Store file" is lost by "clear xxx.msf"
    because the  "Offset of mail data held in Offline-Store file" is held in xxx.msf file only.
If Tb's msgstore/maildirstore feature is fully constructed and design/implementation will be improved, "temporary loss of Offline-Store file data by Repail Folder until re-auto-sync is completed" is minimized.
See meta Bug 859011 for msgstore/maildirstore.
Just an idea.
   1. If imap, include UID in /cur/nnnnnnnn file name(one file per mail), for example, /cur/#<UID_of_mail>-....
       By this change, "Offline-Store file for a mail of UID=mm" can be reached by UID=mm only.
       (meta data of a mail can be held as "string in file name" if maildirstore)
   2. fetch header data of all mails again from imap server if Repair Folder is requested.
   3. if UIDVALIDTY is not changed, "UID is same as before" is guaranteed.
       So, there is no need to discard Offline-Store file content immediately, merely by "Repair Folder" is requested.
   4. if /cur/#<UID_of_mail>-.... already exists, flag the mail as "data is held in Offline-Store", with Check-By-Auto-Sync-Needed.
   5. if Check-By-Auto-Sync-Needed, auto-sync compares RFC822.SIZE and size of /cur/#<UID_of_mail>-.... file.
       If RFC822.SIZE != size of /cur/#<UID_of_mail>-.... file, or if something wrong is found,
       re-fetch entire mail data to /cur/#<UID_of_mail>-.... file in background.
   6. Even if "entire mail data of all mails" is re-fetched by auto-sync after "Repair Folder" as currently done,
       "loss of Offline-Store file content" is limited to one mail only at any moment. 

Possible solution for "sooner or later == severla days to a week" case.

When "Repair Folder" is requested for imap Offline-Use=On folder named FolderX,
   Copy    .../ImapMail/<imap_server>/.../FolderX and FolderX.msf
        to    .../Mail/Local Folders>/<imap_server>.sbd/FolderX and FolderX.msf
   Create .../Mail/Local Folders>/<imap_server> file(null file) and >/<imap_server>.msf file(null file)
   => Because format of FolderX and FolderX.msf for imap is same as XXX & XXX.msf for local mail folders,
         use can see mail data as "mail in FolderX under <imap_server> folder of Local Folders,
         as far as (a) fle size of FolderX < 4GB and (b) "From " line is not contained in mail body data.
Support of "4GB <= fle size of FolderX" in msgstore/berkleystore is pretty hard, support of (b) is pretty esay.
   Escape "From " line by ">From " in above Copy step.
Above "Copy to Local Folders" can be manually executed.

Third way : Never keep huge size of [Gmail]/All Mail  == Split mails to multiple free Gmail accounts
                   == "one mail folder per one Gmail account" strategy.
   Gmail account0 : shared account for mail receive, 
                               for mail send(to hold draft in [Gmail]/Drafts, to hold sent mail copy in [Gmail]/Sent Mail)
                               By message filter, move any mail to Folder1 to FolderN of Gmail accoun1 to accountN.
             SMTP definition with id of account0 only. 
             If needed, register mail-addr of account1 to accountN as usable address to Gmail.
             But there is no need to use mail-addr of Gmail account1 to Gmail accountN in mail send/receive.
             Any mail address  can be defined as an identity of any account in Thunderbird.
             So "mail-addr of Gmail account0" can be used in any mail send.
             Any mail address  can be registered to Gmail as valid addrees for mail send, as far as the mail-addr is reachable,
             although "Sender: mail-addr of Gmail account0" is added by Gmail.
    Gmail account1 to accountN for Folder1 to FolderN. 
             Use only one SMTP definition with id of account0.
             Size of [Gmail]/All Mail of accountN == size of FolderN.
    Offline-Use=On folder : [Gmail]/All  Mail of each account only => No duplicate copy in Tb's Offline-Store file.
    Dummy POP3 account for Search Folder : (Search folder under imap account is better avoided)
         FolderN : Search Folder, search target = Folder N of Gmail accountN
         SentX    : Search Folder, search target = [Gmail]\Sent Mail of Gmail account0
         DraftsX  : Search Folder, search target = [Gmail]\Drafts of Gmail account0
         Define one identity only with mail-addr of Gmail account0
         Use only one SMTP definition with id of account0
         Use this account as default account

Comment 6

[Aaron Marcuse-Kubitza]

> When "Repair Folder" is requested for imap Offline-Use=On folder named FolderX,
>   Copy    .../ImapMail/<imap_server>/.../FolderX and FolderX.msf
>        to    .../Mail/Local Folders>/<imap_server>.sbd/FolderX and FolderX.msf

How difficult is it to set up this copying? Would it be possible to add the warning dialog until the copying procedure is put in place?

> Third way : Never keep huge size of [Gmail]/All Mail  == Split mails to multiple free Gmail accounts
>                   == "one mail folder per one Gmail account" strategy.

I don't think Gmail users with a lot of e-mails are going to want to split all their mail into separate accounts...but yes, it helps to have a smaller All Mail folder (if you can arrange this)!

Comment 7

[WADA:World Anti-bad-Duping Agency]

(In reply to Aaron Marcuse-Kubitza from comment #6)
> > When "Repair Folder" is requested for imap Offline-Use=On folder named FolderX,
> >   Copy    .../ImapMail/<imap_server>/.../FolderX and FolderX.msf
> >        to    .../Mail/Local Folders>/<imap_server>.sbd/FolderX and FolderX.msf
> 
> How difficult is it to set up this copying?

It's pretty simple and easy if manually executed, as far as file size of [Gmail].sbd\All Mail is less than 4GB and "From " line is not contained in message body of any mail.
1. Create Gmail-No-N folder under "Local Folders" account.
2. Create "All Mail" folder under Gmail-No-N of "Local Folders"
3. Terminate Tb.
4. By File Manager such as Windows Explorer,
    Copy    .../ImapMsil/ ... /[Gmail].sbd/All Mail & All Mail.msf
         to   .../Mail/Local Folders/Gmail-No-N.sbd/All Mail & All Mail.msf (replace copy)
5. Restart Tb.
6. Open Local Folders/Gmail-No-N/All Mail folder.
    Rebild-Index is invoked, but because no "From " line in Gmail-No-N.sbd/All Mail, no problem occurs.

If "From " line exists in message bofy of a mail, mail data is split at the "From " line by Rebuid-Index(Re-parse mail data) at step 6.
So, "manual escaping of From Line in message body by >From " is needed.

If you want such job by Thunderbird, please open Enhancement request bug.

I believe that it's a little bit of a stretch for Thunderbird to support huge Unix Mbox format file as MsgStore file or Offline-Store file.
I think "Huge message folder support in Tb" is better done by msgstore/maildirstore support only, even though many peoples are trying to resolve bug 789679.

Comment 8

[klaus]

Unfortunately, the (German) text still does not give any warning similiar to Wayne's proposal.

Especially, for some IMAP servers, repair kills off all tags, so it should really only be done if absolutely necessary. (Loosing tags happens for all IMAP servers that cannot store custom keywords, if emails are re-downloaded. - for example hmailserver, which is the only free (open source??) IMAP server for Windows.)

Comment 9

[Wayne Mery (:wsmwk)]

We should do something, so confirming.

The issue is worse for gmail because of things like bug 697409

Comment 10

[João Carlos Mendes Luís]

Current version (52.7.0) apparently does not download emails, only headers.  Even if folder is selected for offline usage.

Not sure if this is a workaround, but surely is not what I expected.

My folders are in maildirs format.  When I select "Repair Folders", the "cur" dir is deleted, and msf file (headers) is redownloaded from IMAP server.

Comment 11

[Wayne Mery (:wsmwk)]

(In reply to João Carlos Mendes Luís from comment #10)
> Current version (52.7.0) apparently does not download emails, only headers. 
> Even if folder is selected for offline usage.

This bug predates maildir. If you have a maildir issue you might look for another bug report or file a new one. And maildir default should download message body, not just headers - unless you are using pop and have set that option

Comment 13

[Magnus Melin [:mkmelin]]

https://searchfox.org/comm-central/rev/08f79ea753caed459c280e00cb560fb7032e2d7f/mail/base/content/folderPane.js#3422

Comment 14

[Mirko Kerezovic]

Attachment: Bug 1099275 - Warn before repairing folders with offline storage. r=#thunderbird-front-end-reviewers

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/32249b184c9c
Warn before repairing folders with offline storage. r=tobyp

Comment 16

[Magnus Melin [:mkmelin]]

Unfortunately this is causing a test failures - will back it out.

comm/mailnews/protocols/exchange/test/browser/browser_repair_folder.js | test_repair_folder - Test timed out

https://treeherder.mozilla.org/jobs?repo=comm-central&resultStatus=unscheduled%2Ctestfailed%2Cbusted%2Cexception%2Csuccess%2Cretry%2Cusercancel%2Crunning%2Cpending%2Crunnable&selectedTaskRun=T2QvJFoDQ6aVJtD6u_VGkA.0

Comment 17

[Magnus Melin [:mkmelin]]

Backout: https://hg.mozilla.org/comm-central/rev/3fe0f5d10c823a812b14aff3a561ff88aa92b9b0

Comment 18

[Mirko Kerezovic]

Hi Magnus, just a quick heads-up: I updated D332662 after the backout to handle the new confirmation dialog in browser_repair_folder.js.
The Exchange repair-folder test now waits for and accepts the dialog, so the timeout that caused the backout is resolved in the updated patch.
The repair assertions complete successfully with this change. Thanks!

Comment 19

[Magnus Melin [:mkmelin]]

That part looks ok. But I left another comment there, please take a look

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/7d73d3f18fcb
Warn before repairing folders with offline storage. r=tobyp

Comment 21

[Magnus Melin [:mkmelin]]

Unfortunately still test failures: https://treeherder.mozilla.org/jobs?repo=comm-central&selectedTaskRun=efJR590IScyj_Q9n3edzSA.0

I'll land a bustage fix for them.

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/159907c40816
Follow-up: Wait for the folder properties dialog to leave modal state after the repair prompt. rs=me
https://hg.mozilla.org/comm-central/rev/7a25a51c31fb
Follow-up 2: Wait for FolderLoaded around the repair in browser_repairFolder.js. rs=me