1498532 - maildir storage issues - deleting folder moves .msf only to Trash, not the folder (with cur subfolder and messages within)

Status: RESOLVED WORKSFORME

(Thunderbird :: Folder and Message Lists, defect)

Description

[cra3y]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36

Steps to reproduce:

0) You have to account maildir storage enabled.

1) Create folder (for example "test") in INBOX, Archive, Sent - doesn't matter where
2) Copy some messages to this ("test") folder
3) Delete folder


Actual results:

The "test.msf" is moved to Trash, but the "test" folder with "cur" subfolder and messages within's remained in original place (for example in. "INBOX.sbd\test")

It also remains, when You will empty Trash folder. 


Expected results:

the corresponding folder (with maildir structure inside) to .msf file  ("test" in this example) should be moved together to Trash and then deleted, when Trash will be emptied.

Comment 1

[Jorg K (CEST = GMT+2)]

Yes, recently fixed in Daily. Fix not available in TB 60 yet.

Comment 2

[cra3y]

Hmm it is different.

It is not about removal to Trash. It is about to removal to Trash (hit delete on folder that should go to trash but isn't)

Comment 3

[cra3y]

I mean: It is not about removal from trash.

Comment 4

[Jorg K (CEST = GMT+2)]

Could you try it on a Daily (install to different directory):
http://ftp.mozilla.org/pub/thunderbird/nightly/latest-comm-central/

Ben, is this issue fixed, otherwise I'll undo the dupe.

Comment 5

[cra3y]

Ok. Lets test it.
Using compilation nightly from 13 Oct 2018.
https://i.imgur.com/96V4dN3.png

I created "testfolder" in INBOX and there are two messages inside "testfolder"
https://i.imgur.com/DUeC4cl.png

Let delete this folder (without shift, it should move to Trash):
https://i.imgur.com/qLYXHOi.png

Now is something different: the testfolder disappeared from INBOX.sbd (ok), but what happened to it? (in Trash.sbd there is "testfolder.msf" only, but no "testfolder" with maildir structure within).

Comment 6

[Jorg K (CEST = GMT+2)]

Ben will look at it on Monday. Thanks for testing and providing images.

Comment 7

[cra3y]

No problem :-)

My plan is to switch to maildir from mbox because of backup purposes (mbox and large files is pain in ....), and this (folder deletion) is the last obstacle ...

Comment 8

[Jorg K (CEST = GMT+2)]

Personally I think, millions of small files are a pain ;-)

Comment 9

[Ben Campbell]

(Sorry about the delay - was away Monday!)

Just did a build with the latest code in m-c and c-c, and I can't seem to replicate the bug.

I tried with both IMAP folders and local folders (which should also cover the POP3 case I think), and the raw files seemed to correspond with the folder structure in the folder pane.
I'm running Linux. Perhaps there's some OS difference to do with moving directories? (seems like a long shot - all that low-level stuff is pretty well road-tested)

cra3y:
- what sort of server are you using (IMAP? POP?)
- have you tried it with local folders?
and one stupid question (sorry - you've already done the maildir conversion, so this doesn't apply, but I have to ask anyway):
- are you looking at the right folder? The mbox->maildir conversion leaves the old directory in place, and switches to a new one, with a `-maildir` suffix... (eg "myprofile/ImapMail/imap.example.com-maildir" vs "myprofile/ImapMail/imap.example.com")

Comment 10

[cra3y]

- Dovecot (2.2.9) based IMAP server.
- I only tried in IMAP folder
- maildir from beginning - I added this line (below) to prefs.js before account configuration 
user_pref("mail.serverDefaultStoreContractID", "@mozilla.org/msgstore/berkeleystore;1");

Comment 11

[Ben Campbell]

(In reply to cra3y from comment #10)
> user_pref("mail.serverDefaultStoreContractID",
> "@mozilla.org/msgstore/berkeleystore;1");

Shouldn't that be "@mozilla.org/msgstore/maildirstore;1" ? (berkeleystore is mbox format)

From your screenshots, you obviously _have_ got maildir running, so I don't think this is the problem...

Comment 12

[cra3y]

Yes, yes, I pasted wrong directive here. It is "@mozilla.org/msgstore/maildirstore;1" definitely in my config.

Comment 13

[Jorg K (CEST = GMT+2)]

Hmm, this is a little non-conclusive, so I tested it myself.

Created a new profile with an IMAP account, created "testfolder" as subfolder of INBOX, move a message into it. Result:
  ImapMail\server.de\INBOX.sbd\testfolder\cur
has one message.
There is also
  ImapMail\server.de\INBOX.sbd\testfolder.msf

Now deleted "testfolder".

First observation: "testfolder" is still in the folder tree, that's bad. Clicking onto it gives an error.

  ImapMail\server.de\INBOX.sbd\testfolder\
still exists, and is empty. The "cur" directory is gone.
  ImapMail\server.de\INBOX.sbd\testfolder.msf
still exists.

In the trash I see:
  ImapMail\mail.your-server.de\INBOX.sbd\Trash.sbd\testfolder.msf
Nothing else.

This completely confirms comment #5.

Comment 14

[Ben Campbell]

Oh, interesting. I didn't get that at all.
I wonder if it's only an issue on windows?
I'm aiming to go back and try with the nightly build that cra3y tried (but probably won't get a chance until next week - want to try and get some of my folder-lookup/rdf stuff properly nailed down).

Comment 15

[Jorg K (CEST = GMT+2)]

Retesting this in TB 60: The behaviour is different to Daily. After the delete
  ImapMail\server.de\INBOX.sbd\testfolder\cur
is still complete with the message in it.
  ImapMail\server.de\INBOX.sbd\testfolder.msf
is gone and the folder has also been unsubscribed.

There is also
  ImapMail\mail.your-server.de\INBOX.sbd\Trash.sbd
with the MSF file in it as reported here.

The behaviour has changed, but not necessarily for the better.

Comment 16

[Ben Campbell]

(In reply to cra3y from comment #5)
> Ok. Lets test it.
> Using compilation nightly from 13 Oct 2018.
> https://i.imgur.com/96V4dN3.png
> 
> I created "testfolder" in INBOX and there are two messages inside
> "testfolder"
> https://i.imgur.com/DUeC4cl.png
> 
> Let delete this folder (without shift, it should move to Trash):
> https://i.imgur.com/qLYXHOi.png
> 
> Now is something different: the testfolder disappeared from INBOX.sbd (ok),
> but what happened to it? (in Trash.sbd there is "testfolder.msf" only, but
> no "testfolder" with maildir structure within).

Ahh! I think I've figured it out - this is all working as intended.

The reason you're not seeing the "Trash.sdb/testfolder" dir appearing is that when a folder is moved to trash, it's synchronisation settings are changed so that it's contents are no longer automatically downloaded from the server. 
(right click on folder, "synchronization" tab, see the "Select this folder for offline use" checkbox).

TB 60 _does_ have a bug where the maildir ("INBOX.sdb/testfolder") is not deleted. But this appears to be fixed now:
I tested nighties "65.0a1 (2018-11-01) (64-bit)", "64.0a1 (2018-10-13) (64-bit)" and my own latest build from mercurial and they all seem fine.
I suspect this bug was fixed accidentally when I fixed bug 1317117 - the imap folder delete wasn't going through the plugable msgstore, so was effectively hardwired to mbox behaviour, thus the leftover maildir.

Comment 17

[Ben Campbell]

(In reply to Jorg K (GMT+2) from comment #13)
> First observation: "testfolder" is still in the folder tree, that's bad.
> Clicking onto it gives an error.

_This_ is the one that worries me!
But I've not been able to replicate it so far.
Jorg, are you able to get this to happen reliably?
If so, likely deserves a separate bug, I think...

Comment 18

[Jorg K (CEST = GMT+2)]

(In reply to Ben Campbell from comment #16)
> TB 60 _does_ have a bug where the maildir ("INBOX.sdb/testfolder") is not
> deleted.
OK, that's what I said in comment #15: INBOX.sbd\testfolder.msf gone, INBOX.sbd\testfolder *not* gone, MSF in INBOX.sbd\Trash.sbd.

So now that we agree about TB 60, let's repeat the thing with a trunk build repeating comment #13:
This time my milage was different. Folder unsubscribed but I was left with two copies of the MSF file, one in INBOX.sbd, one in INBOX.sbd\Trash.sbd.

In fact, the story is more complicated. I ran through this various times. In most cases everything worked, and after the deletion, the MSF in INBOX.sbd\Trash.sbd was the only thing left as intended. In a few other cases, the MSF in the original loclation, INBOX.sbd, sprang back to life. It's actually an empty file. I have a tool on Windows that watches files and I can see the file reappear after a few seconds.

I'm closing this bug as WFM and I'll file another for the folder that springs back to life.