752697 - Since installing TB12 most of my folder/sub-folder have been duplicated with folder.SBD (if .sbd file extension of .sbd directory is somehow upper case in file system, folder.SBD directory is shown as mail folder named folder.SBD by Tb 12)

Status: NEW

(Thunderbird :: Folder and Message Lists, defect)

Description

[[e-mail]]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/534.55.3 (KHTML, like Gecko) Version/5.1.5 Safari/534.55.3

Steps to reproduce:

Installed TB12


Actual results:

In most cases where I had sub-folders the master folder was duplicated in the folders pane with 'master folder.SBD' as a new folder with some or all of the folder structure copied but empty.  The original master still had the sub-folders with emails in.  When I tried deleting the .SBD sub-folders the originals became detached (present in the file system but invisible in the original master folder.


Expected results:

Folder structure should have remained as it was under TB11.

Comment 1

[David :Bienvenu]

Were you using folders that didn't have .sbd extensions as containers of other folders? That was never really supported.

Comment 2

[WADA:World Anti-bad-Duping Agency]

What is actual directory/file structure of mail directory for the mail account?
- <maildir_of_account> == Path set at Local Directory: of Server Settings
- Tb's a set of files, directory for a mail folder :
  <folder_name>      file. not used if IMAP offline-use=Off folder
  <folder_name>.msf  file(mail summary file, index file) 
  <folder_name>.sbd  directory to hold files for subfolders
- So, following directory/file structure is used by Tb.
> <maildir_of_account>/<folder_name>.msf
> <maildir_of_account>/<folder_name>.sbd/<subfolder>.msf
> <maildir_of_account>/<folder_name>.sbd/<subfolder>.sbd/<subsubfolder>.msf
> <maildir_of_account>/<folder_name>.sbd/<subfolder>.sbd/<subsubfolder>.sbd/<subsubsubfolder>.msf

POP3 account(or Local Folders)? Or IMAP account? Or both?
What is actual folder name and actual file/directory name corresponding to your "master folder"?

Do you use localized Tb 11 or Tb 12?
If yes, what localization? If yes, is your "master folder" localized special folder name? (Inbox, Trash, Drafts, Sent, Templates, Archives, Junk)
What is shown at Location: of folder's Properties/General for folder of "master folder" or "master folder.SBD"?

Comment 3

[WADA:World Anti-bad-Duping Agency]

> "master folder.SBD"

IIRC, Tb ignores file(and directory too) with extension of ".sbd" and doesn't consider it as "file to hold mail data" and comparison of file extension is case insensitive. And, Tb 11 on Win-XP did do it as expected.
However, if file extension is upper case(xxx.SBD) and if xxx/xxx.msf doesn't exist, Tb 12.0.1 on Win-XP showed it as "mail folder named xxx.SBD"(even when xxx.SBD is directory in file system of MS Win).

Case sensitive comparison for ".sbd" part after Tb 12?
Do you see your problem even after ".SBD" is changed to ".sbd"?
Is ".msf extension" upper case too in your environment?

Comment 4

[David :Bienvenu]

What I was trying to say is that Thunderbird ignores directories without .sbd extensions when trying to find folders that contain sub-folders.

Comment 5

[WADA:World Anti-bad-Duping Agency]

Behaviour of Tb 12.0.1 on Win-XP by intentional file/directory creation.
- xxx     : empty directory in file system of MS Win
  xxx.MSF : empty file      in file system of MS Win
  xxx.SBD : empty directory in file system of MS Win
- Tb 12.0.1 showed three mail folders at folder pane.
  xxx     (xxx is shown as mail folder even though directory of file system) 
  xxx.MSF
  xxx.SBD (xxx.SBD.msf file is created by Tb for this DIRECTORY)
- Tb 11.0.1 showed xxx only at folder pane.
  (xxx is shown as mail folder even though directory of file system) 
  (xxx.SBD.msf is not deleted by Tb 11.0.1 as desgined/implemented)

Is "file or directory check" actually executed by Tb?

Comment 6

[[e-mail]]

(In reply to David :Bienvenu from comment #1)
> Were you using folders that didn't have .sbd extensions as containers of
> other folders? That was never really supported.
Sorry for the delay on this.
The folders in the folders pane were 'xxx' and the folders in the file system were 'xxx.SBD'

Comment 7

[[e-mail]]

(In reply to WADA from comment #2)
Sorry abut the delay in replying

> What is actual directory/file structure of mail directory for the mail
> account?
/Users/stancollins/Library/Thunderbird/Profiles/v4ldo4g0.default/Mail/pop.cix.co.uk

> - <maildir_of_account> == Path set at Local Directory: of Server Settings
> - Tb's a set of files, directory for a mail folder :
>   <folder_name>      file. not used if IMAP offline-use=Off folder
>   <folder_name>.msf  file(mail summary file, index file) 
>   <folder_name>.sbd  directory to hold files for subfolders
> - So, following directory/file structure is used by Tb.
> > <maildir_of_account>/<folder_name>.msf
> > <maildir_of_account>/<folder_name>.sbd/<subfolder>.msf
> > <maildir_of_account>/<folder_name>.sbd/<subfolder>.sbd/<subsubfolder>.msf
> > <maildir_of_account>/<folder_name>.sbd/<subfolder>.sbd/<subsubfolder>.sbd/<subsubsubfolder>.msf
> 
> POP3 account(or Local Folders)? Or IMAP account? Or both?
POP3

> What is actual folder name and actual file/directory name corresponding to
> your "master folder"?
> 
> Do you use localized Tb 11 or Tb 12?
I'm using TB12 now but was using TB11 until 4th May.  By localised do you mean am storing emails in 'Local folders'?

> If yes, what localization? If yes, is your "master folder" localized special
> folder name? (Inbox, Trash, Drafts, Sent, Templates, Archives, Junk)
/Users/stancollins/Documents/Thunderbird/Local Folders

> What is shown at Location: of folder's Properties/General for folder of
> "master folder" or "master folder.SBD"?

Just a folder called 'Local Folders' plus 'Local Folders.msf' plus 'Trash' and 'Unsent messages' and their .msf

Comment 8

[WADA:World Anti-bad-Duping Agency]

(In reply to [e-mail] from comment #7)
> By localised do you mean am storing emails in 'Local folders'?

No.
"Localization" means de build of Tb(German version), fr build of Tb(French version), ja build of Tb(Japanese version) and so on.
  de build : "Inbox" is shown as "Posteingang" at folder pane. 
  ja build : "Inbox" is shown as "受信トレイ"  at folder pane.

Comment 9

[[e-mail]]

(In reply to WADA from comment #8)
> (In reply to [e-mail] from comment #7)
> > By localised do you mean am storing emails in 'Local folders'?
> 
> No.
> "Localization" means de build of Tb(German version), fr build of Tb(French
> version), ja build of Tb(Japanese version) and so on.
>   de build : "Inbox" is shown as "Posteingang" at folder pane. 
>   ja build : "Inbox" is shown as "受信トレイ"  at folder pane.
I'm using the English/United Kingdom dictionary if that helps and all the labels, help etc are in English.

Comment 10

[L Andrew]

I believe this to be the same problem I found with SeaMonkey 2.9.1 and Tb 12.0.1 under Windows 7.  (I first noticed it immediately after updating SeaMonkey, which is what I normally use, so the problem seems to be of very recent vintage.)

Here is what I did to reproduce the problem as suggested by Comment 3:

1.  Install Tb 12.0.1.

2.  Under "Local Folders", create a new folder "Test Local Folder".  Within that, create a subfolder "Test Local Subfolder."  Note:  I did not encounter the problem under an IMAP account, but in SeaMonkey it did occur several times under old POP accounts.

3.  Using Windows Explorer, with Tb closed, locate the "Test Local Folder.sbd" file and rename it to "Test Local Folder.SBD".

4.  Restart Tb.  Tb creates a new folder, "Test Local Folder.SBD".

Comment 11

[WADA:World Anti-bad-Duping Agency]

(In reply to L Andrew from comment #10)
> I believe this to be the same problem I found with SeaMonkey 2.9.1 and Tb
> 12.0.1 under Windows 7. 
> (I first noticed it immediately after updating SeaMonkey, which is what I normally use, so the problem seems to be of very recent vintage.)

Did you actually experienced "xxx.SBD(upper case) folder problem" upon upgrade of Sm/Tb in your environment?
If yes, who created directory of upper case ".SBD"?
Did you migrate or move/copy mail folder files/directories from FAT drive in the past?
Or you say "xxx.SBD(upper case) folder problem is reproduced by Sm 2.9.1 on Win7 too by intentional creation of .SBD directory in upper case"?

Comment 12

[WADA:World Anti-bad-Duping Agency]

Regression window of "xxx.SBD(upper case) folder problem".
(Good)
> 2011-12-24-03-00-25-comm-central
> Mozilla/5.0 (Windows NT 5.1; rv:12.0a1) Gecko/20111224 Thunderbird/12.0a1
(Bad)
> 2011-12-25-03-02-53-comm-central
> Mozilla/5.0 (Windows NT 5.1; rv:12.0a1) Gecko/20111225 Thunderbird/12.0a1

(Pushlog, 2011-12-24 00:00:00 to 2011-12-25 04:00:00) 
> http://hg.mozilla.org/comm-central/pushloghtml?startdate=2011-12-24+00%3A00%3A00&enddate=2011-12-25+04%3A00%3A00
Regression by fix bug 4023928(add support for pluggable stores)?
Is there any way to limit such big changes for new feature to trunk only?

Comment 13

[David :Bienvenu]

(In reply to WADA from comment #12)

> Is there any way to limit such big changes for new feature to trunk only?

sadly, the issues tend to only be discovered once you ship the feature.

Comment 14

[L Andrew]

> Did you actually experienced "xxx.SBD(upper case) folder problem" upon
> upgrade of Sm/Tb in your environment?

Yes (regarding SeaMonkey; see below regarding Thunderbird).  Everything was working fine as of my latest full backup of SeaMonkey on 2012-04-30 which still contains the affected upper case .SBD folders.  That was version 2.8.something -- if there is a file that contains it, I can find the exact version number from that same backup if it will help to narrow down the date of any changes.  The duplicate .SBD folders appeared immediately after upgrading to 2.9.1.

> If yes, who created directory of upper case ".SBD"?

Some of the affected folders were created as long ago as 2007 by SeaMonkey under Windows XP, although most were created on 2011-06-23 when I installed SeaMonkey on my latest Windows 7 laptop and reorganized the folder structure.  I am certain that I never manually changed any file suffixes from .sbd to .SBD.  Even if something did change them for some reason (such as a copy utility like FolderMatch -- but I can't imagine why), SeaMonkey was happy with ".SBD" until the latest update.

> Did you migrate or move/copy mail folder files/directories from FAT drive in
> the past?

No, never.  Some of the archived email itself dates back to 1993 from an IBM mainframe, and at some point in the late 1990's I converted those archives to Netscape on a Windows NT (NTFS) system, but those account for only one of about fifty folders affected by the .SBD problem.

> Or you say "xxx.SBD(upper case) folder problem is reproduced by Sm 2.9.1 on
> Win7 too by intentional creation of .SBD directory in upper case"?

I discovered the problem after spending many hours trying to restore my email directories after the SeaMonkey update:  I noticed that the only affected folders had suffix .SBD.  (There were lots of lower case .sbd folders, too, although looking at my backup I can't discern an obvious pattern distinguishing folders with .sbd from those with .SBD; most folders of both categories were created on 2011-06-23.)  I eventually found another blog reference to the problem regarding SeaMonkey that seemed to confirm my diagnosis and got my desired folder structure back by manually changing .SBD to .sbd and deleting duplicate directories.  Then I searched here to determine if anyone had reported the same problem in Thunderbird.  When I did, I downloaded the latest version of Tb and installed it on another Win7 computer for testing and obtained the result described.

Comment 15

[[e-mail]]

I'll try restoring the file structure immediately before the TB12 upgrade, rename all .SBD to .sbd and try running TB again.  I can't do this until tonight (23:00 GMT) because of travel and work but I'll let you know the result overnight.

Does anyone know how to re-download emails by POP because I'll have to get the emails from 4th May forward again.  I intend restoring the account/Inbox directories as well.  Will that be enough?  The emails are still on the server for 30 days.

Comment 16

[WADA:World Anti-bad-Duping Agency]

(In reply to [e-mail] from comment #15)
> Does anyone know how to re-download emails by POP (snip)
> The emails are still on the server for 30 days.

If you already checked "Leve Messages on Server" at Server Settinfs of the POP3 account, and if you didn't execute operation which invoke "delete from POP3 server" yet, you can re-download mails still held at POP3 server again by "delete popstate.dat of the POP3 account and restart of Tb".
However, nothing is deleted by this bug unless you did operation like "delete xxx.SBD folder which was wrongly shown at folder pane". Mail folders/subfolders and mail data in them will be shown again by "rename xxx.SBD(upper case) to xxx.sbd(lower case)" and "delete excess garbage of xxx.SBD.msf".

Comment 17

[WADA:World Anti-bad-Duping Agency]

Bug for "add support for pluggable stores" is bug 402392. Sory for wrong linkify.

Comment 18

[[e-mail]]

(In reply to WADA from comment #16)
> (In reply to [e-mail] from comment #15)
> > Does anyone know how to re-download emails by POP (snip)
> > The emails are still on the server for 30 days.
> 
> If you already checked "Leve Messages on Server" at Server Settinfs of the
> POP3 account, and if you didn't execute operation which invoke "delete from
> POP3 server" yet, you can re-download mails still held at POP3 server again
> by "delete popstate.dat of the POP3 account and restart of Tb".
> However, nothing is deleted by this bug unless you did operation like
> "delete xxx.SBD folder which was wrongly shown at folder pane". Mail
> folders/subfolders and mail data in them will be shown again by "rename
> xxx.SBD(upper case) to xxx.sbd(lower case)" and "delete excess garbage of
> xxx.SBD.msf".

Thanks, I'll remember that.  What I want to do is recover emails in the local folders which have been deleted in the data restore.  I now think that I can do that by copying all since the changeover into a separate folder and then putting them back into account/inbox when the restore is complete and let the rules send them to the right folder.

Comment 19

[David :Bienvenu]

fwiw, tb 12 code doesn't treat the case of .sbd any differently than tb 11 - i.e., they both expect the ".sbd" part of the directory name to be lower case. Which makes me think the difference has to do with how we treat directories we don't expect to see in local mail folder directories. I'll poke around a bit.

Comment 20

[[e-mail]]

My problem is now resolved.

I saved all emails in Local Folders since installing TB12 to a separate folder under the account and closed down TB.  I moved the existing 'Local Folders', 'Local Folders.msf' etc to a new dummy directory and restored the 'Thunderbird' directory which contained them back to the state it was in just before installing TB12.  

I then changed all 'folder.SBD' to 'folder.sbd' in the file system, started TB12 and it was back to normal as at 4th May.  

Finally I moved the emails from the rescue folder under the account to the account/inbox and ran the filters to send them back to their appropriate folder under 'Local Folder'.  Further checks have revealed no problems.

All of the 'folder.SBD' were created via TB but there were also some which were just 'folder.sbd', though not with duplicate folder names.  Perhaps this is a relic of an earlier version since I've been using TB since 2003 if not before then.  Even so, I'm surprised that TB distinguishes between upper and lower case in the extension.

Comment 21

[[e-mail]]

Sorry, I pressed 'Save Changes' too early and didn't thank you all for your help and advice.  So, Thank you one and all.

Comment 22

[[e-mail]]

Just when you thought it was safe to come out...  

I've discovered that the same thing has happened with a .MSF file which has been converted into a folder.  This problem has also been resolved by regenerating the real folder's directory and deleting the empty 'folder.MSF' folder.

Comment 23

[David :Bienvenu]

(In reply to [e-mail] from comment #22)
> Just when you thought it was safe to come out...  
> 
> I've discovered that the same thing has happened with a .MSF file which has
> been converted into a folder.  This problem has also been resolved by
> regenerating the real folder's directory and deleting the empty 'folder.MSF'
> folder.

Do you have an anti-virus program installed? We've heard other reports of random non-mail folder files turned into directories.

Comment 24

[Joe Sabash [:JoeS1]]

(In reply to [e-mail] from comment #22)
> Just when you thought it was safe to come out...  
> 
> I've discovered that the same thing has happened with a .MSF file which has
> been converted into a folder.  This problem has also been resolved by
> regenerating the real folder's directory and deleting the empty 'folder.MSF'
> folder.

Stan..You very well may be the man.
We have an issue in bug 750781 where files were inexplicably converted to folders.
Capitalization as an issue was not explored there.
Thanks for the feedback.

Comment 25

[WADA:World Anti-bad-Duping Agency]

(In reply to [e-mail] from comment #22)
> I've discovered that the same thing has happened with a .MSF file which has
> been converted into a folder.

As for "xxx.MSF" on Win XP, "xxx.msf(lower case) is re-created by internal re-build index by Tb on Win.
- create DIRECTORY for mail filder FILEs of Tb  
  xxx : DIRECTORY, xxx.MSF : DIRCTORY
- Restart Tb.
  Local folder of xxx and xxx.MSF are shown, and xxx.MSF.msf is created.
  By click(open) of mail folder named "xxx",
  xxx.MSF(DIRECTORY) is deleted, and xxx.msf(lower case, FILE) is created by Tb.
- Restart Tb again. 
  Local folder named "xxx.MSF" disappears. xxx.MSF.msf is not deleted.
  Because xxx is DIRECTORY, "copy/move mails to xxx folder" silently fails
  and Tb does do nothing. Error is reported to nowhere by Tb.
So, as for upper case xxx.MSF issue on Win(after Win-NT), problem automatically disappears by "Restart of Tb" only, because MS Win's file system is case insensitive and MS Win's file system keeps case of file/directory name.  

A problem in Mac OS X is; There are two file names.
 (a) Name presented to applicasion by Finder, which pehaps is still partially
     similar to old MS DOS/Win(MS copied baddness of old Apple's System N.x :-) )
 (b) Name presented to application by file system which is similar to Linux.
See http://en.wikipedia.org/wiki/Finder_%28software%29 for Finder, and
http://en.wikipedia.org/wiki/File_system#Mac_OS_X for file system of Mac OS X.
Following is example of "file name length limitation" in (a).
> http://osxdaily.com/2011/11/25/show-long-file-and-folder-names-on-mac-os-x-desktop/

Comment 26

[[e-mail]]

(In reply to David :Bienvenu from comment #23)
> (In reply to [e-mail] from comment #22)
> > Just when you thought it was safe to come out...  
> > 
> > I've discovered that the same thing has happened with a .MSF file which has
> > been converted into a folder.  This problem has also been resolved by
> > regenerating the real folder's directory and deleting the empty 'folder.MSF'
> > folder.
> 
> Do you have an anti-virus program installed? We've heard other reports of
> random non-mail folder files turned into directories.

Yes I have the Sophos free anti-virus for Mac installed.  It's been installed since last autumn and there's been at least one previous update to TB since then.

Comment 28

[Barbara]

I filed a bug report 762337 yesterday, which is similar to this one, though my symptoms were a bit different.  I have now flagged that one as a duplicate.  I'm on Windows 7 and didn't have as extensive a problem as the poster above.
  Barbara

Comment 29

[:aceman]

Well, actually in bug 402392 this code was removed:

-static bool
-nsStringEndsWith(nsString& name, const char *ending)
-{
-  PRInt32 len = name.Length();
-  if (len == 0)
-    return PR_FALSE;
-
-  PRInt32 endingLen = strlen(ending);
-  return (len > endingLen && name.RFind(ending, PR_TRUE) == len - endingLen);
-}
-
-static bool
-nsShouldIgnoreFile(nsString& name)
-{
-    ...
-    return (nsStringEndsWith(name, ".mozmsgs") || nsStringEndsWith(name,".sbd") ||
-            nsStringEndsWith(name,SUMMARY_SUFFIX));
-}

And replaced with:

+nsMsgLocalStoreUtils::nsShouldIgnoreFile(nsAString& name)
+{
+  ...
+  // The .mozmsgs dir is for spotlight support
+    return (StringEndsWith(name, NS_LITERAL_STRING(".mozmsgs")) ||
+            StringEndsWith(name, NS_LITERAL_STRING(".sbd")) ||
+            StringEndsWith(name, NS_LITERAL_STRING(SUMMARY_SUFFIX)));
+}

Notice "name.RFind(ending, PR_TRUE)" in the original code. There are multiple versions of RFind in the codebas, but considering the second argument is a bool, this one looks to be matching:
http://mxr.mozilla.org/comm-central/source/mozilla/xpcom/string/nsTString.h#188
int32_t RFind(const char* aCString, bool aIgnoreCase = false,
189                 int32_t aOffset = -1, int32_t aCount = -1) const;

And the code sent "true" as the second argument, meaning ignore case in string comparison. That means. *.sbd AND *.SBD was being hidden in the folder pane in the old code.

In the new code StringEndsWith() seems to be case sensitive so only *.sbd is hidden. That could explain this bug report. However it does not mean *.SBD folders were ever used for anything useful. They could be empty and not used in old TB versions. Just got exposed after the code change but still doesn't mean they cause any harm.

So I am not sure if there is anything to do in this bug. If we decide to hide *.SBD folders in the UI they will not be used for anything, so will just silently lie dead in the filesystem adding confusion to the user if he looks in the raw filesystem (Local Directory of the account).