752085 - Local folders and messages missing, if full path length exceeds path length limitation of OS(MAX_PATH=260) by rename of mail folder, even though NTFS filesystem already supports full paths up to about 32,767 characters

Status: RESOLVED DUPLICATE of bug 249807

(Thunderbird :: Folder and Message Lists, defect)

Description

[Kevin Fegan]

I moved messages from IMAP folders to new local folders.

Later, I renamed some of the local folders.

Then, when trying to access the local folders, some could not be reached (folders would not show any sub-folders and would show "???" in the "Size" column.

Investigating through the file-system with Windows-Explorer, I found the folders and messages were present, but the path name was possibly too long for Thunderbird to navigate (folder-path 265 characters long).

From the Thunderbird folder list, I renamed some of the parent folders to shorter names so the file-system paths to all the sub-folders and messages was below 230 characters long.

After renaming the folders, all the sub-folders were reachable from the Thunderbird folder list, but the message-files and .msf files were no longer present. Thunderbird deleted these message-files/messages without warning or notification when Thunderbird renamed the folders.

This caused the loss of about 10 message-files and over 100 megabytes of emails.

None of these message files or messages can be found in the Trash or in any other folders.

I believe Thunderbird should be able to handle long-path names, and that Thunderbird should not delete files and messages without prompting to allow the action.

Comment 3

[WADA:World Anti-bad-Duping Agency]

(In reply to Kevin Fegan from comment #0)
> Later, I renamed some of the local folders.

> Investigating through the file-system with Windows-Explorer, I found the
> folders and messages were present, but the path name was possibly too long
> for Thunderbird to navigate (folder-path 265 characters long).

There are two kinds of length limitation.
(a) 256 bytes limit of full file path length by MS Win
(b) Around 128 bytes limit of file name length by Tb.
    If longer, file name length is shorten by hashing.
    See bug 416888.
Because Tb uses <folder_name> and <folder_name>.msf and <folder_name>.sbd, both full file path length of .msf/.sbd and file name length of .msf/.sbd should be within length limitation.
As seen in bug 576025, if IMAP folder name length is too long and full file path for .msf file exceeds length limit of (a), Tb can't show the IMAP folder at folder pane even when subscribe of the IMAP folder itself is successful.

What is actual file path of file/directory for local mail folder which is lost after rename to too long full file path??  
You can see file names under Tb's mail directory by next.
- <mail_dir_for_an_account> = path set at Local Directory: of Server Settings  
- At Command Prompt.
  DIR "<mail_dir_for_an_account>" /S > C:\DIR-LIST.TXT
  notepad.exe C:\DIR-LIST.TXT 

> I believe Thunderbird should be able to handle long-path names, (snip)

How can a software on MS Win use longer path name than limit by MS Win?

Comment 4

[Kevin Fegan]

After further investigation, I found that these folders and files/messages were not actually deleted. They were present on the hard drive, but they were not visible or accessible from Thunderbird.


> There are two kinds of length limitation.
> (a) 256 bytes limit of full file path length by MS Win
> (b) Around 128 bytes limit of file name length by Tb.
In my case all the individual path elements and filenames were shorter that 128 characters.

I understand that there are similar issues with IMAP folders, but in my case, at this time, I am speaking about files and folders within the "Local Folders" path.

> What is actual file path of file/directory for local mail folder which is lost
after rename to too long full file path??

In my case, the path to "Local Folders" is:
C:\Users\<usernamefolder>\AppData\Roaming\Thunderbird\Profiles\fb35fw3z.default\Mail\Local Folders\
this uses just a few characters less than 100.


Step 1:
I created a folder within "Local Folders" named by my name and my IMAP account name, and a further sub-folder named "Archive" so something like:
C:\Users\<usernamefolder>\AppData\Roaming\Thunderbird\Profiles\fb35fw3z.default\Mail\Local Folders\My Full Name-IMAP Account ID.sbd\Archive.sbd

This added about 35 to 50 characters for a total of about 150.


Step 2:
From there I created folders from the structure from my IMAP account.

I have reorganized and renamed the original folders so I can't quote them exactly, but an example would be something like:

\Local Folders\My Full Name-IMAP Account ID\Archive\Some 25 letter foldername\Some thirty--character folder\Thirty character foldername-2\Mailbox

The file would be located like this (about 240 to 250 characters):
C:\Users\usernam\AppData\Roaming\Thunderbird\Profiles\fb35fw3z.default\Mail\Local Folders\My Full Name-IMAP Account ID.sbd\Archive.sbd\Some 25 letter foldername.sbd\Some thirty--character folder.sbd\Thirty character foldername-2.sbd\Mailbox


Step 3:
At some point, I did some re-organizing of these "Local Folders" and ended up moving some folders around, and renaming some folders. An example might be something like:

\Local Folders\My Full Name-IMAP Account ID\Archive\Some 25 letter foldername\Some thirty--character folder\Some slightly longer Fifty character foldername-1\Mailbox

C:\Users\usernam\AppData\Roaming\Thunderbird\Profiles\fb35fw3z.default\Mail\Local Folders\My Full Name-IMAP Account ID.sbd\Archive.sbd\Some 25 letter foldername.sbd\Some thirty--character folder.sbd\Some slightly longer Fifty character foldername-1.sbd\Mailbox

At this point, I can navigate to the "Mailbox" folder but the size shows as "???" and I cannnot see any of the messages in the "Mailbox" folder.


Step 4:
I renamed the "Some slightly longer Fifty character foldername-1" folder back to "Thirty character foldername-2" and everything was back to normal.
\Local Folders\My Full Name-IMAP Account ID\Archive\Some 25 letter foldername\Some thirty--character folder\Thirty character foldername-2\Mailbox

C:\Users\usernam\AppData\Roaming\Thunderbird\Profiles\fb35fw3z.default\Mail\Local Folders\My Full Name-IMAP Account ID.sbd\Archive.sbd\Some 25 letter foldername.sbd\Some thirty--character folder.sbd\Thirty character foldername-2.sbd\Mailbox


Step 5:
Add a "New" folder and move existing folder into the "New" folder:
\Local Folders\My Full Name-IMAP Account ID\Archive\New\Some 25 letter foldername\Some thirty--character folder\Thirty character foldername-2\Mailbox

C:\Users\usernam\AppData\Roaming\Thunderbird\Profiles\fb35fw3z.default\Mail\Local Folders\My Full Name-IMAP Account ID.sbd\Archive.sbd\New.sbd\Some 25 letter foldername.sbd\Some thirty--character folder.sbd\Thirty character foldername-2.sbd\Mailbox

At this point, all of these folders are visible and accessable, and the email messages in "Mailbox" are also readable.


Step 6:
Rename the "New" folder to "New Foldername With Length--30"

C:\Users\usernam\AppData\Roaming\Thunderbird\Profiles\fb35fw3z.default\Mail\Local Folders\My Full Name-IMAP Account ID.sbd\Archive.sbd\New Foldername With Length--30

The path to this folder is about 160 characters long.
At this point, all the folders below the new folder "New Foldername With Length--30" are no longer accessible and no longer visible:

Path length of about 190 but no longer visible: \Some 25 letter foldername
Path length of about 225 but no longer visible: \Some 25 letter foldername\Some thirty--character folder
Path length of about 258 but no longer visible: \Some 25 letter foldername\Some thirty--character folder\Thirty character foldername-2
Path length of over 260 and no longer visible: \Some 25 letter foldername\Some thirty--character folder\Thirty character foldername-2\Mailbox

C:\Users\usernam\AppData\Roaming\Thunderbird\Profiles\fb35fw3z.default\Mail\Local Folders\My Full Name-IMAP Account ID.sbd\Archive.sbd\New Foldername With Length--30.sbd\Some 25 letter foldername.sbd\Some thirty--character folder.sbd\Thirty character foldername-2.sbd\Mailbox


If I view the folders on disk, I can see "New" folder is still in place, including all the missing files and folders below it. The folder was not renamed, but instead, a new empty folder "New Foldername With Length--30.sbd" was created:

C:\Users\usernam\AppData\Roaming\Thunderbird\Profiles\fb35fw3z.default\Mail\Local Folders\My Full Name-IMAP Account ID.sbd\Archive.sbd\New.sbd\Some 25 letter foldername.sbd\Some thirty--character folder.sbd\Thirty character foldername-2.sbd\Mailbox

This step has inconsistant/unpredictable results... After deleting these folders and repeating this step about 5 times, sometimes none of the folders below the renamed folder are visible, and sometimes some of them will be visible but have "???" in the size column.

also, at this point, sometimes if I rename the folder "New Foldername With Length--30" back to "New" then everything will be restored... sometines not.




> How can a software on MS Win use longer path name than limit by MS Win?

The 260 character path/filename limit is a throwback to older versions of Windows and 16-bit filesystems.

The Windows (NTFS) filesystem supports full paths up to (about) 32,767 characters. This has been the case since Windows NT 4.0 (Windows NT 4.0 began shipping about July 31st 1996. Final release Windows NT 4.0 SP6a. Released: July 26th 2001).

Information on how to implement this can be found on this page:
http://msdn.microsoft.com/en-us/library/aa365247(v=vs.85).aspx#maxpath
... Look for the paragraph that begins:
"The Windows API has many functions that also have Unicode versions"

Also a great 3-part article with code examples:
http://blogs.msdn.com/b/bclteam/archive/2007/02/13/long-paths-in-net-part-1-of-3-kim-hamilton.aspx


Even while working within the 260 character limit, there are many issues described above where Thunderbird could have at least provided warning information when attempts were made to exceed the path/filename length limit.

Also, I feel more could be done by Thunderbird to recover child files and folders for these cases when folders are moved or renamed.

Comment 5

[WADA:World Anti-bad-Duping Agency]

(In reply to Kevin Fegan from comment #4)
> This step has inconsistant/unpredictable results...

As seen in Location: of folder's Properties/General, internal mail folder path doen't have .msf/.sbd part which is used for full file path.
You are perhaps looking difference among following cases.
    ...\<folder>            ...\<folder>.msf       ...\<folder>.sbd
(a) within limit            within limit           within limit
    => folder is shown      => count is shown      => child folders are shown
(b) within limit            exceeds limit          exceeds limit
    => folder is shown      => count is NOT shown  => child folders are NOT shown
(c) exceeds limit           exceeds limit          exceeds limit
    => folder is NOT shown  => count is NOT shown  => child folders are NOT shown

> Thunderbird could have at least provided warning information
> when attempts were made to exceed the path/filename length limit.

I definetely agree with you.

> http://msdn.microsoft.com/en-us/library/aa365247(v=vs.85).aspx#maxpath

Do you know path length limitation on Linux and Mac OS X?

Comment 6

[WADA:World Anti-bad-Duping Agency]

Quick observation of behaviour of Tb and Win-XP's Windows Explorer.

(1) If new local mail folder creation, Tb doesn't create requested folder if path length of ...\New_Folder.msf folder exceeds length limit. Two patterns were observed.
(1-A) Tb silently fails to create newfolder. Nothing happen in UI. User needs to cancel.
(1-B) When 1234567 folder exists and path length of ...parent.sbd\1234567.msf is max length, Tb shows "file for folder already exists" error when creation of 12345678 is requested.
The difference may be 8.3 format file name relevant phenomenon.

(2) "Path length of directory" may be different from "path length of file" in Win. This also may be 8.3 format file/directory name relevant phenomenon.
Folder structure.
  Local Folders
    123456789A123456789B123456789C123456789D123456789E      (50 bytes)
      123456789F123456789G123456789H123456789I123456789J    (50 bytes)
        123456789K                                          (10 bytes)
          123456789L                                        (10 bytes)
            123456789M                                      (10 bytes)
Following is full path of max length;
> C:\Documents and Settings\wada\Application Data\Thunderbird\Profiles\ygo7ovdf.XXX\Mail\Local Folders\123456789A123456789B123456789C123456789D123456789E.sbd\123456789F123456789G123456789H123456789I123456789J.sbd\123456789K.sbd\123456789L.sbd\123456789M.msf
(2-1) If creation of 123456789M1 is requested, it silently fails because of length limitation. This is phenomenon of (1-A).
(2-2) Even when parent folder name of 123456789M is shorter like next,
      creation of sub folder under <123456789 to 123>/123456789M
      silently fails.  
  Local Folders
    123456789A123456789B123456789C123456789D123456789E      (50 bytes)
      123456789F123456789G123456789H123456789I123456789J    (50 bytes)
        123456789K                                          (10 bytes)
          123456789                                         ( 9 bytes)
            123456789M                                      (10 bytes)
              unable to create sub folder under 123456789M
          (from 12345678 to 1234 is cut)
          123                                               ( 3 bytes)
            123456789M                                      (10 bytes)
              unable to create sub folder under 123456789M
(2-3) If parent folder name of 123456789M is shorten to "12",
      creation of sub folders under 12/123456789M
      is normally executed until max size.
  Local Folders
    123456789A123456789B123456789C123456789D123456789E      (50 bytes)
      123456789F123456789G123456789H123456789I123456789J    (50 bytes)
        123456789K                                          (10 bytes)
          12                                                ( 2 bytes)
            123456789M                                      (10 bytes)
              1                                             ( 1 byte ) 
              (2 to 123456 is cut)
              1234567                                       ( 7 byte ) 
              If 12345678 is requested, "file already exists" is shown.
              This is phenomenon of (1-B).

(3) If parent folder is renamed to longer, for examle 123456789K is renamed to
    123456789K123456, 123456789K and 123456789K.msf is renamed to 123456789K12345
    and 23456789K123456.msf because of "within limit",
    but "rename of 123456789K.sbd to 123456789K123456.sbd by Tb" or "move/rename
    xxx.sbd or xxx.msf under it by Tb" silently fails due to length limitation.
    You perhaps observed this phenomenon.

(4) Once file of "longer than file path length limitation" is created
    by rename of higher directory by Tb, Windows Explorer of MS Win does nothing
    on file of "longer than file path length limitation" and directory which
    directly contains the file of "longer than file path length limitation".
    Delete/Rename of file/directory via Windows Explorer is impossible.
    Rename of higher directory to shorter one is needed for manual recovery
    operation by user.

Confirming.

Comment 7

[WADA:World Anti-bad-Duping Agency]

For length limitation of file path/file name in Mac OS X.
See http://en.wikipedia.org/wiki/Finder_%28software%29 for Finder, and
http://en.wikipedia.org/wiki/File_system#Mac_OS_X for file system of Mac OS X.
Following is example of "file name length limitation" in Finder.
> http://osxdaily.com/2011/11/25/show-long-file-and-folder-names-on-mac-os-x-desktop/

Comment 9

[sam63]

Well, you can use Long Path Tool, for such issues...

Comment 10

[aiden carter]

The Long path tool is the very best program for error, unlock solution. 
Try it and solved your problem. 
I used the long path tool and I solved my error, unlocks problem solution.