249807 - Hidden IMAP sub-folder if total length of folder branch exceed 128 char (No error message when path_length for local file for mail folder=".msf" exceeds MS Win's path length limit=256 chars) - Implement extended-length path by using \\?\ prefix on Windows

Status: NEW

(Thunderbird :: Folder and Message Lists, defect)

Description

[hansen]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7) Gecko/20040704 Firefox/0.9.0+
Build Identifier: version 0.7+ (20040704)

Thunderbird doesn't show subfolders if the total folderpath length exceeds 128chars.

On local accounts you aren't allowed to exceed the limit - if you do, it just
don't close the "create folder" window.
On imap accounts it continues, but if the exceed the limit, then they aren't shown.

Reproducible: Always
Steps to Reproduce:
1. Create a folder "some stupid very long name"
2. Continue #1, but as subfolder of last created folder

Actual Results:  
you'll only reach 4th level or such, when you aren't allowed to create another
subfolder(local) or they just don't show up(imap)

Expected Results:  
it should show the folders

Comment 1

[mikey]

Please, i need this bug quickly close, I have big problem with that.
We have a lot of subfolders in our company ...

thks,
Alain

Comment 2

[hansen]

everybody need a quick fix - make more parent folders to put all your
sub-folders in...

Comment 3

[hansen]

still exist in TB 20041113 BRANCH

Comment 4

[Gervase Markham [:gerv]]

This is an automated message, with ID "auto-resolve01".

This bug has had no comments for a long time. Statistically, we have found that
bug reports that have not been confirmed by a second user after three months are
highly unlikely to be the source of a fix to the code.

While your input is very important to us, our resources are limited and so we
are asking for your help in focussing our efforts. If you can still reproduce
this problem in the latest version of the product (see below for how to obtain a
copy) or, for feature requests, if it's not present in the latest version and
you still believe we should implement it, please visit the URL of this bug
(given at the top of this mail) and add a comment to that effect, giving more
reproduction information if you have it.

If it is not a problem any longer, you need take no action. If this bug is not
changed in any way in the next two weeks, it will be automatically resolved.
Thank you for your help in this matter.

The latest beta releases can be obtained from:
Firefox:     http://www.mozilla.org/projects/firefox/
Thunderbird: http://www.mozilla.org/products/thunderbird/releases/1.5beta1.html
Seamonkey:   http://www.mozilla.org/projects/seamonkey/

Comment 5

[hansen]

After some thought, this might be a Windows(XP) issue with paths max length at 256.

Well, it's still reproduceable, but what should be done? Error/warning message
perhaps?

Comment 6

[Wayne Mery (:wsmwk)]

suite Bug 283446 confirmed.  I assume the same is true of thunderbird, but I didn't test.

hansen, how do you come up with 128 characters?

Comment 7

[hansen]

I submitted this bug for another person from the forums.

Now I've tested again with Thunderbird version 3 alpha 1 (20060528) which fixes this bug.

There isn't any hold at 128 chars - it continues up to 256 chars, which is max for Windows XP. It might go higher on unix-like systems.

And there's a popup-message, when hitting the OS limit, but sometimes I get an error instead:

Error: [Exception... "Component returned failure code: 0x80004003 (NS_ERROR_INVALID_POINTER) [nsIXULTreeBuilder.getIndexOfResource]"  nsresult: "0x80004003 (NS_ERROR_INVALID_POINTER)"  location: "JS frame :: chrome://messenger/content/msgMail3PaneWindow.js :: EnsureFolderIndex :: line 1593"  data: no]
Source File: chrome://messenger/content/msgMail3PaneWindow.js
Line: 1593

The strange thing is; now when the popup-message doesn't show and I get the error in the console, then the folder is actually created on the server (FreeBSD 5.3, Courier-IMAP 4.0.1), but not on the client (WinXP).

I'm not sure if I should close this bug and open a new one on the error...?

Comment 12

[WADA:World Anti-bad-Duping Agency]

If non-ascii folder name is used at IMAP server, path length of local mail folder file exceeds system limit at "smaller folder depth/shorter mail folder name" than user's expectation, because non-ascii folder name is passed in modified UTF-7 and the string of modified UTF-7 is used as local mail folder file name.
 - Folder name    : N chars (in unicode) == N*2 bytes in UTF-16
 - modified UTF-7 : roughly (N*2)*(4/3) asci chars(bytes)
                    (modified UTF-7 is similar to base64 encoding)

Comment 13

[WADA:World Anti-bad-Duping Agency]

Similar situation can occur on local mail folder(POP3 or "Local Folders"), if mail folder file is manually copied.
> Copied mail folder file : C:\<mail_directory>\...sbd\...\...sbd\file_name
> Path length of ...\file_name is less than or equal to path length limit
> Path length of ...\file_name.msf exceeds path length limit
In this case, Tb tries to create ...\file_name.msf but it fails.

One of the reasons why this kind of issue mainly occurs on MS Win is "very long path of default profile location".
>(example of path for INBOX on MS Win XP)
> C:\Documents and Settings\username\Application Data\Thunderbird\Profiles\cbz7k273.Trunk-Test\ImapMail\imap.gmail.com\INBOX.msf
"C:\Doc...\ImapMail\imap.gmail.com" consumes 116 chars(nearly half of path length limit of MS Win).

Comment 16

[ppau2004]

Happens to Windows7 too. I have to map a drive as a workaround. From the command line:
subst x: /path/to/long/folder

Comment 17

[login]

Still looks like an issue, with bug 1785747

Comment 18

[Lukas.Wringer]

Wow, this "Bug" is ancient... Stumbled across it yesterday and wasted a whole day debugging for one of our users (as it wont happen on my Linux machine this drove me nuts). For the far future you maybe should think of a different way to organize mails than via filesystem hierarchy as even the new maildir still relies on folder-trees to organize the files... (maybe even scrap both ways and use a fast database and only put the attachments to filesystem...)

Could you for now at least show a useful error message on creating, renaming and moving a folder if the new Path would exceed the OSes Limits? At the moment it seems to work (and writes to the IMAP-Server) but the folder is not shown

Also this has not a fixed limit to 128 chars - it depends on how long the base path to the thunderbird profile folder is, how long the URL of the IMAP server is and only then comes the limit inside the mail accounts folder structure...

-> You may update the affected Fields of this bu to Windows 10/11 (or any Windows version) and Thunderbird 115...?

Comment 19

[Lukas.Wringer]

Still Present on Windows 11 and TB 128esr - I also tried the registry key for longer paths in Windows (https://learn.microsoft.com/de-de/windows/win32/fileio/maximum-file-path-limitation?tabs=powershell) which is available since Windows 10 ver. 1607 but sadly it seems Thunderbird has not been configured / patched to be aware of the now longer path limits?

Comment 20

[Richard Leger]

As per comment Bug 124287 Comment 31 issue is still happening in TB 136.0b3 (64-bit) on Windows 10.

"...

Also just experienced recently in TB Beta, after moving folders, some of the sub-folders disappeared from TB in my IMAP account without error or noticed, while there were still present in webmail. This prevented access to the folders and emails it contained.

After some lengthy investigation, it turned out the issue was TB not able to create the .msf file for few of the sub-folders because the 255 full file path limit was exceeded on Windows, so Windows itself refused to create such files with too long path.

At least TB shall raise/log the error in such situation, not just silently hide the folder(s).

Part of the issue is that the prefix path to the mailbox folder is already very long something like:
C:\Users\richard\AppData\Roaming\Thunderbird\Profiles\cnantXXX.default\ImapMail\mail.my-domain.com
...so if you then have a many sub-folders the full file path can easily reach/exceed the 255 characters limit of Windows.

Should it be time for TB to start using \\?\ prefix to specify an extended-length path?

For example, \\?\C:\<very long path> rather than just C:\<very long path> which is causing issues on Windows.

..."

Comment 21

[Worcester12345]

(In reply to Lukas.Wringer from comment #19)

Still Present on Windows 11 and TB 128esr - I also tried the registry key for longer paths in Windows (https://learn.microsoft.com/de-de/windows/win32/fileio/maximum-file-path-limitation?tabs=powershell) which is available since Windows 10 ver. 1607 but sadly it seems Thunderbird has not been configured / patched to be aware of the now longer path limits?

I wonder what percentage of Windows 10 has this capability. Also, I wonder how important this even is any longer, with Windows 10 slipping into history. At what point should developers just give up on the old stuff, and the slightly less old stuff, and just focus on more modern from X point in time forward. Is there any worth or value to program for diminishing returns?

Comment 22

[Lukas.Wringer]

(In reply to Worcester12345 from comment #21)

I do not understand your point - "since Windows 10 ver. 1607" means it is a relatively new feature and it exists in Windows 11 and forward as well - so windows 10 being phased out has nothing to do with it...

You may only argue that this feature has to be enabled - but the other solution for this annoying problem would require a complete rebuilt in how Thunderbird stores messages. I personally am fine with both ways if it fixes the issue - but I presume you have somewhat limited resources you would have to triage?

Comment 23

[Wayne Mery (:wsmwk)]

Sorry, I don't have anything significant to add here other than this capability would need to change in code shared with firefox, which I think is unlikely to happen.

But it certainly would be good to improve messaging to warn the user when this is happening.

Comment 24

[Ben Campbell]

(In reply to Wayne Mery (:wsmwk) from comment #23)

Sorry, I don't have anything significant to add here other than this capability would need to change in code shared with firefox, which I think is unlikely to happen.

I don't have the bandwidth for this at the moment (my windows knowledge is a little out of date), but I definitely think it should be done. I'd consider it a reasonably serious flaw.
Path lengths don't tend to be too bad on personal windows machines, but in corporate/institutional settings, they do have a tendency to explode to very unwieldy lengths!

I think the first step is to make up some unit tests which unambiguously illustrate the problem. I suspect the fix wouldn't actually be tooooo hard - it's just the fear of tinkering with such a core low-level part of m-c (the firefox code).

Comment 25

[Richard Leger]

(In reply to Wayne Mery (:wsmwk) from comment #23)

Sorry, I don't have anything significant to add here other than this capability would need to change in code shared with firefox,

I don't think you would need to wait on Firefox change for that to happen in Thunderbird. Can't you simply start using "\?\D:\very long path" (https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry) to access path especially the local cache folder path for IMAP mailboxes for example? Would that really require a change in Firefox?

The first valuable change in Thunderbird, would be definitely to change the error message to clarify the reason why it cannot write to disk due to path to long, with suggestions to help fix the issue.

The second thing Thunderbird could look at is maybe reducing the path towards the local IMAP cache folders...
Currently it may already take 115ish characters path just to reach the root of the mailbox folder:
C:\Users\<username>\AppData\Roaming\Thunderbird\Profiles\eqs5239z.default-beta\ImapMail\remote.my-supperlongdomain.co.uk
...but that is more tricky as this string may change based on user settings.

Finally Thunderbird could start using "\?\D:\very long path" feature, which I don't see how that is dependant on Firefox.
Doesn't Thunderbird already have its own Application manifest updates that would allow it to declare its long path capability/readiness?
https://searchfox.org/comm-central/source/mail/app/thunderbird.exe.manifest

The application manifest must also include the longPathAware element.
XML

<application xmlns="urn:schemas-microsoft-com:asm.v3">
    <windowsSettings xmlns:ws2="http://schemas.microsoft.com/SMI/2016/WindowsSettings">
        <ws2:longPathAware>true</ws2:longPathAware>
    </windowsSettings>
</application>

Comment 26

[Richard Leger]

For those that may be interested, to fix the issue manually, here is a Powershell script to identify the too long file in ImapMail location (e.g .msf file with too long path) that may be problematic to access within Thunderbird (browsing/compacting):

Get-ChildItem  -Recurse -Force  | Where-Object {$_.FullName.Length -gt 259}| select @{N="PathLength";E={$_.FullName.Length}},FullName|Sort-Object pathlength -Descending|ft -autosize -wrap

Basically any .msf file with a FullName.length >= 260 character will not be usable by Thunderbird. If you try to access the folder, Thunderbird will raise an error.

For each path found by the script, you would need to rename folders and parent folders within Thunderbird to reduce the path to 259 characters or lower e.g 255.

As notice on Windows 11, Thunderbird may not be able to remove the old too long files (e.g msf) after the rename, but you can manually remove them via File Explorer (manual delete) after the rename occurred.

Comment 27

[Richard Leger]

Attachment: fullpathtoolong.png

(In reply to Richard Leger from comment #25)

(In reply to Wayne Mery (:wsmwk) from comment #23)
The first valuable change in Thunderbird, would be definitely to change the error message to clarify the reason why it cannot write to disk due to path to long, with suggestions to help fix the issue.

As per attachment, the error message seems to have been updated already as seen in Thunderbird beta 142.0b2 (64-bit) on Windows 11. It does indicate "full path too long", meaning the full path to the .msf file, within the folder, that cannot be opened by Thunderbird. This of course may also affect compacting, not only browsing for the message list, but I have not seen it yet.

Comment 28

[Worcester12345]

(In reply to Ben Campbell from comment #24)
...

... I definitely think it should be done. I'd consider it a reasonably serious flaw.
...
I think the first step is to make up some unit tests which unambiguously illustrate the problem. I suspect the fix wouldn't actually be tooooo hard - it's just the fear of tinkering with such a core low-level part of m-c (the firefox code).

Has this "first step" been taken yet?