1619218 - emails in folder are unreadable with folder name "RÓŻNE"

Status: RESOLVED WORKSFORME

(Thunderbird :: Folder and Message Lists, defect)

Description

[Piotr Szurko]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:73.0) Gecko/20100101 Firefox/73.0

Steps to reproduce:

Create and name folder by polish word "RÓŻNE" (must be uppercase).

Actual results:

emails in folder are unreadable. Change folder to lowercase or any other name fix issue

Comment 1

[Jorg K (CEST = GMT+2)]

Hmm, that would depend on your system locale. What have you got selected there? I have English-US and I can create such a folder and read e-mails in it. But since RÓŻNE is not in the windows-1252 code page, the folder name is totally encoded and ends up being 79357a77 on disk. What's your folder name on disk, in both the lowercase and uppercase version?

Comment 2

[Wayne Mery (:wsmwk)]

(In reply to Jorg K (GMT+1) from comment #1)

Hmm, that would depend on your system locale. What have you got selected there?
What's your folder name on disk, in both the lowercase and uppercase version?

Comment 3

[ne33us]

(bad use of English language ahead)
Same (ish) problem.
I also have folders named with GREEK characters (i.e "ΦΟΡΗΣ")

Εχample:
Today, Thunderbird showed up two identical folders named "ΦΟΡΗΣ"
None of them showed any messages in them.
Repair folder did nothing (no action).
The location of the folder is mailbox:///X:/somewhere/Inbox.sbd/a3a8ee9a/
in the Inbox.sbd, a new FOLDER (not file) appeared, named "a3a8ee9a" with the corresponding "a3a8ee9a.msf" file.
I was expecting "mailbox:///X:/somewhere/Inbox.sbd/ΦΟΡΗΣ/", since my original
and working, files for that folder still exist in Inbox.sbd, named "ΦΟΡΗΣ" and "ΦΟΡΗΣ.msf" respectively.

Temp solution:
Backed up everything to another folder.
Deleted unwanted new folders and files with gibberish names.
Renamed GREEK named files with English characters names (now "FORIS" and "FORIS.msf")
Folder "FORIS" appeared in Thunderbird with all my messages in it (good).

Testing on temp solution
Within Thunderbird, I renamed folder "FORIS" to "ΦΟΡΗΣ"
Folder shows as "ΦΟΡΗΣ", and still showing the messages.
BUT, in the filesystem, the files (not folders any more) are re-named as "a3a8ee9a" and "a3a8ee9a.msf" respectively.

Thunderbird seem that no longer wants to save files in any other language except English :) (just joking)

---

Name Thunderbird
Version 68.6.0
Build ID 20200310192757
Update History
Update Channel release
User Agent Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Thunderbird/68.6.0
OS Windows_NT 10.0

Internationalization & Localization
Application Settings
Requested Locales ["en-US"]
Available Locales ["en-US"]
App Locales ["en-US"]
Regional Preferences ["el-GR"]
Default Locale "en-US"
Operating System
System Locales ["en-US"]
Regional Preferences ["el-GR"]

On my work PC, Thunderbird saves folders/files with Greek names, just fine, with
Thunderbird Version 68.6.0 (32bit) Build ID 20200310192757 , Windows_NT 10.0
Internationalization & Localization-> Operating System-> System Locales ["el-GR"]

Comment 4

[Jorg K (CEST = GMT+2)]

I'll explain what is happening. Thunderbird stores all folder names internally using so-called Unicode, that's a character set which can store all characters from all languages of the world, from European languages, Slavonic languages, Hebrew, Greek, Chines, Japanese, Korean, Thai, etc.

Windows on the other hand uses a so-called ANSI code page which caters to local needs, there is windows-1252 for European languages, and many more windows-125x, for example windows-1253 is Greek. In the Windows "Region" control panel in the Administrative tab, you can select the ANSI codepage for the system locale. From Windows 10, you can also select Unicode (UTF-8). In that case all problems go away for good.

Let's now assume, you don't use Unicode as system locale, but instead "English US". Now when to want to create a folder ΦΟΡΗΣ, Thunderbird tries whether that name can be stored as a file/folder name. But Greek characters are not on the "English US" code page, so Thunderbird has to improvise and encode that string, as you saw to a3a8ee9a. In the code that happens here:
https://searchfox.org/comm-central/rev/25ea1af1ce1311e386eacd554299517e4cf84709/mailnews/base/util/nsMsgUtils.cpp#298
If the string is not "convertible to native", it is turned into a funny string of letters and numbers here:
https://searchfox.org/comm-central/rev/e3ce12478215dd7cb2986b771da6a45ea12a38d0/mailnews/base/util/nsMsgUtils.cpp#387

So let me summarise what works:

• Use Unicode as system locale and you will never have a problem.
• Use a system locale that matches the language you want to use, in that case folder names in Thunderbird and on disk will match.
• If you use a system locate that doesn't match your language, be prepared to accept "funny" folder names on disk. Thunderbird will still work.

Here is what doesn't work:

• Change the system locale (mostly for games) or use programs that do so.
• Rename folders on disk manually.

The report of the Greek user seeing doubled-up folders is not interesting, it gets reported periodically, last time for example in bug 1582925. That user wanted to play Japanese games and changed the system locale and then forgot about it :-(

What interests me is the original report from the Polish user who reported problems with a folder called RÓŻNE. As per comment #1, and as described above, I get a "funny" name instead, but Thunderbird still works.

Comment 5

[ne33us]

In my case the problem was that "Language for non-Unicode programs" was set to "English (US)", instead of "Greek (Greece)".
My Thunderbird is working as expected, again :)

Jorg K, thanks for your time and for pointing to the right direction.