1267973 - Folder names with turkish characters are not recognized in Windows 10

Status: UNCONFIRMED

(Thunderbird :: Folder and Message Lists, defect)

Description

[modabasi]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0
Build ID: 20160407164938

Steps to reproduce:

Moved my working profile folders from Windows 7 to Windows 10, both running Mozilla Thunderbird v.45. To reproduce you can simply create folders with Turkish characters (i.e. İ,Ş,ş).


Actual results:

Thunderbird did not recognize the folders with Turkish characters (i.e. İ,Ş,ş) and showed them blank. Had to rename folders using non Turkish characters to solve the issue.


Expected results:

Thunderbird should have recognized the folders with Turkish characters.

Comment 1

[Richard Marti (:Paenglab)]

Attachment: turkishFolder.png

No problem here to create a folder with your turkish characters on Win 10.

Have you tried in in safe mode (https://support.mozilla.org/kb/safe-mode) to create such a folder?

Comment 2

[modabasi]

Richard,

I can confirm that creating folders with Turkish characters work as expected. Even without the safe mode.

I guess reproducing this bug will be a little harder, the folders with Turkish characters were previously created in Windows 7 and after migrating to Windows 10, I've moved these folders from my profile backup. I had many folders and folders without Turkish characters were OK, folders with Turkish characters looked empty. (And they were doubled, I mean I had only one folder named "İsmail" and I was seeing 2 empty folders with the same name in the folder tree).

Folder repair through Properties > Repair Folder did not work, so I've tried deleting files foldername and foldername.msf and copying them back from by backups, which again failed. Then I've noticed that all the problematic folders had Turkish characters in them (not all Turkish characters cause this issue btw, like "ö" is OK but "İ" is not). I've renamed the foldername and foldername.msf files without Turkish characters and my mails returned back.

Comment 3

[Richard Marti (:Paenglab)]

(In reply to modabasi from comment #2)
> Richard,
> 
> I can confirm that creating folders with Turkish characters work as
> expected. Even without the safe mode.
> 
> I guess reproducing this bug will be a little harder, the folders with
> Turkish characters were previously created in Windows 7 and after migrating
> to Windows 10, I've moved these folders from my profile backup. I had many
> folders and folders without Turkish characters were OK, folders with Turkish
> characters looked empty. (And they were doubled, I mean I had only one
> folder named "İsmail" and I was seeing 2 empty folders with the same name in
> the folder tree).

Like this it's impossible for me to reproduce. Maybe it's a error of your backup tool which doesn't support some of your characters. What do you use for the backup?

> I've renamed the foldername and foldername.msf files without Turkish
> characters and my mails returned back.

And when you rename the folder again with turkish characters on TB itself?

Comment 4

[Richard Marti (:Paenglab)]

Ace, my folder's name from screenshot in my file system is now 05c9fe17 and 05c9fe17.msf. Was there some change to protect the file system to not use invalid characters?

Comment 5

[modabasi]

(In reply to Richard Marti (:Paenglab) from comment #3)
> Like this it's impossible for me to reproduce. Maybe it's a error of your
> backup tool which doesn't support some of your characters. What do you use
> for the backup?

The profile folder was copied to an external hard drive with Windows' file manager. No backup software.

> And when you rename the folder again with turkish characters on TB itself?

I've just tried that on Windows 10, after renaming "Ismail" and "Ismail.msf" files to "İsmail" and "İsmail.msf", I see empty folders in Thunderbird frontend again.

Comment 6

[Richard Marti (:Paenglab)]

(In reply to modabasi from comment #5)
> (In reply to Richard Marti (:Paenglab) from comment #3)
> > And when you rename the folder again with turkish characters on TB itself?
> 
> I've just tried that on Windows 10, after renaming "Ismail" and "Ismail.msf"
> files to "İsmail" and "İsmail.msf", I see empty folders in Thunderbird
> frontend again.

Renaming the folder in Explorer itself shows also a empty folder. Rename it in TB.
And let's wait what aceman replies.

Comment 7

[modabasi]

(In reply to Richard Marti (:Paenglab) from comment #6)

> Renaming the folder in Explorer itself shows also a empty folder. Rename it
> in TB.
> And let's wait what aceman replies.

Renaming in TB works fine except that the file names are changed to some random stuff as you've already reported.

Comment 8

[:aceman]

(In reply to Richard Marti (:Paenglab) from comment #4)
> Ace, my folder's name from screenshot in my file system is now 05c9fe17 and
> 05c9fe17.msf. Was there some change to protect the file system to not use
> invalid characters?

Yes, I think there is a protection and we intentionally escape special characters and produce the numbers in the name on the filesystem.

(In reply to modabasi from comment #7)
> (In reply to Richard Marti (:Paenglab) from comment #6)
> 
> > Renaming the folder in Explorer itself shows also a empty folder. Rename it
> > in TB.
> > And let's wait what aceman replies.
> 
> Renaming in TB works fine except that the file names are changed to some
> random stuff as you've already reported.

Yes, I think that is the intended behaviour. What is wrong with it?

Did you have the İsmail folder written like this on the filesystem (visible via Windows explorer)?

For me the İsmail folder created inside TB was also stored like İsmail on the filesystem, but that is on Linux, UTF8.

Comment 9

[:aceman]

Paenglab, you can also try creating a long folder name in TB (like 50 characters) and it will be truncated and padded with the numbers on the filesystem (at least for me).

Comment 10

[modabasi]

(In reply to :aceman from comment #8)
> > Renaming in TB works fine except that the file names are changed to some
> > random stuff as you've already reported.
> 
> Yes, I think that is the intended behaviour. What is wrong with it?

It's not a big deal as long as you're creating these folders on the same operating system, one can live with it, although Turkish users are not going to be very happy seeing their file names changing to some obscure stuff. The bigger problem is, if you're moving your folder from, say Windows 7 to Windows 10 like me, your folders with Turkish characters which work just fine on Windows 7 suddenly turn out to be empty. Help pages on support.mozilla.org do not even mention this kind of behaviour and I could not find a single page telling me how to fix this issue. I've been using Thunderbird for a long time and long ago I've done this migration from Windows XP to Windows 7 without any problems. Why did this behaviour change with Windows 10?

And a word about this "special characters" thing...

Turkish characters should not be so "special" anymore, we've started using these characters in 1928. This is a country with over 70 million people living on it and millions of others abroad. I think this really needs a fix because we are a huge community, Turkish users are naming their folders with Turkish characters, like "İsmail" which is a friend's name, and many are switching from Windows 7 to 10 like me every day. I'm a coder and I could find the fix myself, think about the avereage (and below average) computer user.

Comment 11

[:aceman]

It has nothing to do with Turkey itself :) It is probably more of a legacy problem that Windows has/had problems with non-ascii characters in older versions OR some of its filesystems (like FAT). So TB has special code to not save special characters to filesystem when you create the folder inside TB. It just stores the proper folder name inside the folder.msf file and does not show the on-disk escaped name.

On the other hand, if you create a folder on the filesystem directly with special characters, TB should pick it up as is and work with it. It does for me on Linux.

I would understand if there was a change in behaviour between Win98 and Win 7. But why would it suddenly not work in Win 10 is strange.

Paenglab, so you have also seen that it does not work if you create the special character names directly in explorer? If you copy your Inbox to İsmail, then it appears empty?

Comment 12

[modabasi]

I've tried it again, created a folder named "Deneme" in TB, copied some mail into it and renamed the folder to "Şeneme" in Windows Explorer. TB created files 86ef9916 and 86ef9916.msf, I can still see a folder named "Şeneme" in TB but it's empty.

Comment 13

[:aceman]

I have now tested this on Windows XP on NTFS. I renamed an existing folder (with no special chars) to Şeneme in Explorer and then started TB. The folder was picked up with that name and was shown normally, also containing messages.

What is your filesystem type?

Comment 14

[modabasi]

(In reply to :aceman from comment #13)
> What is your filesystem type?

NTFS

Comment 15

[Richard Marti (:Paenglab)]

Tried it now under XP, Win 7 and Win 10. When I create the folder İŞtestş in TB on all systems the escaped name is used. When I rename a folder from test to İŞtestş in filesystem TB creates in filesystem a folder and a msf file with the escaped name. The original files are untouched.

All is NTFS

Comment 16

[Wayne Mery (:wsmwk)]

Does this still reproduce?