268185 - Folder contents lost on upgrade from 0.8 to 0.9 if folder name contains non-ASCII characters

Status: RESOLVED FIXED

(Thunderbird :: General, defect)

Description

[Dave Miller [:justdave]]

This bug report contains non-ascii characters.  My character coding is set to
UTF-8, so make sure you're using UTF-8 to view this page.

I've been using the release version of Thunderbird 0.8 (Mac OS X).  I just
upgraded this afternoon to 0.9.  I have a folder titled "•Other Mailing Lists"
which has the • character in front in order to make it alphabetize to the end of
the list.  Upon upgrading to Thunderbird 0.9, the folder is now listed TWICE in
my folder list (it is displayed correctly, no garbage in the name in place of
the non-ascii characters), and both copies are empty.

Examining the filesystem shows the following files/folders present:

-rw-r--r--   1 dave  admin         0  6 Nov 19:10 "Other Mailing Lists
-rw-r--r--   1 dave  admin      1352  6 Nov 19:23 "Other Mailing Lists.msf
-rw-------   1 dave  admin         0 31 Mar  2004 •Other Mailing Lists
-rw-r--r--   1 dave  admin      1526  8 Sep 19:47 •Other Mailing Lists.msf
drwxr-xr-x   6 dave  admin       204 20 Oct 09:15 •Other Mailing Lists.sbd/

The problem was first discovered when an incoming message came in which would
have sorted to one of the subfolders came in, and it showed up in my inbox
instead of filtering.  A quick trip to the filter manager showed the filter had
been disabled (what happened to the alert box saying that a filter applies to
that message but points at a nonexistant folder, btw?).

Comment 1

[Jungshik Shin]

Do you also have this problem with a *new* folder (created after the upgrade)
with non-ASCII characters? If so, bug 264071 has to be reopened for TB 0.9. 
 

Comment 2

[Dave Miller [:justdave]]

(In reply to comment #1)
> Do you also have this problem with a *new* folder (created after the upgrade)
> with non-ASCII characters? If so, bug 264071 has to be reopened for TB 0.9. 

Creating a new folder "seems" to work, but it creates both copies of it on the
filesystem:

-rw-r--r--   1 dave  admin         0  7 Nov 00:58 "Yet Another Folder
-rw-r--r--   1 dave  admin      1257  7 Nov 00:58 "Yet Another Folder.msf
drwxr-xr-x   4 dave  admin       136  7 Nov 00:59 "Yet Another Folder.sbd
-rw-------   1 dave  admin         0  7 Nov 00:58 •Yet Another Folder

So I get two mbox files, one with the correct coding, one with the • turned into
a ", and an index file and subdirectory folder with the " instead of •

Comment 3

[Dave Miller [:justdave]]

(In reply to comment #2)
> So I get two mbox files, one with the correct coding, one with the • turned into
> a ", and an index file and subdirectory folder with the " instead of •

OK, and after restarting Thunderbird, it's now listed twice in the folder list,
and both of them have identical contents.

Comment 4

[WADA:World Anti-bad-Duping Agency]

Next two bugs seems to be same problem as this bug.
 Bug 267660 : sub folders of the inbox unavailable and corrupted after upgrade
from 0.8 to 0.9
 Bug 267690 : Foulders with double bytes name don't show their sub-folders

Comment 5

[Jungshik Shin]

I just lost my remote connection to a Mac (I was using a vncviewer) after a
reboot  and won't get to it until tomorrow. Anyway, Wada, do you have any of
these problems  (this, bug 267660, and bug 267690) with TB 0.9 on Windows?
Unless you change the system locale to Western European, you can't test with '•'
(it's U+2022 and is not covered by Shift_JIS while it's covered by
Windows-1252). However, you don't have to test it with U+2022. Just trying to
see if any of three problems (with Japanese characters) are reproducible on
Windows would help. 

'•' is U+2022 and '"' is U+0022 so that somewhere
AssignWithConversion/ToNewCString/LossyCopyUTF16toASCII must be used where it
shouldn't. 
 

Comment 6

[Jungshik Shin]

(In reply to comment #5)

> '•' is U+2022 and '"' is U+0022 so that somewhere
> AssignWithConversion/ToNewCString/LossyCopyUTF16toASCII must be used where it
> shouldn't. 

Oh, my gosh. NS_CopyNativeToUnicode/NS_CopyUnicodeToNative on Mac OS X is
broken. See bug 268219

Comment 7

[WADA:World Anti-bad-Duping Agency]

No problem for Japanese chars on MS Win(both TB 0.9 and Mozilla Suite) as I
described in Bug 264071.
This is true even on HEXA-string/HEXA-string.msf file set for folder of 'ソ'
(0x835C in Shift_JIS) which was created before Bug 264071.
These files could be used with no problem as folder of 'ソ' even after fix for
Bug 264071 since folder name is saved in ".msf",
even though file set of 'ソ' / 'ソ.msf' is used for newly created folder of 'ソ
'after fix for Bug 264071.

I tested U+2022 case on Japanese MS Win, just out of curiosity.
This bug seems to be similar situation to Case-2 in my test.
If former Moz/TB used U+2022 for folder file name, and if new Moz/TB won't use
U+2022 for folder file name, similar situation may occur.
# I know this can't occur on Mozilla/TB, Japanese MS Win.
# Jshin, you need't to worry about it ;-)

[Test result on with Thunderbird version 0.6+ (20041103) on Win-2K]
(Case-1) Create mail folder by Thunderbird
 (1-1) Folder name of '"'(U+0022) => 6f9712b5 / 6f9712b5.msf
       Normal on Win, becuase double quote is real illegal file name character
       when MS Win
 (1-2) Folder name of '•' (U+2022) => _ / _.msf
       '_' is used on Win. (Probably 0x5F, Underscore)
(Case-2) When files is created manually on Windows Explerer
 (2-1) Create a file. File name = '•' (U+2022)
 (2-2) Restart Thunderbird
       => Mail folder of '?' appeared  (0x3F probably)
       => bd0bf74a / bd0bf74a.msf were created by Thunderbird
 (2-3) Shutdown and Restart Thunderbird
       Two mail folders named '?' appeared (0x3F probably)
       "Copy Folder Location" displays bd0bf74a for both '?' folder
        - First  '?' folder : mailbox:/C|/...../bd0bf74a
        - Second '?' folder : mailbox:/C|/...../bd0bf74a
 (2-4) Delete first '?' folder
       => bd0bf74a / bd0bf74a.msf were moved to Trash.sbd
       => File of '•' remains in mail directry
 (2-5) Empty Trash
 (2-6) Delete '?' folder(remainder, second '?' folder at step 2-3)
       => Confirmation of delete is displayed but nothing happened
 (2-7) Shutdown and Restart Thunderbird
       => Same as step (2-2)
 (2-8) Shutdown and Restart Thunderbird
       => Same as step (2-3)
(End of Test)

Comment 8

[Simon Montagu :smontagu]

Bug 260854 might also be related

Comment 9

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #5)
> '•' is U+2022 and '"' is U+0022 so that somewhere
> AssignWithConversion/ToNewCString/LossyCopyUTF16toASCII must be used where it
> shouldn't. 

Jshin, does it mean Bug 264071(correction of fix for bug 264467/bug 257986) is
incomplete?

Comment 10

[Jungshik Shin]

No, see comment #6. This bug depends on bug 268219

Comment 11

[Scott MacGregor]

This  has now been fixed on the aviary 1.0 branch by jshin's patch for Bug #268219

Comment 12

[WADA:World Anti-bad-Duping Agency]

Following test result has been reported to Bugzilla Japan.

Tested with both Mozilla-nightly (2004-11-08-06-trunk) and  Thunderbird-nightly
(2004-11-10-02-0.9) on Japanese mac OS X.
Test were done on next problematic characters.
 (1) Folder name of 'ソ'
     (0x5C in 2nd byte of Shift_JIS)
     'ソ' and 'ソ.msf' was created
     (Before fix, folder creation failed) 
 (2) Folder name of 'がぎぐげご'
     (NFC format characters, NFD format is also defined for these characers)
     'がぎぐげご' and 'がぎぐげご.msf' was created
     (Before fix, same symptom as U+0022 case in comment #2 occured)
Folders were successfuly created/moved/deleted.

Although migration is not tested yet, I think fix has been verified, because
cause of migration failure was invalid folder file creation due to unexpected
file name character use.

Comment 13

[Frankie]

*** Bug 271009 has been marked as a duplicate of this bug. ***