1786191 - Folders with half-width Kana names are corrupted when updating to Thunderbird 102

Status: RESOLVED FIXED

(Thunderbird :: Folder and Message Lists, defect)

Description

[EarlgreyTea]

Attachment: half-width_kana_folder_are_corrupted.jpg

User Agent:
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.1.2
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Thunderbird/104.0

Steps to reproduce:

1. Start Thunderbird 91.
2. Create folders with names containing half-width kana characters in Thunderbird.
   For example, "半角ｶﾅ", "迷惑ﾒｰﾙ", "ﾌﾟﾛﾊﾞｲﾀﾞ" (See upper left of attached image)
3. Put some messages in that folder and make sure they show up in the message list.
4. Start Thunderbird 102 with the same profile.
5. Exit Thunderbird 102 and start it again.

Actual results:

On the Thunderbird, the folders with the half-width Kana names are in the following situation. (See bottom left of attached image)

• The same folder appears twice in the folder pane.
• Message lists in those folders are empty.

The local directory where the messages are stored has the following situation. (See bottom right of attached image)

• Original folder file (mbox) and index file (.msf) are unchanged.
• Index (.msf) files are created with file names that have corruped half-width kana parts.
• Empty directories with the same names as the index files are created.

Expected results:

There is no change in the folders named in half-width katakana, and the saved messages are displayed.

Additional Information:

• This bug has been reported to the MozillaZine.jp forum and I have reproduced and confirmed it.
• This bug is a regression by bug 1746052 and is fixed by bug 1778429.
• I have confirmed that it is fixed in Thunderbird Daily 105.0a1 (2022-08-19). But the beta and release versions are not fixed.

Comment 1

[Wayne Mery (:wsmwk)]

If https://bugzilla.mozilla.org/show_bug.cgi?id=1778429#c27 is correct, then the patch should be picked up by Thunderbird 102.2.0. You can try the candidate from https://archive.mozilla.org/pub/thunderbird/candidates/102.2.0-candidates/build2/

Comment 2

[EarlgreyTea]

(In reply to Wayne Mery (:wsmwk) from comment #1)

If https://bugzilla.mozilla.org/show_bug.cgi?id=1778429#c27 is correct, then the patch should be picked up by Thunderbird 102.2.0. You can try the candidate from https://archive.mozilla.org/pub/thunderbird/candidates/102.2.0-candidates/build2/

I tried 102.2.0-candidates/build2/ but it doesn't seem to fix.

Comment 3

[EarlgreyTea]

Just in case, when I checked the build fixed by mozregression, it was the following.

pushlog_url: https://hg.mozilla.org/comm-central/pushloghtml?fromchange=e17f4d5f301989c3780f31e11dfd82eec67f63a9&tochange=609f81cdf45703a328280c152e9bd17c1c3f0c0a

Comment 4

[Magnus Melin [:mkmelin]]

I guess https://hg.mozilla.org/mozilla-central/rev/e0b4c0e343efad5a3f447e33e128aae12b078d9d (bug 1778429) is key to understanding what would need to change, and why likely the rework in bug 1782347 fixed it for trunk

For release/beta, what a mess. This looks like some of it:
https://searchfox.org/comm-esr102/search?q=.ReplaceChar%28&path=&case=false&regexp=false

Comment 5

[Magnus Melin [:mkmelin]]

Or maybe it's easy enough to fix.
https://searchfox.org/comm-esr102/source/mailnews/base/src/nsMessenger.cpp#123

- aResult.ReplaceChar(FILE_PATH_SEPARATOR FILE_ILLEGAL_CHARACTERS, '-');
+ aResult.ReplaceChar(u"" FILE_PATH_SEPARATOR FILE_ILLEGAL_CHARACTERS, '-');

Comment 6

[Magnus Melin [:mkmelin]]

And check the other FILE_PATH_SEPARATOR FILE_ILLEGAL_CHARACTERS usages.

Comment 7

[Magnus Melin [:mkmelin]]

Attachment: Bug 1786191 - [esr102 only] Use utf-16 character literals when calling ReplaceChar. r=benc

If paasing single-byte string to be compared using a utf16 string it won't work, breaks e.g. for half-width katakana.

Comment 8

[Magnus Melin [:mkmelin]]

Doesn't reproduce on linux since I think "" needs to be in there... and it's obviously not.
I think the fix should do it though. Set off a try run here: https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=2e69ffe5273eb4085c7c78aa51d71b71f6c37244

Comment 9

[Magnus Melin [:mkmelin]]

Windows test build can be downloaded from https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/YN-3VwjBRtG-qW0yPmYo-A/runs/0/artifacts/public/build/setup.exe

Comment 10

[EarlgreyTea]

(In reply to Magnus Melin [:mkmelin] from comment #9)

Windows test build can be downloaded from https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/YN-3VwjBRtG-qW0yPmYo-A/runs/0/artifacts/public/build/setup.exe

I tried downloading the file from that URL, but the size of the downloaded file (setup.exe) is only 838 KB.

Comment 11

[Magnus Melin [:mkmelin]]

Sorry, should be https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/YN-3VwjBRtG-qW0yPmYo-A/runs/0/artifacts/public/build/install/sea/target.installer.exe

Comment 12

[EarlgreyTea]

(In reply to Magnus Melin [:mkmelin] from comment #11)

Sorry, should be https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/YN-3VwjBRtG-qW0yPmYo-A/runs/0/artifacts/public/build/install/sea/target.installer.exe

I confirmed that the half-width kana name folders are not corrupted.

Comment 13

[Magnus Melin [:mkmelin]]

Great, thanks for testing!
I could not reproduce the bug on Windows either (with any version). Perhaps the windows filesystem has to be something non-western for it to trigger... My folder names were always fully hashed.

Comment 14

[Magnus Melin [:mkmelin]]

Comment on attachment 9290842 [details]
Bug 1786191 - [esr102 only] Use utf-16 character literals when calling ReplaceChar. r=benc

[Approval Request Comment]
Regression caused by (bug #): bug 1746052
User impact if declined: may get folder duplication and other related bad issues related to folders
Testing completed (on c-c, etc.): c-c got fixed through other refactoring, so this particular fix only applies to 102 and can be tested only there.
Risk to taking this patch (and alternatives if risky): in theory safe

Comment 15

[Wayne Mery (:wsmwk)]

Comment on attachment 9290842 [details]
Bug 1786191 - [esr102 only] Use utf-16 character literals when calling ReplaceChar. r=benc

[Triage Comment]
Approved for esr102

Comment 16

[Rob Lemley [:rjl]]

Thunderbird 102.2.1:
https://hg.mozilla.org/releases/comm-esr102/rev/edac1d393f54

Comment 17

[Rob Lemley [:rjl]]

Thunderbird 102.2.1:
https://hg.mozilla.org/releases/comm-esr102/rev/c0f48e08688a