1445853 - Cannot access emails in local folder which includes Japanese character in the folder name

Status: RESOLVED FIXED

(MailNews Core :: Backend, defect)

Description

[Takanori MATSUURA]

In case the folder below exists, Thunderbird can access to the folder and cannot access to the email on Windows.

Local Folder
  +- あ (Folder)
      +- test mail (Email)


When I click the email, Thunderbird returns the URL below is not found.
mailbox:///C:/Users/%82%A0/AppData/Roaming/Thunderbird/Profiles/xxxxxxxx.x/Mail/Local%20Folders/%82%A0?number=2
("%82%A0" means "あ" in Shift_JIS.)


Environment:
Thunderbird 60.0a (20180312030201) win32
OS: Windows 10, x64
Folder Type: Maildir

Additional Info:
In IMAP, the folder "あ" is generated as "&MEI-" at local Windows storage.
However, in Local Folder, it is generated as "あ" at local Windows storage.

Comment 1

[Jorg K (CEST = GMT+2)]

Interesting problem.

IMAP folders are encoded in MUTF-7 which uses pure ASCII, hence the &MEI-.

On Windows with default locale "en", I get a local folder of "e85eae05" which is hash encoded, see recent discussion in bug 1440278 comment #14 and bug 1440278 comment #19.

That bug should have fixed the issue in TB 60. Now you're saying it doesn't work :-( - Did it work previously in earlier versions? Did we break it?

I don't have a Japanese system to investigate. Masatoshi-san, could you please take a look.

Comment 2

[Masatoshi Kimura [:emk]]

Attachment: Work around of mailbox:// URL non-supporting UTF-8 paths

Comment 3

[Masatoshi Kimura [:emk]]

Hmm, apparently my theory was wrong. Please try this patch (I couldn't build the tree due to nsIURL issues.)

Comment 4

[Jorg K (CEST = GMT+2)]

Thanks for the patch.

C-C was building last night with M-C at rev c488b8d0e074, looks like Dailies with M-C at 97160a734959 (tip at time of writing) aren't busted either:
https://treeherder.mozilla.org/#/jobs?repo=comm-central&revision=b7d028d0299cd9456bb2be2f5702d4138aa5abf3

I can do I try build and get you and Takanori-san to try it out.

Comment 5

[Jorg K (CEST = GMT+2)]

https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=03ba8e234f1a47e79394472e76419dd088e2829a

Comment 6

[Jorg K (CEST = GMT+2)]

Sorry, I forgot, Win32 on buildbot is busted, here's a TaskCluster build:
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=7f572e667c5eb32318e82db7e4b452f06a908ab8
I'll post links to the binaries when done.

Comment 7

[Jorg K (CEST = GMT+2)]

The executables with this patch are now available:
Win 32bit:
https://queue.taskcluster.net/v1/task/DMzXma6eS3i6Ffho4zmzqg/runs/0/artifacts/public/build/target.zip
https://queue.taskcluster.net/v1/task/DMzXma6eS3i6Ffho4zmzqg/runs/0/artifacts/public/build/setup.exe

Win 64bit:
https://queue.taskcluster.net/v1/task/VsceEfkrReWAEcjREc30gQ/runs/0/artifacts/public/build/target.zip
https://queue.taskcluster.net/v1/task/VsceEfkrReWAEcjREc30gQ/runs/0/artifacts/public/build/setup.exe

Takanori-san, can you please try this build.

I tried to reproduce the problem. I switched my system locale to Japanese and created a folder called あ. I checked in the file system, that folder is present with that name and also あ.msf.

I don't have a problem accessing messages in the folder with TB 61 Daily, if I view the source I see %82%A0 in the message URL.

The problem was reported for Maildir, so I'll try this next.

Comment 8

[Jorg K (CEST = GMT+2)]

I created a new profile with Local Folder with maildir storage and can't see the problem there either.

Comment 9

[Wayne Mery (:wsmwk)]

This is a regression of bug 1440278?

Comment 10

[Jorg K (CEST = GMT+2)]

Yes, it would be.

Switching the system locale back to English makes those folder inaccessible, the mbox as well as the maildir one.

Comment 11

[Joe Sabash [:JoeS1]]

FWIW I ran the try build with an existing profile with no ill effects

Comment 12

[Takanori MATSUURA]

I tested win32 binary on win64.
This works good for me.

(In reply to Jorg K (GMT+1) from comment #7)
> The executables with this patch are now available:
> Win 32bit:
> https://queue.taskcluster.net/v1/task/DMzXma6eS3i6Ffho4zmzqg/runs/0/
> artifacts/public/build/target.zip
> https://queue.taskcluster.net/v1/task/DMzXma6eS3i6Ffho4zmzqg/runs/0/
> artifacts/public/build/setup.exe
> 
> Win 64bit:
> https://queue.taskcluster.net/v1/task/VsceEfkrReWAEcjREc30gQ/runs/0/
> artifacts/public/build/target.zip
> https://queue.taskcluster.net/v1/task/VsceEfkrReWAEcjREc30gQ/runs/0/
> artifacts/public/build/setup.exe
> 
> Takanori-san, can you please try this build.
> 
> I tried to reproduce the problem. I switched my system locale to Japanese
> and created a folder called あ. I checked in the file system, that folder is
> present with that name and also あ.msf.
> 
> I don't have a problem accessing messages in the folder with TB 61 Daily, if
> I view the source I see %82%A0 in the message URL.
> 
> The problem was reported for Maildir, so I'll try this next.

Comment 13

[Jorg K (CEST = GMT+2)]

Thanks for the confirmation. Could anyone tell me how to reproduce the problem locally?

Comment 14

[Jorg K (CEST = GMT+2)]

Comment on attachment 8959783 [details] [diff] [review]
Work around of mailbox:// URL non-supporting UTF-8 paths

I can't test this, but we have positive confirmation that it works. It also "reverts" these changes from bug 1440278:
https://hg.mozilla.org/comm-central/rev/1580fa402264#l26.14
https://hg.mozilla.org/comm-central/rev/1580fa402264#l27.14

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/a1baa186817e
Work-around for mailbox:// URL not supporting UTF-8 paths. r=jorgk

Comment 16

[Jorg K (CEST = GMT+2)]

Beta (TB 60):
https://hg.mozilla.org/releases/comm-beta/rev/43daf17883d1e37688af61ed84f0a1a87ad85324