255541 - Trash folder name with unicode chars not interpretated correctly

Status: RESOLVED WORKSFORME

(MailNews Core :: Networking: IMAP, defect)

Description

[punkadiddle]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8a3) Gecko/20040812
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8a3) Gecko/20040812

I have an IMAP account with a trash folder named "Gelöscht". For quite some time
I used the following preference setting to make Mozilla recognize this folder as
trash folder:
user_pref("mail.server.server1.trash_folder_name", "Gel&APY-scht");

I got the unicode string by setting the folder as Draft folder in the GUI and
then copied it the name to the trash_folder_name property in the prefs.js file.

Since the latest Mozilla update there is a problem with the interpretation of
the unicode folder name "Gel&APY-scht". This results in Mozilla showing two
folders insted of one which are "Gelöscht" and "Gel&APY-scht". The latter is
used as trash folder.

I think this is either a bug in the IMAP / folder functions or a change which
makes it neccessary to change the preference to something different.

Does anyone have the same problem? What's the bug?

Reproducible: Always
Steps to Reproduce:
1. Create a folder named "Gelöscht" in your IMAP account
2. Set it as trash folder using a preference like
user_pref("mail.server.server1.trash_folder_name", "Gel&APY-scht");
3. Run Mozilla Mail

Actual Results:  
As you can see in the IMAP log following the server name is reported correctly
as above by the IMAP server. Further down it is then referenced as
"Gel&-APY-scht", not found and created.

0[364540]: 2015210:imap.gmx.net:NA:SetupWithUrl: clearing IMAP_CONNECTION_IS_OPEN
2360[1fcfc80]: ImapThreadMainLoop entering [this=2015210]
2360[1fcfc80]: 2015210:imap.gmx.net:NA:ProcessCurrentURL: entering
2360[1fcfc80]:
2015210:imap.gmx.net:NA:ProcessCurrentURL:imap://[e-mail]:143/select%3E/INBOX:
 = currentUrl
2360[1fcfc80]: ReadNextLine [stream=2015998 nb=35 needmore=0]
2360[1fcfc80]: 2015210:imap.gmx.net:NA:CreateNewLineFromSocket: * OK GMX IMAP4
StreamProxy ready.

2360[1fcfc80]: 2015210:imap.gmx.net:NA:SendData: 1 capability

2360[1fcfc80]: ReadNextLine [stream=2015998 nb=60 needmore=0]
2360[1fcfc80]: 2015210:imap.gmx.net:NA:CreateNewLineFromSocket: * CAPABILITY
IMAP4 IMAP4rev1 IDLE AUTH=LOGIN AUTH=CRAM-MD5

2360[1fcfc80]: ReadNextLine [stream=2015998 nb=27 needmore=0]
2360[1fcfc80]: 2015210:imap.gmx.net:NA:CreateNewLineFromSocket: 1 OK CAPABILITY
completed

2360[1fcfc80]: 2015210:imap.gmx.net:NA:SendData: 2 authenticate login

2360[1fcfc80]: ReadNextLine [stream=2015998 nb=16 needmore=0]
2360[1fcfc80]: 2015210:imap.gmx.net:NA:CreateNewLineFromSocket: 

2360[1fcfc80]: 2015210:imap.gmx.net:NA:SendData: Logging suppressed for this
command (it probably contained authentication information)
2360[1fcfc80]: ReadNextLine [stream=2015998 nb=16 needmore=0]
2360[1fcfc80]: 2015210:imap.gmx.net:NA:CreateNewLineFromSocket: 

2360[1fcfc80]: 2015210:imap.gmx.net:NA:SendData: Logging suppressed for this
command (it probably contained authentication information)
2360[1fcfc80]: ReadNextLine [stream=2015998 nb=22 needmore=0]
2360[1fcfc80]: 2015210:imap.gmx.net:NA:CreateNewLineFromSocket: 2 OK LOGIN completed

2360[1fcfc80]: 2015210:imap.gmx.net:A:SendData: 3 lsub "" "*"

2360[1fcfc80]: ReadNextLine [stream=2015998 nb=24 needmore=0]
2360[1fcfc80]: 2015210:imap.gmx.net:A:CreateNewLineFromSocket: * LSUB () "/"
"Ablage"

2360[1fcfc80]: ReadNextLine [stream=2015998 nb=27 needmore=0]
2360[1fcfc80]: 2015210:imap.gmx.net:A:CreateNewLineFromSocket: * LSUB () "/"
"Aktuelles"

2360[1fcfc80]: ReadNextLine [stream=2015998 nb=30 needmore=0]
2360[1fcfc80]: 2015210:imap.gmx.net:A:CreateNewLineFromSocket: * LSUB () "/"
"Entw&APw-rfe"

2360[1fcfc80]: ReadNextLine [stream=2015998 nb=30 needmore=0]
2360[1fcfc80]: 2015210:imap.gmx.net:A:CreateNewLineFromSocket: * LSUB () "/"
"Gel&APY-scht"

2360[1fcfc80]: ReadNextLine [stream=2015998 nb=27 needmore=0]
2360[1fcfc80]: 2015210:imap.gmx.net:A:CreateNewLineFromSocket: * LSUB () "/"
"Geschaeft"

2360[1fcfc80]: ReadNextLine [stream=2015998 nb=26 needmore=0]
2360[1fcfc80]: 2015210:imap.gmx.net:A:CreateNewLineFromSocket: * LSUB () "/"
"Gesendet"

2360[1fcfc80]: ReadNextLine [stream=2015998 nb=24 needmore=0]
2360[1fcfc80]: 2015210:imap.gmx.net:A:CreateNewLineFromSocket: * LSUB () "/"
"Handel"

2360[1fcfc80]: ReadNextLine [stream=2015998 nb=23 needmore=0]
2360[1fcfc80]: 2015210:imap.gmx.net:A:CreateNewLineFromSocket: * LSUB () "/" "INBOX"

2360[1fcfc80]: ReadNextLine [stream=2015998 nb=29 needmore=0]
2360[1fcfc80]: 2015210:imap.gmx.net:A:CreateNewLineFromSocket: * LSUB () "/"
"Junk-e-mail"

2360[1fcfc80]: ReadNextLine [stream=2015998 nb=24 needmore=0]
2360[1fcfc80]: 2015210:imap.gmx.net:A:CreateNewLineFromSocket: * LSUB () "/"
"Privat"

2360[1fcfc80]: ReadNextLine [stream=2015998 nb=30 needmore=0]
2360[1fcfc80]: 2015210:imap.gmx.net:A:CreateNewLineFromSocket: * LSUB () "/"
"Spamverdacht"

2360[1fcfc80]: ReadNextLine [stream=2015998 nb=25 needmore=0]
2360[1fcfc80]: 2015210:imap.gmx.net:A:CreateNewLineFromSocket: * LSUB () "/"
"Studium"

2360[1fcfc80]: ReadNextLine [stream=2015998 nb=21 needmore=0]
2360[1fcfc80]: 2015210:imap.gmx.net:A:CreateNewLineFromSocket: * LSUB () "/" "Thw"

2360[1fcfc80]: ReadNextLine [stream=2015998 nb=23 needmore=0]
2360[1fcfc80]: 2015210:imap.gmx.net:A:CreateNewLineFromSocket: * LSUB () "/" "Trash"

2360[1fcfc80]: ReadNextLine [stream=2015998 nb=26 needmore=0]
2360[1fcfc80]: 2015210:imap.gmx.net:A:CreateNewLineFromSocket: * LSUB () "/"
"Vorlagen"

2360[1fcfc80]: ReadNextLine [stream=2015998 nb=22 needmore=0]
2360[1fcfc80]: 2015210:imap.gmx.net:A:CreateNewLineFromSocket: 3 OK LSUB completed 

2360[1fcfc80]: 2015210:imap.gmx.net:A:SendData: 4 list "" "INBOX"

2360[1fcfc80]: ReadNextLine [stream=2015998 nb=35 needmore=0]
2360[1fcfc80]: 2015210:imap.gmx.net:A:CreateNewLineFromSocket: * LIST
(\Noinferiors) "/" "INBOX"

2360[1fcfc80]: ReadNextLine [stream=2015998 nb=21 needmore=0]
2360[1fcfc80]: 2015210:imap.gmx.net:A:CreateNewLineFromSocket: 4 OK LIST completed

2360[1fcfc80]: 2015210:imap.gmx.net:A:SendData: 5 list "" "Gel&-APY-scht"

2360[1fcfc80]: ReadNextLine [stream=2015998 nb=21 needmore=0]
2360[1fcfc80]: 2015210:imap.gmx.net:A:CreateNewLineFromSocket: 5 OK LIST completed

2360[1fcfc80]: 2015210:imap.gmx.net:A:SendData: 6 create "Gel&-APY-scht"

2360[1fcfc80]: ReadNextLine [stream=2015998 nb=23 needmore=0]
2360[1fcfc80]: 2015210:imap.gmx.net:A:CreateNewLineFromSocket: 6 OK CREATE completed

2360[1fcfc80]: 2015210:imap.gmx.net:A:SendData: 7 subscribe "Gel&-APY-scht"

2360[1fcfc80]: ReadNextLine [stream=2015998 nb=27 needmore=0]
2360[1fcfc80]: 2015210:imap.gmx.net:A:CreateNewLineFromSocket: 7 OK SUBSCRIBE
completed 

2360[1fcfc80]: 2015210:imap.gmx.net:A:SendData: 8 list "" "Gel&-APY-scht"

2360[1fcfc80]: ReadNextLine [stream=2015998 nb=43 needmore=0]
2360[1fcfc80]: 2015210:imap.gmx.net:A:CreateNewLineFromSocket: * LIST
(\Noinferiors) "/" "Gel&-APY-scht"

2360[1fcfc80]: ReadNextLine [stream=2015998 nb=21 needmore=0]
2360[1fcfc80]: 2015210:imap.gmx.net:A:CreateNewLineFromSocket: 8 OK LIST completed

2360[1fcfc80]: 2015210:imap.gmx.net:A:SendData: 9 select "INBOX"


Expected Results:  
The folder "Gelöscht" should be used as trash folder. No other folder should be
created.

I didn't have this problem in 1.7 and early 1.8 versions of Mozilla. The same
problem occurs in the Thunderbird release 0.7.3.

Comment 1

[punkadiddle]

The HTML for "Gelöscht" (which doesn't show in the original charset here) would
be "Gelöscht".

Comment 2

[punkadiddle]

Attachment: Test run on Mozilla 1.7.2

Testcase was run on fresh installation of Mozilla 1.7.2 with a new profile. It
shows Mozilla creating an additional folder which corresponds to the quoted
unicode name of the folder set in "trash_folder_name".
Includes logs of telnet IMAP session and Mozilla IMAP logfile.

Comment 3

[punkadiddle]

I finally was able to compile the Mozilla trunk.

After running through nsImapMailFolder::CreateSubFolders the folder list is
fine. It includes the specified trash folder in correct unicode chars (with
local characters).

The second folder named in ascii quoted unicode ("Gel&APY-scht") is created in
the view and ("Gel&-apy-scht") is created in the IMAP account within the
function nsImapProtocol::DiscoverMailboxSpec.

Before starting Mozilla I unsubscribed and deleted all expect the original trash
folder ("Gel&APY-scht") through a telnet IMAP session.

I don't think I can track this bug all on my own, but I am pretty sure there
must have been changes causing this bug sometime in August.

Comment 4

[punkadiddle]

After some more hours debugging I found the solution to this problem.
Mozilla applied a UTF7 conversion to the trash_folder_name preference string
which already was encoded in UTF7. This resulted in "Gel&APY-scht" being
converted to "Gel&-apy-scht" which then didn't match the folder name returned by
the IMAP server.
Mozilla must have changed my preference file adding the preference
user_pref("prefs.converted-to-utf8", true);
without converting the trash_folder_name value from UTF7 to UTF8 and without
converting the prefs.js file from ascii to unicode.

I don't know why this happend. So the real bug here is adding this setting
without convertig the trash_folder_name preference.

To fix this problem the setting can either be set to
user_pref("prefs.converted-to-utf8", false);
In this case the trash_folder_name setting can remain unchanged.

Modifying the trash_folder_name setting by assigning an UTF8 encoded string (as
done with the mailnews.label.description settings works only one time. Mozilla
overwrites the setting with the non-working UTF7 value.

You can also convert prefs.js to unicode (double-byte characters) and then use
normal characters without any encoding. That is use the folder name as displayed
in the GUI for trash_folder_name setting. This might have been intended...? But
you will need a unicode capable text-editor for this task.

I'm leaving this bug open because there remains some bug between prefs.js
character encoding, trash_folder_name character encoding and
"prefs.converted-to-utf8" setting. This is not a real IMAP protocol problem, but
a problem in the way the trash_folder_name setting is handled slightly different
than other preference values.
If this happens in a 1.8 final this problem will occur to lots of german people
 (and other people with non-us-conform trash folder names). 

Comment 5

[Wayne Mery (:wsmwk)]

fixed by bug 33451?

Comment 6

[David :Bienvenu]

we have a UI for picking the trash folder name - does that do the right thing with the encoding? If so, this bug would be fixed.

Comment 7

[Gerald Vogt]

The UI does not make a difference to the issue in this bug. It is an issue with the encoding of the prefs.js file and how it is converted internally. The UI would put the correct folder name into the preferences which are then incorrectly converted. 

I don't know whether this bugs is still relevant. The preference prefs.converted-to-utf8 which Michael mentioned is now back to default false. With that preference setting it works. I have temporarily switch the preference back to true in TB 2.0.0.19 but was not able to recreate the bug anymore with a few tests. Either my tests were not exhaustive enough or the handling and conversions of preference strings have been corrected.

I would suggest to set this bug to FIXED unless someone objects and can still reproduce the problem. I think in TB2 it works correctly now.

Comment 8

[punkadiddle]

It works in 2.0.0.19. Can't wait to finally see the GUI to configure that folder.