650623 - Non-ASCII tags are corrupted on IMAP servers due to lowercased MUTF-7 encoding

Status: RESOLVED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[Сундуков Алексей]

User-Agent:       Opera/9.80 (Windows NT 5.1; U; ru) Presto/2.8.131 Version/11.10
Build Identifier: 3.1.9

Mozilla Thunderbird correct stored non latin IMAP folder, but breaks flags. Mozilla Thunderbird stored non system flag (http://tools.ietf.org/html/rfc2060#se...) in UTF-7, but conver character in lower case. Example: I set marker "Тест" (like 'test' in russian language), in UTF-7 it's presented a string "+BCIENQRBBEI": 

 echo 'Тест' | iconv -f UTF-8 -t UTF-7 
 +BCIENQRBBEI 

 but Thunderbird convert it in &bcienqrbbei-. I see this in dovecot-keywords file, I see this in TCP traffic via Wireshark: 

 DONE 
 10 OK Idle completed. 
 11 uid store 20 +FLAGS (&bcienqrbbei-) 
 * 2 FETCH (UID 20 MODSEQ (96) FLAGS (\Seen &bcienqrbbei-)) 
 11 OK Store completed. 
 12 IDLE 
 + idling

Reproducible: Always




Because of this error I can not integrate with its own software. Via IMAP my software good read/write non system flags, but Mozilla Thunderbird don't see its and set broken flags.

Comment 1

[Wayne Mery (:wsmwk)]

Сундуков will be running new tests

Comment 2

[WADA:World Anti-bad-Duping Agency]

&bcienqrbbei- is not utf-7. It's modified utf-7 of perhaps "Тест" in Unicode or system charset or other charset.

In Tb, IMAP flag is "Tag", and is defined in prefs.js.
  mailnews.tags.<used_tag=flag>.tag = <Typed_tag=flag>
  <used_tag=flag name> : lower case of <Typed_tag=flag>,
                         with removing prohibited special chars
  In IMAP, <used_tag=flag> is used by "store Flags".
  <Typed_tag=flag> is shown at Thread pane, Message Header pane.
Tb looks to use <used_tag=flag> = modified-utf-7 of lower case of <Typed_tag=flag_>, or lower case of modified-utf-7 of <Typed_tag=flag>.

To Сундуков Алексей(bug opener) :
What is saved in Tb's prefs.js for "Тест"?

By the way, "Flag" in IMAP is merely an "8bits ascii string without some special characters" and is case insensitive.
What is reason to use non-ascii character for IMAP flag intentionally?
Do you know RFC which refers to or defines "non-ascii string use in IMAP Flag"?
Note:
- For Mbox name, modified-utf-7 use is defined by IMAP.
  Tb looks to apply same rule to typed tag name.
- For authentication string in SASL, use of utf-8 is defined.

Comment 3

[WADA:World Anti-bad-Duping Agency]

What is stored as Flag for the "Тест" by other application or mail client?
Is following a workaround?
  mailnews.tags.<stored Flag string by other app>.tag = Тест

Comment 4

[mguessan]

Please check http://sourceforge.net/p/davmail/feature-requests/72/ for additional details on IMAP flags support issue.

DavMail needs to store IMAP flags as Outlook categories with non encoded values.
However in Thunderbird keyword is RFC2060 encoded *before* toLowerCase
=> the flag value is corrupted

Comment 5

[WADA:World Anti-bad-Duping Agency]

(In reply to mguessan from comment #4)
> Please check http://sourceforge.net/p/davmail/feature-requests/72/ for
> additional details on IMAP flags support issue.
>   user_pref("mailnews.tags.it.tag", "IT");
>   I tried modifying these setting to have an uppercase variable name,
>   like mailnews.tags.IT.tag, but that doesn't seem to work.
>     user_pref("mailnews.tags.IT.tag", "IT");

Which is "doesn't work as you expect"?
    user_pref("mailnews.tags.IT.tag", "IT");
(a) When Tag named "IT" is added by Tb,
    Tb stores "it" instead of "IT". "uid xx store +Flags (it)" is used
    because flag in IMAP is case insensitive.
    Then other mail client fails to detect IMAP named "it".
(b) Tb doesn't detect already stored flag of "IT" in IMAP mail.
(c) Tb detects already stored flag of "IT" in IMAP mail, and shows it
    as "Tag labeled IT" at Thread pane/Header pane,
    but, when remove is requested in Tb, fails to remove flag of "IT".
    - "uid xxx store -Flags (it)" is issued by Tb because flag in IMAP
      is case insensitive,
      then server doesn't remove flag named "IT".
(d) other

IIRC, problem is (a) and (b), and "using same case when remove flag" is already requested to avoid problem like (b), although I'm not sure.

Anyway, rule on IMAP flag defined in IMAP4rev1 is;
- Characters are 7-bit US-ASCII unless otherwise specified.
- Usable character in IMAP flag = <any CHAR except atom-specials>
- IMAP flag is also case insensitive.
And, Tb's current implementation on Tag(flag when IMAP) is;
- Definition in prefs.js
  user_pref("mailnews.tags.<flag_name>.tag", "<Label_string_of_flag>");
- Detection of already stored flag is case insensitive.
- When store flag, Tb uses lower case of <flag_name> as flag name.
    "uid xxx store +/-Flags ( lower case of <flag_name> )
- Upon Tag definition, Tb generates <flag_name> string from string of
  <Label_string_of_flag> which is typed by user at UI,
  according to IMAP rule on IMAP flag, and rule on variable name in
  JavaScript.

Please isolate issues:
(i)  Unexpected <flag_name> string generation from string of
     <Label_string_of_flag> which is typed by user at UI
     when <Label_string_of_flag> contains non 7bits-ascii.
     Note-1:
     Tb looks to generate modified-utf-7 string.
     Note-2:
     This can be corrected by manual modification of <flag_name> part.
(ii) Tb uses lower case of <flag_name> in "uid store +/-Flags" command.
     It may be different from expectation of server and other client.

Comment 6

[WADA:World Anti-bad-Duping Agency]

I checked with Tag=日本語, IMAP Mbox name=日本語, with Tb 17 on Win-XP.
(日本語=="Japanese" in Japanese)
> prefs.js entry create by Tb for Tag=日本語
>   user_pref("mailnews.tags.&zevnliqe-.tag", "日本語");
> File name for IMAP Mbox name=日本語
>   &ZeVnLIqe-.msf
> IMAP Server returns modified-utf-7 for non-ascii Mbox name.
> Tb uses "modified-utf-7" + ".msf" for .msf file.

When non-ascii Tag name is requested,
(1) Tb converts the non-ascii string to modified-utf-7 first,
(2) Then changes to lower case for prefs.js entry name==IMAP flag name.
Because no rule is defined for "IMAP flag of non 7bits-ascii" which is RFC violation, IMAP server and IMAP clients can do anything on "IMAP flag of non 7bits-ascii" requested by user.
So, this is never RFC violation by Tb(Tb uses 7bits-ascii only always).
However, converting to modified-utf-7, which is rule on Mbox name only in IMAP, is not good practice, at least when it's used as IMAP flag.

Confirming.

And, Tb converts "Tag name part" in prefs.js entry to lower case, when Tb issues "uid xxx store +/-Flags (tagname)".
So, even if user manually changed "&zevnliqe-" part in prefs.js to string of "utf-8 binary consists of 7bits-ascii binary only", Tb converts it to lower case if the utf-8 binary contains upper case 7bits-ascii binary.
So, IMAP server and/or IMAP client, who expects utf-8 or who expects upper case 7bits-ascii for IMAP flag, can't understand IMAP flag stored by Tb.

Is change like "use Tag name part of pres.js entry as-is, if the string consists of 7bits-ascii only" needed?
Note:
This kind of change in "delete existent Flag of IMAP" is needed at least for next case.
- Old Tb/Sm used $LabelN(where N is digit) for standard Tag=Flag in
  IMAP.
- After a while, spec of Tb/Sm was changed, and $labelN is used always.
- Thus, Tb/Sm can't remove old $LabelN stored in existent mail,
  which was stored by Tb/Sm himself,
  if IMAP server doesn't treat IMAP flag case-insensitive correctly.
  Or, Tb/Sm doesn't issue "uid xxx -Flags". (I don't remember well.)

Comment 7

[Сундуков Алексей]

I can't test Tb. Dovecot log: "TLS: SSL_read() failed: error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca: SSL alert number 48". But telnet work fine: 

$ telnet localhost 143
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS AUTH=LOGIN AUTH=PLAIN] Dovecot ready.
a login [e-mail] *******
a OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS] Logged in
b select inbox
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft &bcienqrbbeiepgqybdaetw-_&bdwenqrcbdoema- $label1 $label2 $label3 $label4 $label5 test &beienqq6beeeqg- &0YLQtdGB0YI- &bcienqrbbei- &bbwenqrcbdoema- &0KHQu9C+0LLQvg-)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft &bcienqrbbeiepgqybdaetw-_&bdwenqrcbdoema- $label1 $label2 $label3 $label4 $label5 test &beienqq6beeeqg- &0YLQtdGB0YI- &bcienqrbbei- &bbwenqrcbdoema- &0KHQu9C+0LLQvg- \*)] Flags permitted.
* 3 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1298927723] UIDs valid
* OK [UIDNEXT 23] Predicted next UID
* OK [HIGHESTMODSEQ 99] Highest
b OK [READ-WRITE] Select completed.
e logout
* BYE Logging out
e OK Logout completed.
Connection closed by foreign host.

PHP example set IMAG flag:
<?php
$mbox = imap_open('{ru-region.ru:993/imap/ssl/notls/novalidate-cert}INBOX', '[e-mail]', '*******');
imap_setflag_full( $mbox, '1,2', imap_utf7_encode('Слово') );
imap_close($mbox);
?>

Use this script http://stackoverflow.com/questions/3285690/how-to-get-imap-flags I can read flag:
<?php

include('ImapSocket.php');

$imap = new ImapSocket(array(
	'server'   => 'ru-region.ru',
	'port'     => 993,
	'login'    => '[e-mail]',
	'password' => '*******'
), 'INBOX');

$imap_flag_list = $imap->get_flags(1);
foreach ($imap_flag_list as $flag) {
	echo imap_utf7_decode($flag) . "\n";
}
?>

Result:
$ php -f get_flag.php
\Answered
\Seen
тест
Слово

Dovecot save my flag correct:
$ cat /var/vmail/ru-region.ru/13/dovecot-keywords
0 &bcienqrbbeiepgqybdaetw-_&bdwenqrcbdoema-
1 test
2 $label1
3 $label2
4 $label3
5 &beienqq6beeeqg-
6 &0YLQtdGB0YI-
7 $label4
8 $label5
9 &bcienqrbbei-
10 &bbwenqrcbdoema-
11 &0KHQu9C+0LLQvg-

I want use IMAP flags in my php application and Tb.

Comment 8

[Franta Hanzlik]

Same problem. Claws mail dispaly garbage instead of original flag (with national Latin accented characters). IMO TB/SM behavior is bad - it should either restrict users to using only lowercase ASCII letters, or allow full utf-8 and pass them correctly to IMAP server (not cripled to lowercase). Personaly I vote for second, as it will work with non-latin languages.

Comment 9

[gene smith]

Note: I worked with another Russian to solve issues with his tags (in Russian) and there was no complaints that they appear as lower-case ASCII. But maybe this is a different issue, not sure.

Comment 10

[vlad.nicola]

Is there any update on this? I have the same problem. I try to add the Tag called â which in UTF7 modified would be &AOI- but Thunderbird sends it via IMAP as &aoi- which is first of all invalid because the last 2 bits resulted from the base64 modified encoding are not 00, but 10 (see reference from rfc below), but most importantly, ignoring that, this character would translate into utf8 to 檈

UTF7 is case sensitive. Thunderbird should not send invalid UTF7 via IMAP. I would rather have it not expose non-ascii tags or even not expose tags at all.

From RFC-2151: UTF-7
Instead, when encoding, zero bits are
added to pad to a Base64 character boundary. When decoding, any
bits at the end of the Modified Base64 sequence that do not
constitute a complete 16-bit Unicode character are discarded. If
such discarded bits are non-zero the sequence is ill-formed.

Comment 11

[Hartmut Welpmann [:welpy-cw]]

Attachment: Bug 650623 - Rewrite IMAP tag key generation to produce valid atoms and drop broken MUTF-7. r=#thunderbird-reviewers

Historically, custom tag names were converted to IMAP keywords using a Modified UTF-7
encoder, followed immediately by a lowercase conversion. Because MUTF-7 relies on
case-sensitive Base64, the forced lowercasing irreversibly corrupted the string for
any non-ASCII tags (e.g., Cyrillic, Kanji), leading to garbled output on other
clients (Bug 650623).

Furthermore, MUTF-7 is strictly defined for folder names, not flags. RFC 3501
mandates that IMAP flags must be case-insensitive, 7-bit ASCII atoms. The legacy
MUTF-7 conversion often produced the shift character '&', which is an atom-special
that crashes strict IMAP parsers (like iCloud) during -FLAGS commands.

This patch completely modernizes Tag Key generation, prioritizing both protocol
safety and lossless data preservation for international users:

• Drops the broken MUTF-7 conversion entirely.
• Introduces a custom "Modified Quoted-Printable" hex encoding (=XX) for non-ASCII
  characters, spaces, and IMAP atom-specials. This ensures the resulting key is
  a 100% compliant, server-safe IMAP atom while retaining the exact UTF-8 byte
  sequence for lossless reverse-decoding.
• Explicitly escapes the = character to prevent decoding collisions.
• Replaces the legacy Netscape-era A, AA, AAA deduplication suffix loop
  with readable integer suffixes (_1, _2) to gracefully handle naming
  collisions.

Comment 12

[Hartmut Welpmann [:welpy-cw]]

Attachment: Bug 650623 - Modernize CSS selector generation for message tags. r=#thunderbird-reviewers

The nsMsgTagService::GetSelectorForKey function contained an obsolete
comment referencing the legacy MUTF-7 tag encoding, as well as a Mozilla
String Guide violation where SetCapacity() was immediately neutralized by
a subsequent Assign() call.

This commit refactors the CSS selector generator:

• Replaces the unnecessary double-pass loop with a clean, single-pass build.
• Adopts mozilla::IsAsciiAlphanumeric and BeginReading()/EndReading()
  for standard, safe XPCOM string iteration.

Comment 14

[Hartmut Welpmann [:welpy-cw]]

(In reply to Hartmut Welpmann [:welpy-cw] from comment #11)

• Introduces a custom "Modified Quoted-Printable" hex encoding (=XX) for non-ASCII
  characters, spaces, and IMAP atom-specials. This ensures the resulting key is
  a 100% compliant, server-safe IMAP atom while retaining the exact UTF-8 byte
  sequence for lossless reverse-decoding.

This might be a useful change considering a possible implementation for bug 731143.

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/7c5287edb0db
Rewrite IMAP tag key generation to produce valid atoms and drop broken MUTF-7. r=BenC
https://hg.mozilla.org/comm-central/rev/e5354e11ec68
Modernize CSS selector generation for message tags. r=mkmelin