481387 - Non-ASCII characters in mail headers must always be encoded

Status: RESOLVED DUPLICATE of bug 481379

(MailNews Core :: Internationalization, defect)

Description

[dynamis (Tomoya ASAI)]

When we input non-ascii strings in mail headers (to/cc/bcc etc), they must be encoded into 7-bit encodings (base64 or quoted-printable).

If we input non-ascii strings (say "日本語文字列") as a display name like:
  "日本語文字列" <[e-mail]>
  日本語文字列 <[e-mail]>
the multi-byte characters will be properly base64 encoded to:
  To: =?UTF-8?B?IuaXpeacrOiqnuaWh+Wtl+WIlyI=?= <[e-mail]>
  To: =?UTF-8?B?5pel5pys6Kqe5paH5a2X5YiX?= <[e-mail]>
# with Japanese Firefox, not UTF-8 but ISO-2022-JP like:
# To: =?ISO-2022-JP?B?GyRCRnxLXDhsSjg7ek5zGyhC?= <[e-mail]>

But if we input non-ascii strings only (without <[e-mail]>) like:
  日本語文字列
this will not base64 encoded and send as is in UTF-8 encoding:
  To: 日本語文字列

Thunderbird must encode all headers into 7-bit encoding.

I guess that to/cc/bcc fields will be treated as mail address and they will not encoded unless using <[e-mail]> and display name, in other words, only display name part will be encoded into 7-bit.
# concerning IDN, they must be encoded into punny-code I think but not sure yet


 - Note about erorr dialogs you may see:
If this is the only address field (to/cc/bcc) you input, you'll see error dialog saying it isn't valid mail address form (user@host).
But if you input at least other valid mail address, you will not see the error dialog and the mail will be sent.
# see bug 481379

Some mail server will reject invalid mail address and you'll see error message from the smtp server but some servers will not check from/to/cc/bcc headers and the mail will be sent.

 - Note about importance of this bug:
Some servers (and clients) cannot treat 8bit chars within header and it may cause serious problems like:
 dataloss: The message may be going to be garbage text (mojibake)
 privacy:  Some (poor) servers may treat following part as body and
           bcc headers will be send as a part of body
# since 8bit chars must only appear wihtin body, poor server may treat 8bit
# chars as a hint of body part start (even before double line breaks)

Comment 1

[Mark Banner (:standard8)]

(In reply to comment #0)
> But if we input non-ascii strings only (without <[e-mail]>) like:
>   日本語文字列
> this will not base64 encoded and send as is in UTF-8 encoding:
>   To: 日本語文字列

Is this an error anyway, or are you saying a valid email address can be produced entirely in japanese characters?

Are you thinking about the case where the server allows the local-part only and appends the domain automatically?

> # concerning IDN, they must be encoded into punny-code I think but not sure yet

No, or at least not in this bug. See bug 127399 and the related standards that are currently being published.

Comment 2

[dynamis (Tomoya ASAI)]

I didn't intend to say we should treat email address which includes Japanese.
What I want to say was merely we should not send 8-bit chars within headers.

Thunderbird is now actually sending 8-bit chars within headers and I filed this bug but actually it can be reproduced with invalid email address.
If the email validator block invalid email address properly, 8-bit chars will not be appear within headers of actually sent email.

As Mark say it is email address error, not error of encoder of headers
# unless we find other case which let thunderbird send 8bit chars as is.

So what we have to do is to fix email address validator not to allow invalid email address before sending mails.
The bug I filed about email address validator is the bug 481379 and this problem (utf-8 chars will appear within the header) will be solved if it solved.

 -> RESOLVED DUPLICATE of bug 481379


> > # concerning IDN, they must be encoded into punny-code I think but not sure yet
> 
> No, or at least not in this bug. See bug 127399 and the related standards that
> are currently being published.

I see. Thanks information.