Closed Bug 150422 Opened 24 years ago Closed 24 years ago

Non-ASCII text gets UTF-8 encoded in To: field

Categories

(MailNews Core :: Internationalization, defect)

x86
Windows 98
defect
Not set
normal

Tracking

(Not tracked)

VERIFIED INVALID

People

(Reporter: shill, Assigned: nhottanscp)

Details

From Bugzilla Helper: User-Agent: Mozilla/5.0 (Windows; U; Win98; en-US; rv:1.0.0) Gecko/20020530 BuildID: 2002053012 When I use non-ASCII characters in the To: field of a message, then examine the headers, I note that non-ASCII characters have been translated to different *non-ASCII* characters. For example, if I try to send an email to ийклмноп@isp.com: 54 6F 3A 20 20 C3-A8 C3 A9 C3 AA C3 AB C3 To: ......... AC C3 AD C3 AE C3 AF 40-69 73 70 2E 63 6F 6D .......@isp.com In ISO-8859-1, ийклмноп is encoded E8 E9 EA EB EC ED EF. Here ийклмноп was encoded (note the trailing space): 20 C3 A8 C3 A9 C3 AA C3 AB C3 AC C3 AD C3 AE C3 AF Perhaps this is UTF-16 as each character seems to be coded using 16 bits? However, I don't understand: if it is legal to send non-ASCII characters in the message header, why not use the actual non-ASCII character? Why do we go through the trouble of converting non-ASCII text into ASCII text in the Subject: field of the message headers? Relevant RFC is 2047, I think. ийклмноп is converted to =?ISO-8859-1?Q?=E8=E9=EA=EB=EC=ED=EE=EF?= Reproducible: Always Steps to Reproduce: 1. ^M 2. Type non-ASCII characters in To: field 3. Select "Send Later" 4. View message source Actual Results: Non-ASCII characters were translated to different non-ASCII characters. Expected Results: Not sure. Either leave "as is" (if 8-bit characters in the header is not a problem) or translate to ASCII, as is done in the Subject: field.
This text is UTF-8 encoded resummarizing
Summary: Strange representation of non-ASCII characters in To: field of the message header → Non-ASCII text gets UTF-8 encoded in To: field
I read the UTF-8 and Unicode FAQ: http://www.cl.cam.ac.uk/~mgk25/unicode.html U-00000000 - U-0000007F: 0xxxxxxx U-00000080 - U-000007FF: 110xxxxx 10xxxxxx The Unicode character U+00E8 = 1110 1000 (é) is encoded in UTF-8 as 11000011 10101000 = 0xC3 0xA8... It is indeed UTF-8 that is used in the To: field. What I don't understand is why we go to great length to have a 7-bit clean Subject: field while it is OK to have an 8-bit To: field. Are we worried that some mail servers might not handle 8-bit headers? Shouldn't we use UTF-8 to encode the Subject: field?
shill, I suppose it's just a bug that To: isn't encoded like subject. I should really read the spec, but 8-bit characters are not allowed in Mailheaders.
>For example, if I try to send an email to ийклмноп@isp.com: I think that is not legal even if MIME encoded, but I don't find a document so far. cc momoi, taka
This is an UI bug. UI must NOT add "@isp.com" when user entered a non ASCII character.
Found it in RFC2822: http://www.rfc-editor.org/rfc/rfc2822.txt 2.2. Header Fields Header fields are lines composed of a field name, followed by a colon (":"), followed by a field body, and terminated by CRLF. A field name MUST be composed of printable US-ASCII characters (i.e., characters that have values between 33 and 126, inclusive), except colon. A field body may be composed of any US-ASCII characters, except for CR and LF. However, a field body may contain CRLF when used in header "folding" and "unfolding" as described in section 2.2.3. All field bodies MUST conform to the syntax described in sections 3 and 4 of this standard. Christian was right when he said that non-ASCII characters were not allowed in the header. Of course, the problem is not limited to the To: field. It holds for all the email address fields (To, CC, BCC, Reply-To). Should we never allow the user to send a message to an address with non-ASCII characters? I suppose operating systems can handle non-ASCII characters in the username. Is there a scheme, similar to that used for the Subject: field available for email addresses? bébé -> =?ISO-8859-1?Q?b=E9b=E9?= (RFC 2047)
Status: UNCONFIRMED → NEW
Ever confirmed: true
RFC 2047 states: http://www.cis.ohio-state.edu/cgi-bin/rfc/rfc2047.html#page-8 + An 'encoded-word' MUST NOT appear in any portion of an 'addr-spec'. + An 'encoded-word' MUST NOT appear within a 'quoted-string'. + An 'encoded-word' MUST NOT be used in a Received header field. I guess the Internet is not ready for ångström@isp.com ;(
Okay, mark as invalid. The UI problem can be filed separately.
Status: NEW → RESOLVED
Closed: 24 years ago
Resolution: --- → INVALID
I'm not sure I understand why this bug has been marked INVALID. Is it because it's the UI's job to reject/refuse non-ASCII characters in the To, Cc, Bcc and Reply-To fields? If Mozilla Mail produces non-compliant mail headers, isn't that a bug? Naoki, when you say "The UI problem can be filed separately" what do you mean?
I mean that the original report was filed with the assumption that the input was not invalid and has to be processed. But as we found out that is not the case. We want to prohibit such cases by UI. In my experience, once the meaning of the bug changes then people tends to be confused about the current issue. I marked the bug as invalid since the original report expectation was not correct. I think it is less confusing to file a separate bug to target the UI issue.
Filed bug 150946 for the UI issue.
Marked it as verified.
Status: RESOLVED → VERIFIED
Product: MailNews → Core
Product: Core → MailNews Core
You need to log in before you can comment on or make changes to this bug.