82591 - HTML mail from News.com has '?' in the body (using IMAP)

Status: VERIFIED FIXED

(MailNews Core :: Internationalization, defect, P4)

Description

[David Park]

HTML mail from News.com doesn't seem to understand certain tags that are not 
visible.  I have '?' appearing through the mail.  Displays fine in 4.x 
Messenger.

Comment 1

[(not reading, please use [e-mail] instead)]

can you attach the rfc/822 message to this bug?

load the message, and do "File | Save As | File".

Comment 2

[David Park]

Attachment: saved mail file

Comment 3

[David Park]

Attachment: screenshot of problem mail

Comment 4

[Andreas M. "Clarence" Schneider]

The message claims to use charset="us-ascii", but actually uses 8 bits (0xA0,
non-breaking space in ISO-8859-1). 0xA0 appears as '?' here, which is correct
because it isn't defined in US-ASCII.

Comment 5

[David Park]

But it seems to work fine in all other browsers N4.x as well as IE

Comment 6

[(not reading, please use [e-mail] instead)]

that attached mail file worked fine for me on win2k.

I'll try my winnt and win98 builds.

cc'ing nhotta who would know what we are supposed to do in this case.

Comment 7

[(not reading, please use [e-mail] instead)]

works for me on my win98 box running 20001052504.

dyp, do you see the problem when you save
http://bugzilla.mozilla.org/showattachment.cgi?attach_id=36098 as "foo.eml" and
then open it up in the brower?

if not, I'll turn it into a folder and see if I can see it.

Comment 8

[David Park]

saved it as *.eml file and viewd in Browser.  Looks fine in the browser.

Comment 9

[nhottanscp]

I think the question marks are non breaking spaces.
Nbsp is not in "us-ascii" range. In the message charset is set to "us-ascii".
Content-Type: text/html; charset="us-ascii"

I think the question mark will also appear in HTML if that explicitly labeled as
"us-ascii" like below.
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">

David, could you try that?

Comment 10

[Boris Zbarsky [:bzbarsky]]

Attachment: web page using us-ascii (and the 0xA0 character)

Comment 11

[Boris Zbarsky [:bzbarsky]]

Attachment: web page using iso-8859-1 (and 0xA0)

Comment 12

[Boris Zbarsky [:bzbarsky]]

Not a mail-only problem.  Over to i18n.

I would probably suggest wontfix on this one....

Comment 13

[nhottanscp]

There is optimization in mail code for "US-ASCII".
We could verify the data is really 7 bit before applying the optimization.
That would prevent invalid UTF-8 data to be sent to the layout and might also 
fix this problem.

Comment 14

[nhottanscp]

Attachment: Patch, added ASCII check to the US-ASCII optimization code in order to avoid generating illegal UTF-8.

Comment 15

[Andreas Becker]

Switching qa contact to [e-mail] for now.

Comment 16

[Jean-Francois Ducarroz]

Looks good but the code

+    // make sure if the data is really ASCII in order to prevent generating
illegal UTF-8
+    PRInt32 len = inLength;
+    PRBool bASCII = PR_TRUE;
+    for (PRInt32 i = 0; i < inLength; i++) {
+      if (inBuffer[i] & 0x80) {
+        bASCII = PR_FALSE;
+        break;
+      }
     }

can be written more efficiently! You can use a pointer instead of a array. Also,
you could scan 32bits data at once instead of 8bits.

Comment 17

[nhottanscp]

Attachment: Patch, updated with ducarroz's comment.

Comment 18

[Jean-Francois Ducarroz]

R=ducarroz if you replace the line

+      if (*p & (PRUint32)0x80000000) {

by

+      if (*p & (PRUint32)0x80808080) {

Comment 19

[nhottanscp]

Attachment: Patch, changed 0x80000000 to 0x80808080.

Comment 20

[(not reading, please use [e-mail] instead)]

sr=sspitzer

assuming that similar code doesn't already live somewhere else in the tree.

Comment 21

[nhottanscp]

Fixed in trunk.

Comment 22

[(not reading, please use [e-mail] instead)]

it looks like there is similar code else where.

see nsMsgSearchOnlineMail::Encode() in nsMsgImapSearch.cpp

I'll log a bug on nhotta (or ducarroz) to move that code out and into a common 
place.

Comment 23

[(not reading, please use [e-mail] instead)]

I opened bug #92220 to track the duplicate code.

Comment 24

[nhottanscp]

Reopen, the change was backed out because of a regression bug 92330.
nsMsgHeaderParser also calls it with a wrong charset, filed as bug 92420.

Comment 25

[nhottanscp]

The last patch has a problem. If the srs cad dst charsets are equal then it
should just dup the string no 7 bit check is needed.

Comment 26

[Frank Tang]

reassign to jbetak and mark m9.4

Comment 27

[[e-mail] (away - not reading bugmail)]

nominating for nsbranch

Comment 28

[Jaime Rodriguez, Jr.]

Per Selmer suggestion, I am doing this myself.

Moving nsbranch designation from the Status Whiteboard to Keywords.

Comment 29

[Frank Tang]

so... this is a bug for mailer which does not implement MIME correctly. Delay
the fix till nhotta back. mark it as m0.9.6

Comment 30

[msanz]

nsbranch- since Frank moved it to 0.9.6

Comment 31

[nhottanscp]

Reassign to nhotta.

Comment 32

[nhottanscp]

*** Bug 104282 has been marked as a duplicate of this bug. ***

Comment 33

[nhottanscp]

By bug 92420, this is not used by headers.
And part of bug 12481 caches charset convertors in mime object. After that is
fixed, we can remove the optimization code for us-ascii.

Comment 34

[nhottanscp]

Attachment: If charset is "us-ascii" then use "ISO-8859-1".

Comment 35

[Jean-Francois Ducarroz]

Comment on attachment 43301 [details] [diff] [review]
Patch, changed 0x80000000 to 0x80808080.

R=ducarroz, SR=sspitzer

Comment 36

[Jean-Francois Ducarroz]

Comment on attachment 54880 [details] [diff] [review]
If charset is "us-ascii" then use "ISO-8859-1".

I far I know, us-ascii == ISO-8859-1. R=ducarroz

Comment 37

[(not reading, please use [e-mail] instead)]

Comment on attachment 54880 [details] [diff] [review]
If charset is "us-ascii" then use "ISO-8859-1".

sr=sspitzer

the strcasecmp is fine, since both are C strings.

Comment 38

[nhottanscp]

Checked in to the trunk.

Comment 39

[ji]

Verified as fixed with the latest build.