29312 - nbsp in mail converted to unprintable character 0xa0

Status: RESOLVED FIXED

(MailNews Core :: Composition, defect, P3)

Description

[Akkana Peck]

nbsp's in mail are converted to the unprintable character 0xa0 on sending.

I seem to recall that there was another bug on this already, but thought it was
resolved.  Yet I'm still seeing this, in two important places which users will see:

1. Use html compose, and have an html signature that contains nbsp entities
(e.g. I used "    --me", which seems like a fairly typical
thing to want to put in a signature).  Compose a message, and have it convert to
plaintext on send.  The nbsps should get converted to spaces, not to unprintable
characters with the high-bit set.  (They show up as question marks if you read
the resulting message in mozilla mail.)

2. Use html compose to send a message, and type groups of consecutive spaces
between words, which the editor will convert to nbsp.  Send the message as
plaintext.

Note that when I type consecutive spaces in the editor and do "Output
Plaintext", I don't see this happen.  So it's something mail is doing, not
something the editor is doing.

Comment 1

[nhottanscp]

This only happens composing in html and send as plain?
I got that too, pasting part of the message.

To: [e-mail]
Subject: compose html send as plain
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: quoted-printable

compose html send as plain
=A0 - naoki -

Comment 2

[Jean-Francois Ducarroz]

reassign to rhp.

Comment 3

[rhp (gone)]

Akkana,
Remember there were 2 sides to this problem. 

First (my side), was that I was converting spaces in plain text messages on 
quoting, etc.. into  's and that was showing this problem. I fixed that by 
not converting anymore.

Second (your side) was in the HTML to TXT converter. It seems to be converting 
&nbsp into this strange 0xA0 combination.

Does this make sense?

- rhp

Comment 4

[Akkana Peck]

That conversion is the charset converter.  Wasn't there some discussion about
mail wanting entities because international people wanted to search in messages,
or something, and so explicitly calling the charset converter with flags to use
the unicode codes for the entities instead of the html representation?

I'm unclear for the rules on when we should and shouldn't do this conversion. If
the rules have changed and the default flags used from nsHTMLToTXTSinkStream
need to be other than they are, assign the bug back to me and I'll change them.

Comment 5

[nhottanscp]

  are supposed to be mapped to 0xA0, no problems there.
If we want to map 0xA0 to 0x20 for text/plain, we can do that somewhere (but not 
in charset converter). In fact, if composed and sent as text/plain, only spaces 
to be sent, no 0xA0. I don't know where this mapping is done (editor or 
messenger).
The behavior reported by this bug only happens when composed in text/html and 
sent as text/plain. In order to be consistant, 0xA0 needs to be mapped to 0x20 
for this case too.
So who is currently do the mapping (0xA0 to 0x20)?

Comment 6

[nhottanscp]

0xA0 to 0x20 mapping is done in nsHTMLToTXTSinkStream::EncodeToBuffer

#define CH_NBSP 160
    for (PRInt32 i = 0; i < mBufferLength; i++)
    {
      if (mBuffer[i] == char(CH_NBSP))
        mBuffer[i] = ' ';
    }

Reassing to akkana.

The bug about creating entity is 27376 but that is for text/html (so not 
related to this bug).

Comment 7

[Akkana Peck]

There's really no way to ask the charset converter to do this -- we really have
to have a separate loop over the string?  Sorry, I thought there was.  

Okay, I'll take this back and clean up that code (I hadn't touched it because I
thought it was just temporary) and find out why it isn't working for signatures.

Comment 8

[Akkana Peck]

I checked in a fix that cleans up this code.  The nsHTMLToTXTSinkStream should
never emit nbsp chars (0xa0) now.

Comment 9

[Akkana Peck]

This is still happening.

To reproduce: reply to a plaintext mail message in the html editor.  Include
some runs of multiple spaces (so that in html, you'll have some nbsps).  Send as
plaintext only.  The message which arrives on the other end will have 0xa0
characters.

Comment 10

[Akkana Peck]

Checked in a more general fix.

Comment 11

[esther]

changing qa