48041 - PR2 msg composer sends CERs not recognized by 4.x messenger

Status: VERIFIED FIXED

(MailNews Core :: Composition, defect, P2)

Description

[tao]

A sample message excerpt:

  ... [snip]
  PR2 includes some exciting new features not found in the PR1 release,
  including Themes, which allows users to change the look of Netscape 6 by
  applying "skins" to the browser.  PR2 is currently available in two themes, 
  the default “modern” theme and for those who just can’t do 
  without their 4.x interface, a “Classic” theme retains the look 
  and feel reminiscent of earlier Netscape browsers.  Theme Switcher allows you 
  change back and forth as often as you like, and all components of the
  browser take on the look-and-feel of the theme you select.
  ...[snip]

Comment 1

[nhottanscp]

Nominating for nsbeta3.

Comment 2

[tao]

Yes,...

As you can see from the excerpt that the paragraph containing CERs are not 
legible; not a commercial grade product's behavior...

Comment 3

[Katsuhiko Momoi]

So what is the approach to fixing this problem?
Sending raw 8-bit characters for these special
symbols?

Comment 4

[nhottanscp]

I am thinking about two changes.
1) Use HTML32 CER only for mail send. Others are fallback to NCR.
2) In charset conversion, if out of charset range error happens for ISO-8859-1 
then try again with windows-1252 converters and re-label as windows-1252.

I see 4.x displays smart quotes by doing both cases.

Comment 5

[bobj]

I bet the CERs in 4.x are still using the CERs defined by HTML 3.2 which
does not include the smart quote characters.

If we want to be backwards compatible, we should restrict the generated CERs
to the HTML 3.2 set and use NCRs instead for those outside that set.

There was another discussion (bug?) that we should not use CERs/NCRs if
the the character exists in the current document encoding.  So for smart
quotes, we'd use raw characters if encoding in cp1252 and use NCRs if encoding
in iso-8859-1.

Should we file a bug for 4.75 to add the HTML 4.x additional CERs?

Comment 6

[bobj]

whoops.  I was responding to momoi's comment and didn't notice nhotta's
intervening comment...

Why restrict HTML 3.2 CERS only for email?  We'll have the same problem
viewing HTML page with these CERs with the 4.x browser.

Do we really want to pollute the internet with MS cp1252?  Aren't we trying
to be the standards browser?  Or am I being an impractical purist????

Comment 7

[nhottanscp]

We may use HTML 3.2 CER only for editor too (the change is trivial). For editor, 
it's nicer if the user can control versions, cc Akkana.

I don't care about sending windows-1252 for this special case. But it may look 
weird to send windows-1252 from Macintosh when the mail contains smart quotes.
I would like to know which is more supported by other mail clients, NCR or 
windows-1252.

Comment 8

[Frank Tang]

nsbeta3+ per bug meeting. mark it nsbeta3+ P2

Comment 9

[nhottanscp]

Attachment: patch to generate only CER supported by 4.x

Comment 10

[nhottanscp]

By the patch, it will not generate HTML 4 CER both for mail and composer. 
Instead NCR will be generated.
If smart quotes are used in plain text mail, they will be transliterated to 
plain quotes before it is sent. The user will see the out of charset range alert 
since the message contains window-1252 characters in ISO-8859-1 mail.

Comment 11

[nhottanscp]

Checked in the patch.

Comment 12

[nhottanscp]

A research about other clients:
AOL 5 clients (Win and Mac) does not interpret smart quotes in NCR. But they 
understand smart quotes encoded as windows-1252.
AOL 5 clients (Win and Mac) send smart quotes encoded as UTF-8.

Comment 13

[lchiang]

kat - can you verify this one?

Comment 14

[Katsuhiko Momoi]

Sure.

Comment 15

[Katsuhiko Momoi]

** Checked with 9/1/2000 Win32 build **

This seems to be working as intended:

0. When you send ISO-8859-1, 

1. Win-1252 proprietary characters (0x80-0x9F) are sent using 
   NCRs in HTML mail.
2. HTML 3.2 entities are still sent in CERs.
3. If you use plain text mail, these are transliterated where available
   and otherwise sent as "?".
   A warning does come up.

4. I've looked at all the remaining HTML 4 entities, i.e. other than 3.2
   type and Win-1252 type. They are all sent in NCRs in HTML and 
   "?"s in plain text.

These are the desired result. 
The main purpose of this bug was to make things work for 
Win-1252 only characters, I will verify this fix based on 
this Windows result. While these characters can be sent 
on Unix and Mac, that is unlikely.

Marking the fix verified.

Comment 16

[Katsuhiko Momoi]

These changes are also reflected in HTML composer. 
I think we should file a new bug for future as follows:

1. Create an advanced pref option to choose between Win-1252, NCRs, or UTF-8 
   in HTML Mail Composition if the user input such characters under 8869-1.
2. For plain text mail, offer to send such mail in UTF-8 or Win-1252.
    in addition to "send it anyway" option.
   (This could be another advanced pref option.)