53147 - Charset detection for replying message

Status: RESOLVED WONTFIX

(MailNews Core :: Internationalization, enhancement)

Description

[Masaki Katakai]

Currently Mozilla Mail takes a charset of original message
at replying message. For example, when I try to reply
a message that charset=us-ascii, the composing message
is considered as iso-8859-1 charset and it will be sent
as charset=iso-8859-1.

However, users often do the following operation and
meet an alert dialog,

1) select a message that charset=us-ascii
2) click Reply button
3) Delete original recipient(s)
4) Add new recipients who can read Japanese
5) Modify the contents and insert Japanese character

6) Click Send button

Mozilla Mail shows alert that

This message contains characters not found in the
chosen Character Coding. The message may be unreadable
to recipient. Send it anyway or "Cancel" to choose a
different Character Coding.

Problems are,

- users may be confused because he/she started Mozilla in
japanese locale and just inserted japanese character,
but the alert says extra operations are needed.

- users may not understand what Character Coding is

Character Coding can be selected by View menu,
but there is no word "View menu" in this alert

- users may not understand what iso-2022-jp is?

Normally users (not Mozilla developer!) is not
familiar with iso-.... word

It would be best we can provide Auto-detection
to determine the charset. Or more useful alert
messages should be provided for normal users.

Comment 1

[nhottanscp]

There is a bug 52429 for wording about the warning.

Comment 2

[nhottanscp]

When we can guess outgoing mail charset from the unicode body text, we can put 
the guessed result into the alert box. That would help the user to choose 
charset. I think applying the guessed result directly without the alert may send 
out a mail with wrong charset label, so we want to avoid that.
Currently we do not have the guessing functionality. Are there anything 
available anywhere? Or any idea if we need to create one?

Comment 3

[Katsuhiko Momoi]

QA contact to ji.

Comment 4

[Travis Crump]

"Currently Mozilla Mail takes a charset of original message at replying
message."(Original Post)   Ummm, this would be nice, but I am not seeing it.  I
am in the en-US locale and the majority of my mail is in iso-8859-1. 
Occasionally I get mail with some japanese in it.  When I try to reply, the
charset=iso-8859-1 is used and the quoted text becomes gobbeldy-gook.  Even if I
then 'fix' the charset by changing the charset to iso-2022-jp, the quoted text
remains gobbeldy-gook although it does send correctly.  In order to really fix
it, I have to save the reply as a draft after changing charset, close it, and
open the draft.  I notice that in my View->Character Coding->AutoDetect it is
set to off and my Character Coding is set to Western.  If I try to click
'Japanese' or 'Universal' in my AutoDetect menu nothing happens(ie if I return
to the AutoDetect menu it is still set to 'off').  This is undoubtedly the cause...

Comment 5

[Boris 'pi' Piwinger]

Let me suggest a way to do it (that's what Forte Agent does): There is an
ordered list of charsets a user wants to allow, e.g., US-ASCII, ISO-8859-1,
ISO-8859-15, utf-8. The mail/article is sent with the leftmost charset which
works (unless explicitely set by the user).

Further, any ASCII message should always be sent as such if possible.

pi

Comment 6

[nhottanscp]

Similar feature is available on the trunk by bug 109342.
Per charset, fallback charset list can be supplied.
For example,
user_pref("intl.fallbackCharsetList.ISO-8859-1", "ISO-8859-15, UTF-8");

Comment 7

[Boris 'pi' Piwinger]

> Per charset, fallback charset list can be supplied.
> For example,
> user_pref("intl.fallbackCharsetList.ISO-8859-1", "ISO-8859-15, UTF-8");

Thanks for the hint. This is certainly useful. But just to make clear for
everybody else: I basically also meant the other way round, i.e., reducing to a
"smaller" charset if possible to enhance the chance for the recipient to read it.

I found the following in prefs.js:
user_pref("intl.charsetmenu.browser.static", "ISO-8859-1, ISO-8859-15, UTF-8");
user_pref("intl.charsetmenu.mailedit", "ISO-8859-1, ISO-8859-15, UTF-8");

This looks almost like my suggestions. It just does not have the effect I meant.
Maybe these setting (probably the later) can be used if you like to follow my
suggestion in comment 5 (order of preference for charset use).

pi

Comment 8

[nhottanscp]

>reducing to a
> "smaller" charset if possible to enhance the chance for the recipient to read it.
bug 136664

Comment 9

[Boris 'pi' Piwinger]

Sorry, for making myself not clear enough. Bug 136664 is about headers. I
(mainly) talk about the body.

pi

Comment 10

[Misha Kruk]

I think this bug and bug 104064 are the same. As I say in the 104064 discussion,
I feel that this is not enhancement, but rather a bug, because I expect that
after setting my send mail to KOI8-R I do not have to worry about anything when
sending emails which can be handled by KOI8-R. If there is a way to test if the
illegal characters are legal in the specified send mail encoding, it could be
used without any additional interaction with the user. Or at least this encoding
should be an options in the dialog. 

Comment 11

[Mike Cowperthwaite]

The duplicate proposes a specific detection in the case of replying to an ASCII message, as outlined in comment 0.  Note that since this bug was opened, a preference was added (with UI) to allow all replies to always use the 'default' charset instead of following the original message; and the error box that pops up now provides an option to send as UTF-8 (bug 194862).

*** This bug has been marked as a duplicate of 219036 ***

Comment 12

[Boris 'pi' Piwinger]

I don't think it is a dupe of the respective bug. That suggest a behavior to choose a charset independent of the characters in the message (i.e. charset detection as the summary says). So it would create misbehavior but not solve the problem of this bug.

pi

Comment 13

[Rolf Leggewie]

bug 410333 will almost solve this bug.  Except that many Japanese users don't have access to UTF-8-aware software (mobile phones are a good example, so are many popular freemailers in Japan).  This regression is being discussed in bug 448842.  There is also the problem about mail headers -> bug 369067

Comment 14

[Joshua Cranmer [:jcranmer]]

We silently fallback to UTF-8, except for ISO-2022-JP, where we silently mangle instead because Japan resists UTF-8 with a passion. On most locales, UTF-8 is the default, so you have to explicitly flip the charset if you don't want to use it, at which point you're hopefully cognizant of what that entails.

There's no need to fix this bug.