194862 - Incorrect encoding warning dialog should be more user friendly (maybe add button 'Use UNICODE' ?)

Status: RESOLVED FIXED

(MailNews Core :: Composition, defect)

Description

[Eglė K]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1) Gecko/20030114 Debian/1.2.1-9
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1) Gecko/20030114 Debian/1.2.1-9

Lots of my friends use Mozilla Mail, but they still send mail with incorrect
encoding :(

Problem is that incorrect encoding warning dialog is not user friendly - it
isn't intuitive and users don't know how to choose correct encoding and also
can't choose encoding directly from warning dialog.

The simplest (and the best in my opinion) solution would be adding button "Use
UNICODE" to the incorrect encoding warning dialog. I think users and programers
should use unicode (UTF8) more intensively (for example in Opera 7 mail program
are no possibility to select encoding - all mail with non iso-8859-1 is send as
UTF8)

More advanced solution is to add list of suitable encodings to the incorrect
encoding warning dialog. (suitable encodings list is made from all encodings
list thowing away encodings not containing symbols found in email)

Reproducible: Always

Steps to Reproduce:
1. Press CTRL+M in mozilla
2. Choose ISO-8859-1 encoding and write some characters, that aren't in this
encoding (for example àèæëáðøûþ ÀÈÆËÁÐØÛÞ)
3. Press "Send" button

Actual Results:  
Incorrect encoding warning dialog without instructions how to select correct
encoding and without posibillity to choose correct encoding is displayed.

Expected Results:  
Incorrect encoding warning dialog with posibillity to choose correct encoding in
that dialog (for example UTF8 always is correct) or at least instructions how to
select correct encoding should be displayed.

If you don't have time to solve this problem quickly - at least add instructions
 in incorrect encoding warning dialog how to select correct encoding (for
example: "Select View->Character Coding->Unicode(UTF8)").

I think this bug should be solved (at least partially) ASAP. Maybe in Mozilla
1.3 final version instructions can be added and in Mozilla 1.4a posibillity to
choose correct encoding in the dialog in incorrect encoding warning dialog can
be added ?

Comment 1

[Ramūnas Lukaševičius]

Attachment: Corrects this bug.

I wrote patch for this bug. Please test in and include to mozilla.
I changed the OK button. Now it automatically changes message encoding to the
UTF-8 and sends message - it will be always correct.
You do not need to think which encoding to use. A lot of people do not know
even what is it "encoding" and press OK and send message with incorrect
encoding. Now it will be allways correct.

Comment 2

[Eglė K]

It's hard to include 2 lines patch ?

Comment 3

[Jungshik Shin]

It's not hard, but you have to ask for review and superreview. (please, read
http://www.mozilla.org/hacking) Besides, people working on Mozilla-mail was not
on the CC list. I like the patch, but some people may want to have 

1. Convert to UTF-8 and send
2. Send any way
3. return and choose a different encoding

BTW, this may be a dupe.

Comment 4

[David :Bienvenu]

I don't have a strong opinion - whatever you think is good, Jungshik. If you
want a r/sr for this, just let me know.

Comment 5

[Katsuhiko Momoi]

We seem to be on an evolutionary path. Referring to jungshik's options, we had
up to now 2 & 3. The proposal here is to add 1 and leave 3 as is. I think we
shoul try next 1 & 3 without 2. I don't see much advantage in leaving option #2
in. Sometimes it will end up creating unreadable msgs. Other times, it will work
OK. That's not a kind thing.

The warning language needs a bit of editing. I'm reviewing it now and will make
suggestions later. 

Comment 6

[Katsuhiko Momoi]

I would like to suggest the following rather than the proposed
wording in the patch in comment #1.

"The message you composed contains characters not found in the selected
Character Coding. While you can choose a different Character Coding, it is
always safe to use Unicode for mail. To send or save it as Unicode (UTF-8),
click OK. To return to the Composer window where you can choose a different
Character Coding, click Cancel."

Two changes from the patch:

1. Use the uppercase "U" for Unicode.
2. Place the adverb "always" in a better position and build a slightly more
explanatory sentence.

Comment 7

[Jungshik Shin]

Thanks, David and Kats for comment and help.

There a couple of remaining issues. Stupid web mail services such as Hotmail,
Yahoo mail and many country-specific ones (and open source web mail programs
widely deployed at schools and elsewhere these days) don't support UTF-8 well
(how about AOL mail?) I haven't checked Hotmail and Yahoo mail recently, but I'm
pretty sure that in the message display pane of those stupid web mail services,
subjects and senders in UTF-8 (RFC 2047 encoded or not) would be garbled. They
made a classical mistake of tightly binding UI languages to character encodings.
that is, if you choose English as your UI language, they assume that you'd be
content with ISO-8859-1/Windows-1252. In case of the message body, it'd be
initially garbled, but I guess either that there's a button/menu ('this message
is 'foreign language' or 'different encoding'. to read it, press this button) or
that end-users have to change the character encoding manually.  Given this, I
think  we have to 'reword'  'always work' part slightly. 

 The other problem is that if the message format is text/html, it's possible to
include any Unicode characters in the message body (text/html part) in any
character encoding  and Mozilla is actually able to do that with NCRs. (however,
characters in the message headers are turned into question marks. Here, RFC 2047
encoding with UTF-8 can be used.). In addition, stupid web mail services work
well in this case. 

So, it's a bit more complicated. How about this (assuming that it's not much
work. I have to take a look to see how it can be done)

A. text/plain: offer 1 and 3

B. text/html : offer 1, 2 and 3 

C.  multipart/alternative (text/plain + text/html) : I'm not sure. 

In B-2, characters in headers would be garbled so that the warning (as it is) is
 valid. 
 
 

Comment 8

[Jungshik Shin]

*** Bug 104064 has been marked as a duplicate of this bug. ***

Comment 9

[Jungshik Shin]

Attachment: an alternative

I implemented what I wrote (for text/plain and multipart/alternative, it's the
same as now while for text/html, three options are offered), but I don't like
it much because nsIPromptService::Select() turned out not to be what I wanted.
Either I have to write an xul/js file (a la askSendFormat.xul/js) or just have
to go with attachment 135553 [details] [diff] [review] plus rewording by Kats (with 'always' replaced by
'in most cases' :-))

Comment 10

[Katsuhiko Momoi]

jungshik, you're trying to spec Mozilla mail behvaior against 
a moving target. Sure, some of the web mail may not be able to handle
UTF-8. But if you send the msg as is, no web mail will be able to 
handle the headers because they are encoded incorrectly. Maybe the body
can be displayed. 

My point is that we are not going to get completely correct display
unless both headers and body are labeled correctly. UTF-8 offers
the only sane choice. Also I say it's a moving target because
(I believe) most web mail will be improving. A quick check shows that
you can display UTF-8 mail (headers & body) with English version of
Yahoo web mail. Usually the English version is ahead of country specific
versions. I would think it would be a matter of time before country specific
Yahoo mail implement the same ability with UTF-8 mail. 

With hotmail, you can view UTF-8 mail headers and body with the encoding
set to UTF-8. I used Japanese version of hotmail and lost the UI language
because it's set to Shift_JIS but UTF-8 header and body became readable 
when I switched the encoding.

I suspect that webmail will be changing. Hopefully, Mozilla's decision to
go with UTF-8 as the deafult in this case should help expedite that process.
There are already other mailers such as Outlook Express, AOL Communicator, AOL
Mail that send UTF-8 when the characters input don't match the default
encoding of the client. Mozilla's joining this rank will further accelerate the
trend and that is the right thing to do at this point.

For these reasons, I vote for doing only 1 and 3 and making it easier for
average users.

Comment 11

[Jungshik Shin]

OK. I'm sold :-) partly because I don't like the idea of having to write two
files (xul and js) to implement what I wrote in comment #7 (the selection box
generated by nsIPromptService::Select() looks akward on Windows). However, I
think we have to replace 'always' with 'in most cases' :-) simply because it's
not the case.

Good to hear that things are getting better in Yahoo-mail, but it's unfortunate
that Hotmail hasn't changed a bit since the last time I checked (needless to
say, changing the encoding of the browser to UTF-8 always worked with the UI
part garbled). It's frustrating that they're slow in making changes (see
http://bugs.horde.org/show_bug.cgi?id=1052 A couple of years ago, I patched
Horde IMP and put it up at my computer and  offered it to my friends as an
alternative to my school's web-mail interface which doesn't know anything other
than Latin-1.) 

I'll upload a patch in a moment (virtually identical to attachment 135553 [details] [diff] [review] except
for word-smithing thanks to you).

Comment 12

[Katsuhiko Momoi]

In comment #11, jungshik says:

"OK. I'm sold :-) ... However, I think we have to replace 'always' with 
'in most cases' :-) simply because it's not the case."

Thanks. I would like to suggest the following wording:

"The message you composed contains characters not found in the selected
Character Coding. While you can choose a different Character Coding, it is
usually safe to use Unicode for mail. To send or save it as Unicode (UTF-8),
click OK. To return to the Composer window where you can choose a different
Character Coding, click Cancel."

For UI language, I think "usually" sits better than statistical sounding
"in most cases".

Comment 13

[Jungshik Shin]

Attachment: patch

patch as I promised (with 'usually')

Comment 14

[Jungshik Shin]

Attachment: update

It turned out that we should not ignore fallbackCharset because there's a pref.
entry for 'promoting' charset (i.e. ISO-8859-1 -> Windows-1252). See

http://lxr.mozilla.org/mozilla/source/modules/libpref/src/init/all.js#707
http://lxr.mozilla.org/mozilla/source/mailnews/compose/src/nsMsgCompose.cpp#4780

(and follow the code path from there)

The dialog only comes up if even the fallback charset (Windows-1252 for
ISO-8859-1) fails to cover characters in the message. Only in that case, we
have to use 'UTF-8'. 

In this patch, I also included the patch for thunderbird.

Comment 15

[Jungshik Shin]

Comment on attachment 138104 [details] [diff] [review]
update

asking for r/sr.

I just tried including U+2018 and U+2019 (not covered by ISO-8859-1 but by
Windows-1252) with ISO-8859-1 and Mozilla properly 'fell back' to Windows-1252.
When I included a Korean character, the dialog box came up as it should and
pressing OK worked as intended.

Comment 16

[Jungshik Shin]

Thank you all.
fix checked into the trunk.

Comment 17

[Jungshik Shin]

sorry for bugspam.
David, do you think I have to ask for a1.6 as well? It's a low risk item, but
drivers may think it's too late in the 1.6 cycle.

Comment 18

[Christian :Biesinger (don't email me, ping me on IRC)]

"To send or save it as Unicode (UTF-8), click OK. To return to the Composer
window where you can choose a different Character Coding, click Cancel."

why not just label the buttons "Send as UTF-8" and "Return to compose window"?

Comment 19

[David :Bienvenu]

it's fine to ask for 1.6 approval - I'll ask Seth.

Comment 20

[(not reading, please use [e-mail] instead)]

Comment on attachment 138104 [details] [diff] [review]
update

a=sspitzer

Comment 21

[Jungshik Shin]

fix checked into 1.6branch.
re: comment #18. that's a good idea. let's deal with it for 1.7alpha.

Comment 22

[Isaac Hwak Han]

*** Bug 209200 has been marked as a duplicate of this bug. ***

Comment 23

[Isaac Hwak Han]

*** Bug 204735 has been marked as a duplicate of this bug. ***

Comment 24

[Birkir Barkarson]

I have a problem with this solution.
I come across a situation where the encoding selection dialog come up and
reports that my message contains characters not in the selected encoding.  In my
situation the message does not contain such characters but perhaps some of the
characters are mis-identified (perhaps they are already unicode encoded).  
My problem is mostly that this prevents me from sending a mail in the encoding
that I need to send.  In this case I am trying to send a mail in shift_jis
encoding but the program give me only the option of sending in unicode or change
the encoding (which is already correctly set and should contain only characters
in that encoding).  I have been using Outlook Express but I wanted to change to
Thunderbird, for many reason one of which is that it correctly displays sender
names and subject in mail with encodings outside the local codapage.  In a
similar situation Outlook offers the choice of sending as is (perhaps scrambling
some characters), send as utf8 or cancel.  This lack of choice (along with the
apparent mis-identification of letters in the mail) renders Thunderbird useless
to me.  The reason for this is that I have to send a lot of mail to mobile
phones and such in Japan, most (or all) of which don't support utf8 encodings
for CJK characters.

Hope there is someway to resolve this issue.

Comment 25

[AZUMA Takao]

(In reply to comment #24)
I have the same problem, too.
In my case, the receiver is Lotus Notes via its SMTP gateway, this also doesn't
support CJK with UTF-8 encoding.

I just wanted to forward mail from Swedish people to my colleague with my
Japanese comment.
Previously I've used Netscape 7.1, and this works fine for me, selecting
ISO-2022-JP encoding and send it, warning, but send it anyway, diacritical
marked characters in the original mail are converted properly as I expected.
But now, I'm using Mozilla 1.7.2, I have to manually retype all diacritical
marked characters before choosing ISO-2022-JP before sending !

Please bring back the option "2. Send anyway".

Comment 26

[Tsuyoshi GEMMA]

(In reply to comment #24)
I also have the same trouble. The situation is as follows; I wanted to forward
the mail in English to my friends with my comments in Japanese (ISO-2022-JP
encoding). When I tried to send it, warning dialog about encoding appeared and
my mail was sent in UTF-8 encoding with no choice. After that, a friend of mine
(Yahoo web mail user) complained that he could not read my message at all. 

In Japan, a vast number of mail client does not support UTF-8 except for very
few clients developed outside Japan. Thus, current solution is quite problematic
and harmful. Enforcing usage of UTF-8 is a dogmatic solution IMHO, at least, as
of writing this message.

As AZUMA said, I hope that Mozilla mail (and Thunderbird) behaves again like
Netscape 7.1; I need the choice (sending "as is" or "as UTF-8" or "cancel").       

Comment 27

[Jungshik Shin]

Please, add your comment to bug 249530 (and add yourself to Cc instead just
adding comment and driving away - drive-in comment) if you believe Mozilla needs
to offer 'send as is'.  This bug was resolved as fixed a long time ago !!

> In Japan, a vast number of mail client does not support 
> UTF-8 except for very few clients developed outside Japan.

  It seems like you're talking in terms of email clients supporting or not
supporting UTF-8.  How about the  'market' share of email clients supporting
UTF-8?  

Comment 28

[Mike Cowperthwaite]

*** Bug 229209 has been marked as a duplicate of this bug. ***