271508 - thunderbird forces the FROM and the BODY charset to be the same

Status: NEW

(MailNews Core :: Composition, enhancement)

Description

[u162355]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.5) Gecko/20041118 Firefox/1.0
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.5) Gecko/20041118 Firefox/1.0

my "Your Name" is Gábor Farkas.
when i send a mail, it usually becomes:

From: =?UTF-8?B?R8OhYm9yIEZhcmthcw==?= <[e-mail]>

that's ok.

but, when i want to send a japanese mail..or better to say a mail in ISO-2022-JP
encoding, thunderbird refuses to send it,
and complains that 

"The message you composed contains characters not found in the selected
character encoding"

in a way he is correct, because my from-name (Gábor Farkas) contains "á",
which is not available in ISO-2022-JP.

but that should not be a problem.

the FROM header does not have to have the same encoding as the body.

an exampel from rfc2047:

"

   From: Nathaniel Borenstein <[e-mail]>
         (=?iso-8859-8?b?7eXs+SDv4SDp7Oj08A==?=)
   To: Greg Vaudreuil <[e-mail]>, Ned Freed
      <[e-mail]>, Keith Moore <[e-mail]>
   Subject: Test of new header generator
   MIME-Version: 1.0
   Content-type: text/plain; charset=ISO-8859-1
"

as you see, the FROM is in iso-8859-8, and the body is in iso-8859-1.

thunderbird should just use for example UTF-8 for the FROM and whatever encoding
was specified for the body.

or at least allow me to have the FROM in a different encoding.

btw. if i change my YourName to not have accented characters, then it works.

Reproducible: Always
Steps to Reproduce:
1. specify your name (the "Your Name" field in the settings) to have an "á"
2. compose a mail with only ascii characters (even an empty mail is fine)
3. set the character encoding for the mail to ISO-2022-JP
4. try to send it

Actual Results:  
thunderbird refused to send it with a dialog saying:

"The message you composed contains characters not found in the selected
character encoding"



Expected Results:  
either:
simply use a suitable charset for the from-header, and use the specified charset
for the body

or:
the same as the previous one, but first display a warning.

Comment 1

[Jean-Marc Desperrier]

There's a very old related bug where Mozilla Mail/TB will not handle properly a
message where the encoding of the subject is not the same as the one of the body
(they will display properly, but get it wrong when you try to reply to the message).

It's not great to use one encoding for the header and another for the body,
because the recipient has to be able to decode both to handle properly the
message => That makes all UTF-8 better.

Of course, what makes the problem worse is the fact the dialog doesn't say what
character is the culprit and the user might not suspect it's his from and not
anything in the body (and I don't know if we have an open bug on that).

Comment 2

[u162355]

i understand that it's better to use utf-8 everywhere.

and i do use utf-8 everywhere. ...where i can.

this problem actually happened when posting to the sci.lang.japan newsgroup,
where the only-allowed encoding is iso-2022-jp.

so everytime when i want to post to that newsgroup i have to change my name from 
"Gábor Farkas" to "Gabor Farkas".

very annoying :)

hmmm..now that i'm thinking about it..
as a workaround i could set up my name directly in that header-encoded for.
so i could set up my name as "=?UTF-8?B?R8OhYm9yIEZhcmthcw==?=".
ugly as hell, but theoretically it could work. will have to check it out ;)

Comment 3

[u162355]

yes, it works ;)

but as i wrote, it's ugly.

is this thing so hard to be fixed?

maybe the problem is that the thunderbird source code is written in such a way
that it's only able to cope with one encoding per mail?

Comment 4

[David :Bienvenu]

Jungshik can probably give a definitive answer here..

Comment 5

[Mike Cowperthwaite]

Reproduced with TB 1.0, Win2K.

Comment 6

[Mike Cowperthwaite]

Hm.  With TB 1.0+0302, which has the new "Send Anyway" feature when you're 
warned about characters not matching, the From is *still* tagged (per RFC-2047) 
as the body's encoding.

I have an old identity I used from testing another bug that has a real name of 
"bokförlaget".  I tried using that identity send an ISO-2022-JP message (text 
copied from some spam).  The From: was sent as:
  From: =?ISO-2022-JP?B?Ym9rZm8icmxhZ2V0?= <[e-mail]>
which converts to (as ASCII):
  From: bokfo"rlaget <[e-mail]>

(That double-quote shouldn't be a problem, but because Mozilla mishandles 2047 
text, it results in the display of the name with an additional, closing double-
quote character.)

Comment 7

[Jungshik Shin]

The original report is not a bug per se. at most 'enhancement' request if not
WONTFIX. Mike's comment seems to be about the display problem, which needs to be
filed separately. 

Comment 8

[u162355]

excuse me, but why isn't this a bug?

there can be 2 cases:
1. thunderbird does not handle non-ascii FROM headers.
     -in this case thunderbird should DISALLOW the setting of non-ascii
from-addresses

2. thunderbird does handle non-ascii FROM headers.
     -in this case, thunderbird does not implement it correctly. it is a bug.


imho this is case #2.

Comment 9

[Jungshik Shin]

(In reply to comment #8)
> excuse me, but why isn't this a bug?

> 2. thunderbird does handle non-ascii FROM headers.
>      -in this case, thunderbird does not implement it correctly. it is a bug.

  Name me just one mail program that allows you to use different MIME charsets
in the message header and the message body when you compose a *fresh new*
message. Anyway, what do we do incorrectly? Not allowing you to use different
mime charsets for message header and message body is NOT doing it incorrectly.
It's a feature that would be nice if we offer (which is why it's categorized as
'enhancement'), but its priority is very low. 

Comment 10

[u162355]

(intro: just to make sure everyone understands..thunderbird-is-a-great-program,
i-love-it-and-use-it, only-reporting-here-to-make-it-better and all these
prefixes apply ;)

>Name me just one mail program that allows you to use different MIME charsets
>in the message header and the message body when you compose a *fresh new*
>message. 

does this mean that if every other mail client sux, tbird should suck too?

>Anyway, what do we do incorrectly? Not allowing you to use different
>mime charsets for message header and message body is NOT doing it incorrectly.

afaik we are talking about rfc2047 here..
if you check my original report, i showed an excerpt from the rfc:

"

   From: Nathaniel Borenstein <[e-mail]>
         (=?iso-8859-8?b?7eXs+SDv4SDp7Oj08A==?=)
   To: Greg Vaudreuil <[e-mail]>, Ned Freed
      <[e-mail]>, Keith Moore <[e-mail]>
   Subject: Test of new header generator
   MIME-Version: 1.0
   Content-type: text/plain; charset=ISO-8859-1
"

as you see, the FROM is in iso-8859-8, and the body is in iso-8859-1.


in my interpretation you do not support this part of the rfc. so your
implementation is wrong. you are doing something wrong. i don't know if
thunderbird is really unable to send a mail with differently encoded parts, or
it's just a "gui" problem..

and let me please return to the first sentence..



>Name me just one mail program that allows you to use different MIME charsets
>in the message header and the message body when you compose a *fresh new*
>message. 

i think i clearly demonstrated why is it needed for me to send a *fresh new*
mail with different charset in the body and in the headers...


summary:
1. thunderbird does not behave the way he should (by rfc2047)
2. i demonstrated that it is not just a theoretical question, because i really
need it when using thunderbird.

your turn :) 

Comment 11

[Jungshik Shin]

> 1. thunderbird does not behave the way he should (by rfc2047)

I know RFC 2047 from the top to the bottom. You don't. 

Comment 12

[u162355]

>> 1. thunderbird does not behave the way he should (by rfc2047)
>I know RFC 2047 from the top to the bottom. You don't. 
you're right.

i did not read the whole rfc2047..well..maybe i did but i don't remember
exactly.. so please tell me why is my interpretation  (that it IS ALLOWED to
have a different encoding in the FROM-header and in the body) wrong.

i do not want to fight here.. i only want to be able to send mails...

so please explain to me why do you think that thunderbird's behaviour is correct.
thanks

Comment 13

[Mike Cowperthwaite]

Jungshik, what about the conversion in the header of the o-with-umlaut to 
ASCII-o plus double-quote  (ö => o")  when Send Anyway is selected?  Is that 
considered correct?  It seems more than just a display issue -- altho I agree, 
it's not *this* bug.


Gábor, you've discovered a workaround already -- enter the 2047/UTF-8 string as 
the identity's name.  But if the group you're posting to only allows 2022-JP, 
that implies that at least some of the people reading those messages will be 
using clients that cannot display the problem character anyway.  What you want 
seems to be in conflict not with Mozilla but with the group's self-imposed 
refusal to use Unicode.  How about simply using "G. Farkas" as your name for 
that group?

Comment 14

[Jungshik Shin]

I'm sorry I shouldn't have pressed 'commit' button so rashily. I was annoyed a
little bit too much by your assertion that we're not compliant to RFC 2047.
Nonetheless, I should have given you more detailed explanation.

Your interpretation that it is ALLOWED to .... is absolutely correct. (that's
one of intents of RFC 2047 authors) Therefore, we handle those cases very well
when rendering emails received. (MS OE doesn't as you already know.)  However,
your assertion that Mozilla-mail is NOT compliant to RFC 2047 by not allowing
you to have different MIME charsets in the message header and the message body
is a different story. It'd be nice to offer that feature/enhancement to advanced
multilingual users like you. However, not doing so doesn't make us incompliant
to RFC 2047. Given this, I don't regard this as a high priority item. It may not
as complicated as I initially thought to implement this. Still, it's not high in
my queue.

Besides, as Mike pointed out, in your scenario, even if we offer that feature,
your recipient is not likely to get your name rendered correctly. (if their mail
client is so good at RFC 2047 handling as we're, it'd be able to handle UTF-8 as
well.)

As Mike found out, if you just go ahead with 'Send anyway', you get your name
transliterated. That is, we have a second line of 'defense' (yes, Mike, that's
what we intended). However, Mike also found some problems with '"' when letters
like o-umlaut and u-umlaut are transliterated. I have to take a look at it
perhaps in another bug.

Comment 15

[Mike Cowperthwaite]

*** Bug 310514 has been marked as a duplicate of this bug. ***

Comment 16

[Mike Cowperthwaite]

*** Bug 312540 has been marked as a duplicate of this bug. ***