716983 - Forward message with vcard, wrong encoding (if second or later part in multipart/mixed has charset=utf-8 in Content-Type:, Forward window uses utf-8, but binary of iso-8859-2 in first text part is used as-is without conversion to utf-8)

Status: RESOLVED DUPLICATE of bug 1026989

(MailNews Core :: Composition, defect)

Description

[raal]

Attachment: test e-mail

User Agent: Mozilla/5.0 (Ubuntu; X11; Linux i686; rv:9.0.1) Gecko/20100101 Firefox/9.0.1
Build ID: 20111221202246

Steps to reproduce:

Preferences - Display - Formatting - Advanced 
Characters Encodings: incoming & outgoing mail - Central European (ISO-8859-2)
Steps to reproduce:
1 - set tb account settings to "Attach my vCard to messages"
2 - write an e-mail. Vcard is automaticaly added. Write some special ISO-8859-2 characters  -  ěščřž
3 - Go to Sent folder. E-mail is corectly readable.
4 - Forward this e-mail



Actual results:

Wrong encoding of text - UTF8, should be ISO-8859-2


Expected results:

Readable forwarded e-mail

Comment 1

[raal]

Attachment: printscreen of e-mail in sent folder

Comment 2

[raal]

Attachment: forwarded message

Comment 3

[WADA:World Anti-bad-Duping Agency]

(In reply to raal from comment #0)
> Created attachment 587419 [details]
> test e-mail

Mail structure.
> Content-Type: multipart/mixed;
> Content-Type: text/plain; charset=ISO-8859-2; format=flowed
> Content-Type: text/x-vcard; charset=utf-8;
Message body glyph for string in text/plain part
    Viewed with ISO-8859-2   : křen
    Viewed with iso-8859-1   : køen
    Viewed with windows-1252 : køen
    Viewed with utf-8        : k�en
Glyph for vcard data of =C5=A0karek Ji=C5=99=C3=AD in quoted-printable.
    Viewed with ISO-8859-2   : Ĺ karek JiĹĂ­
    Viewed with iso-8859-1   : Škarek Jiří
    Viewed with windows-1252 : Škarek Jiří
    Viewed with utf-8        : Škarek Jiří

(In reply to raal from comment #1)
> Created attachment 587420 [details]
> printscreen of e-mail in sent folder

Glyph of message body : křen => shown as iso-8850-2 or equivallent one, as expected.

(In reply to raal from comment #2)
> Created attachment 587422 [details]
> forwarded message

Glyph of message body in Forward window: køen

Next is not included in screen shot, but seen with Tb 9.0.1.
  Character encoding shown in title bar=UTF-8.
  (if different from default charset for composition, char set is shown)
  View/Character Encoding also shows UTF-8.

When the forward mail is saved in Outbox by Send Later, with mail.strictly_mime=true.
> Content-Type: text/plain; charset=UTF-8; format=flowed
> Content-Transfer-Encoding: quoted-printable
> k=C3=B8en

Hexa decimal data relevant to phenomenon.
>        iso-8859-2 unicode / utf-8
> ŕ    : 0xF8       Unicode Character 'LATIN SMALL LETTER R WITH CARON' (U+0159)
>                   UTF-8 (hex)	 0xC5 0x99 (c599)
>
>        iso-8859-1 unicode / utf-8
> 0xF8 : ø          Unicode Character 'LATIN SMALL LETTER O WITH STROKE' (U+00F8)
>                   UTF-8 (hex)  0xC3 0xB8 (c3b8)

(A) When mail is viewed, text/plain part(mail body part, iso-8859-2 in your case) is shown correctly with charset specified in Content-Type:.
However, View/Character Encoding shows charset in last & proper Content-Type: xxx/yyy; charset=abc-def-ghi(utf-8 in your cse).
(B) Tb uses UTF-8 in composition window of forward mail, even though charset of mail body is iso-8859-2. Tb uses charset of second part for Vcard, utf-8, which is shown by View/Character Encoding in (A).
(C) When Tb uses utf-8 as charset of mail composition, Tb doesn't correctly convert 0xF8 in text/plain; charset=iso-8859-2 part to Unicode.
Tb uses 0xF8 of iso-8859-2 in original mail body as-is in composition window
, and interprets the 0xF8 of iso-8859-2 as U+00F8, and convert it to 0xC3B8 of UTF-8.

(A) and (B) is same phenomenon as bug 597821. However, that bug is for incorrectly generated mail due to issue in AVG(perhaps bug of AVG), no one perhaps currently cares for.
(B) is found in bug 505172.

Confirming.

Comment 4

[WADA:World Anti-bad-Duping Agency]

Because bug 597821 is for wrongly added part case, and because bug 505172 is for incorrect charset selection by Tb, and because "selection of utf-8 as composition charset" in this bug's case can't be called "Incorrect", I think this bug, for normal mail case, is better keep open, unless already opened bug for this problem exists.
Setting dependency of these bugs to this bug.

Comment 5

[raal]

Regression. Tested with old TB 2.0.0.4 - forwarding of "test e-mail.eml " is correct.

Comment 8

[Christian Fertig]

It's happens not only, when forewarding a vcard.

I've got the problem here with a multipart message created by "X-Mailer: iPad Mail (10B141)"

--Apple-Mail-E03CC6E5[...]
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

quoted-printable encoded text with umlauts: e.g. Pr=C3=BCfung
--Apple-Mail-E03CC6E5[...]
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

Von meinem iPad gesendet
--Apple-Mail-E03CC6E5[...]--

Thunderbird uses the encoding of the last message part when forwarding or "edit as new". Which is wrong. It should take the encoding which matches the message part. So forwarding the above message results in broken Umlaut characters.
If I manually delete the 2nd part of the message (either with header tools or directly on my imap server), then the forward encoding is correct.

It's a little bit like described here: http://wanwisau.blogspot.de/2010/01/thunderbird-font-encoding-changed-when.html
(they have a problem with Thai, I think there was a bug here in bugzilla too, but I don't find it again)

I've tested with the following thunderbird versions on windows 7 (I used auto update to skip from version to version)

thunderbird 2.0.22     ok
thunderbird 2.0.24     ok
thunderbird 3.1.10     ok
thunderbird 3.1.20     ok
thunderbird 12.0.1     broken
thunderbird 17.0.1     broken

Comment 10

[Bruno Guerreiro]

Hello.
I've found this issue to be present also on 17.0.6. with attachments non-vCard, eg. PDF, 7z

Comment 11

[Vasilis Lourdas]

This also affects me (Thunderbird 24, Windows and 17.0.9, Linux).

Comment 14

[WADA:World Anti-bad-Duping Agency]

Root cause of "charset is pcked up from last attachment" was resolved by bug 1026989 at last.

Some valuable bug reports from users were hidden/lost/ignored by "duping to bug 716983, then duping bug 716983 to other bug".
Unfortunately, or fotunately, only "garbled display in Forward Inline case" was resolved by unknown bug while the other bug was being processed, so nothing was resolved by the other bug.

Duping this bug to bug 1026989, for ease of tracking, search, understanding current status of problem.

Comment 15

[WADA:World Anti-bad-Duping Agency]

FYI.
Patch of bug 1026989, one liner patch, was proposed in the other bug to which bug 716983(and bug 701818 too) had been duped to. The patch was dig out from the other bug by bug 1026989, and long lived "charset is picked up from last attachment" issue has been resolved by bug 1026989.

Why such "digging out hidden important patch from the other bug" should be needed in B.M.O?