542181 - Thunderbird strips characters not fitting into chosen character set

Status: NEW

(Thunderbird :: Message Compose Window, defect)

Description

[Rimas Kudelis [use needinfo to ping]]

When sending a multipart (HTML and plain text) message, Thunderbird doesn't switch to UTF-8 upon encountering a character that does not fit into the chosen character set. Instead, it encodes all unfitting characters using HTML entities in HTML part, and uses ugly decomposition (e.g. letter 'ū' becomes 'u-') for the plain text part. This decomposition is not really acceptable, TB should use UTF-8 at least for the plain text part in such case.

I've personally seen this bug manifesting in TB3.0.1/mac and 3.0/Win.

To suggest a move even more radical, perhaps this letter decomposition code should be totally got rid of?

I guess this also affects SeaMonkey, but haven't tested.

To reproduce:
compose and send yourself a message in ISO-8859-1 and use some formatting as well as some accented characters not in ISO-8859-1 (e.g. ą č ę ė į š ų ū ž) in it. Check its source, and you'll see the problem manifesting.

Comment 1

[Rimas Kudelis [use needinfo to ping]]

Note: the problem seems to only happen when the Subject (and maybe other headers) doesn't contain any accented characters. Otherwise Thunderbird still chooses UTF-8.

Comment 2

[Dan Mosedale (:dmosedale, :dmose)]

Agreed that your proposed change sounds like a good one.  I wonder if it's also going to entrain some preference structure changes and overall design thought as well.

Given that 3.1 is mostly about blockers that would prevent Tb2 users from upgrading, I don't think we'd hold for this unless it were a regression.  Marking as blocking-, wanted+ for that reason.

If it is indeed a regression, feel free to renominate.

1.8.1.x is seeing very little action these days; this bug isn't severe enough that we'd take a fix for it there.

We might or might not take it on 3.0.x; if someone writes a patch with tests and feels that it's an appropriate amount of risk without string changes for that branch, feel free to nominate the patch for approval for that branch.

Comment 3

[Rimas Kudelis [use needinfo to ping]]

This is still the case with Thunderbird 10