1358463 - wrong character encoding in subject

Status: RESOLVED INVALID

(Thunderbird :: Untriaged, defect)

Description

[hartnegg]

User Agent: Mozilla/5.0 (Windows NT 6.1; rv:52.0) Gecko/20100101 Firefox/52.0
Build ID: 20170316213829

Steps to reproduce:

Look at email


Actual results:

Thunderbird 52 shows one word in a subject line as "Beitr�ge"



Expected results:

The word should be shown as "Beiträge", like Thunderbird 45.8 did.

Comment 1

[hartnegg]

The change is both in the pane which lists all emails, and in the pane which lists the selected email.

It does not happen with all emails with German umlauts. In the ones where it does not happen, the source view shows that the subject line starts with =?UTF-8?

The error in the view of the current email goes away if I select View, Text Encoding, Middle European (Windows). But the error in the pane with the list of emails remains.

Comment 2

[Jorg K (CEST = GMT+2)]

Please view the message source and paste the entire subject line here. And also the message encoding. Or attach the entire message.

I suspect that the subject line is invalid, that is they used an "ä", but not raw UTF-8 encoded (which is permissible according to https://tools.ietf.org/html/rfc6532#section-3.2) but as ANSI/windows-1252.

For "Beiträge" I'd expect:
Subject: =?UTF-8?B?QmVpdHLDpGdl?= (binary) or
Subject: =?UTF-8?Q?Beitr=C3=A4ge?= (quoted)
but then you said, those with ?UTF-8? work as they should.

We've made changes from 45.8 to 52 in this field, so some invalid encoded headers now look invalid like you've indicated.

Comment 3

[hartnegg]

Attachment: mailheader.txt

header of he email, slightly edited to remove IDs and shorten it

Comment 4

[hartnegg]

The sender uses a perl script, and assumes that all recipients use windows.

However Thunderbird behaves at least inconsistent:
If I select View, Text Encoding, Middle European (Windows), this corrects the subject line in the pane with the email contents, but not in the pane with the list of incoming emails.

Comment 5

[Jorg K (CEST = GMT+2)]

As I suspected, the file you uploaded is ANSI encoded with a raw header of:
Subject: NDR Visite: Beiträge und Videos vom 25. April 2017

That is invalid. Header with non-ASCII characters must be RFC2047 encoded but raw UTF-8 is also permissible according to RFC6532.

I encoded the message in UTF-8 and you can see that it works.

Please contact the NDR and tell them not to waste the money payers of the Rundfunkgebühr by sending invalid messages.

Comment 6

[Jorg K (CEST = GMT+2)]

Attachment: mailheader.eml - NDR message UTF-8 encoded and working

Comment 7

[Jorg K (CEST = GMT+2)]

Working on bug 1394244 I noticed that you can get the invalid header to display correctly if you set the fallback character set of the folder to "Western". Maybe that's a help, but the NDR shouldn't send such messages in the first place.