1285481 - Unencoded Finnish characters in Subject header incorrectly displayed in thread list

Status: RESOLVED DUPLICATE of bug 1023285

(Thunderbird :: Untriaged, defect)

Description

[KalleMP]

Attachment: thread view Subject encoding bug.png

User Agent: Mozilla/5.0 (Windows NT 6.3; rv:47.0) Gecko/20100101 Firefox/47.0
Build ID: 20160623154057

Steps to reproduce:

Receive mail from automated server via gMail IMAP with no header charset specified. 

Also made local copy in mail folder and symptoms persist.


Actual results:

Non ASCII characters in Subject 'åäöÅÄÖ' are displayed as a questio mark in black diamond in thread view pane.

Changing View/TextEncoding, View/CharacterSets or folder/Properties/FallbackTextEncoding does not help.

Subject characters are correctly displayed in message pane (unless messing around with View/TextEncoding settings which can break it temporarily).


Expected results:

If Message pane can display Subject correctly then Thread pane should also be able to do so.
Similar to bug 513472

Comment 1

[KalleMP]

Attachment: Notes I took while trying to figure this out

Comment 2

[Jorg K (CEST = GMT+2)]

In the subject header all non-ASCII characters must be RFC 2047 encoded.
Clearly this is not the case in the subject you reported:
Subject: Tervetuloa Tori-tilin käyttäjäksi

So the message is invalid. The thread pane still uses other system components to display the invalid subject better. This phenomenon has be reported various times.

See bug 1279804 comment #1.

Comment 3

[alta88]

hi jorg,

i think we need to note that there have been updates since 2047, most importantly 6532.  it would actually be a big move forward in reducing email encoding insanity to start testing for and accepting utf8. that said, i didn't check if this was actually emitted as utf8 or not.

https://tools.ietf.org/html/rfc6532#section-3.2

see also regressing bug 1023285 where a self described spec reader wasn't all that good at it.

(btw, many ++ for closing out junk bugs)

Comment 4

[Jorg K (CEST = GMT+2)]

So do we have a bug for implementing RFC 6532?
How would a message using UTF-8 content in headers be recognised? Sorry, I'm not familiar with RFC 6532.
The message given in bug 1279804 comment #0 didn't have anything special.

In this bug here, there was no message attached, I took the subject from the reporter's notes in attachment 8769072 [details].

Comment 5

[alta88]

Not sure, probably not. However, "other system components display the .. subject better".  As in the bug I referenced, which doesn't correctly display utf8 in the From field in messagepane headers, and which prior to jsmime did work. Meaning things that may have worked have been lost in the course of significant decoder and mime processing changes. This may be such a case, ie it may work in Tb31 pre jsmime.

The point though is that strictly speaking, non ascii no longer must be 2047 encoded for mail headers, per spec.  If a header string doesn't start with a 2047 =? or such a test, then it would be appropriate to treat it as utf8.  In all components.

Comment 6

[Jorg K (CEST = GMT+2)]

As I understand it, jsmime tightened the rules. Yes, I assume this worked "better" before jsmime arrived.

I think we should move the discussion to either bug 1146099 or bug 1023285.

Comment 7

[Jorg K (CEST = GMT+2)]

Attachment: Test message showing the UTF-8 in subject header already works

UTF-8 in the subject header already works.

The reporter's message (which he never enclosed) most likely contained another 8bit encoding, like ANSI/windows-1252.

Comment 8

[Jorg K (CEST = GMT+2)]

Attachment: Test message showing the UTF-8 in subject/from/to header already works

Actually, UTF-8 already also works in other headers, like From and To.

This is only shown correctly on the tread pane which uses jsmime.

The message header in the message pane don't show the correct content, since this is used the pre-jsmime function, see bug 1146099.

Comment 9

[Jorg K (CEST = GMT+2)]

Let's dupe it over to bug 1023285 which will be fixed soon ;-)