1066916 - character encoding keeps changing to 'Western instead of UTF-8

Status: RESOLVED WORKSFORME

(Thunderbird :: Untriaged, defect)

Description

[Don Moldover]

Attachment: TBird_options.JPG

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0
Build ID: 20140716183446

Steps to reproduce:

Tools--> Options--> Display (Formatting)--> Advanced (see attached TBird_options.jpg) --> OK




Actual results:

Then went to preview an email in my inbox and found actual text had funny characters appearing inline and when I tried View --> Character Encoding ...I found that the 'dot' was next to 'Western'


Expected results:

The email should not have had funny characters and the Character Encoding should have been set to 'Unicode'

Comment 1

[Don Moldover]

This encoding problem does not appear in all emails in my inbox but whenever I select View --> Character Encoding I find the 'dot' next to 'Western'

Comment 2

[Magnus Melin [:mkmelin]]

The default encoding for incoming is only used if the message doesn't declare a charset. 
View Source and check the "Content-Type:" header. What does it say?

Comment 3

[Don Moldover]

(In reply to Magnus Melin from comment #2)
> The default encoding for incoming is only used if the message doesn't
> declare a charset. 
> View Source and check the "Content-Type:" header. What does it say?

Here is what View Source says...
Content-Type: multipart/alternative; boundary=001a113340ca507b910503ae447e

--001a113340ca507b910503ae447e
Content-Type: text/plain; charset=UTF-8

Comment 4

[Don Moldover]

I would have replied sooner but the email address you had on file was outdated

Comment 5

[Magnus Melin [:mkmelin]]

Could you attach the whole mail as .eml? (Edit out personal details if needed.)

Comment 6

[Don Moldover]

Attachment: A recent email that exhibits this behavior

This email is an example, as recently requested, of one which appears in my preview window (and would appear the same way if I did a reply) with funny characters in it.  My character encoding (under the View menu) shows itself as 'Western' while my default option is set to UTF-8.  There are more than a few other emails that show up with no funny characters; however, whenever I check the View menu, it shows the character encoding selected is Western

Comment 7

[Magnus Melin [:mkmelin]]

Comment on attachment 8493725 [details]
A recent email that exhibits this behavior

Please don't set flags you don't know what they are

Comment 8

[Magnus Melin [:mkmelin]]

That message shows as Unicode for me.

Comment 9

[Don Moldover]

Attachment: Thunderbird options that I have set (captured by snipping tool)

Comment 10

[Don Moldover]

I'm glad...once I 'reset' the 'View' menu to UTF-8 it shows as Unicode for me too!  The problem is that TB keeps changing the 'View' option on its own!!!!!  See the attached snip for my current Display Options(In reply to Magnus Melin from comment #8)
> That message shows as Unicode for me.

Comment 11

[Magnus Melin [:mkmelin]]

Is View | Character Encoding | Auto-detect set to off?

Comment 12

[Don Moldover]

(In reply to Magnus Melin from comment #11)
> Is View | Character Encoding | Auto-detect set to off?

Yes it is

Comment 13

[Magnus Melin [:mkmelin]]

What about the Folder properties (Fallback Character Encoding)?

Comment 14

[Don Moldover]

(In reply to Magnus Melin from comment #13)
> What about the Folder properties (Fallback Character Encoding)?
That property was hard to find.  All of the references I have seen up to now have not explained that in order to find the property, I have to right click on the actual mail folder within the inbox rather than the inbox itself.  

When I found the properties box, I found that it was set to Western (ISO-8859-1).  After changing the Fallback Character Encoding to UTF-8 and clicking on a different mail folder and then clicking on the mail folder again the problem seems to have gone away.  I think you need one ***very prominent*** comprehensive article on this problem since I have been struggling with it for months and have conducted multiple internet searches on it and come up with bits and pieces each time but never the complete configuration details.  It is clear from those searches that many people have experienced the same problem.

Comment 15

[Don Moldover]

How do I mark this problem resolved?

Comment 16

[Don Moldover]

I should have said, I would mark it resolved if you can show me a decently written document addressing 'how and where to set the character encoding values'

Comment 17

[Magnus Melin [:mkmelin]]

I don't know if there's a support article for it, but here is how it's supposed to work (AFAIK)

If the message has a Content-Type has a charset set, use that
If not set
 - use the folder Fallback Character Encoding (if the "Apply encoding to ..... " checkbox is set)
Else, use the Incoming Mail Character Encoding

Things may be a bit different for news, but for mail, you should generally not have to touch these at all.

If your case, I guess you had the "Apply encoding to .... " checkbox ticked in the folder properties? It's generally not a good idea and I think we should just hide it for mail folders to prevent people from shooting themselves in the foot.

Comment 18

[Don Moldover]

(In reply to Magnus Melin from comment #17)
> I don't know if there's a support article for it, but here is how it's
> supposed to work (AFAIK)
> 
> If the message has a Content-Type has a charset set, use that
> If not set
>  - use the folder Fallback Character Encoding (if the "Apply encoding to
> ..... " checkbox is set)
> Else, use the Incoming Mail Character Encoding
> 
> Things may be a bit different for news, but for mail, you should generally
> not have to touch these at all.
> 
> In your case, I guess you had the "Apply encoding to .... " checkbox ticked
> in the folder properties? It's generally not a good idea and I think we
> should just hide it for mail folders to prevent people from shooting
> themselves in the foot.

If this is an example of end user documentation, I would be less than enthused about the product.  I've been a user for quite a few years and only began experiencing this problem in the last 6 months or so.  

I suspect that the interface has grown to accommodate new features in the data stream over time or organizations that don't comply with standards...whatever...that is no excuse to avoid simple and straightforward configuration instructions.  If the instructions can't be simple, then consider making a video for step-by-step configuration as some product vendors now do because the marketplace has become so polluted with non-conforming products.  

I understand that the product is free and I have patience but I still would not consider this bug-report as closed until the solution is documented to the degree that someone less well trained in the art of technology than I am can successfully configure it. I've been a software developer since 1969 when I joined IBM as a systems engineer and this kind of problem is what makes people tear their hair out and hate computer geeks.  It may well be that most people don't fall into the black hole that I did with conflicting Character Encoding in the mail folder and the global options but I'm an old hand at software and you can't expect most end users to put up with this.

Comment 19

[Magnus Melin [:mkmelin]]

Of course a bug comment can't be considered end user documentation.
My point is we shouldn't even provide that feature as it can only cause harm. 

Can you verify you had the checkbox ticked? If that's the case, you used a feature we should not support - and the fix is to simply remove it from the UI.

Comment 20

[Don Moldover]

(In reply to Magnus Melin from comment #19)
> Of course a bug comment can't be considered end user documentation.
> My point is we shouldn't even provide that feature as it can only cause
> harm. 
> 
> Can you verify you had the checkbox ticked? If that's the case, you used a
> feature we should not support - and the fix is to simply remove it from the
> UI.

I thought it was clear from my comment 14 above; however, yes...the checkbox was 'ticked'.  I have subsequently changed the value next to the checkbox to be UTF-8 and the problem has not recurred.

Comment 21

[Magnus Melin [:mkmelin]]

Ok, the mail should render fine without it ticked too.

Comment 22

[Don Moldover]

(In reply to Magnus Melin from comment #21)
> Ok, the mail should render fine without it ticked too.

You are correct...I have verified it.