664710 - Some characters display ASCII HEX blocks while Editor displays correctly, ASCII code also incorrect

Status: RESOLVED WORKSFORME

(Thunderbird :: Mail Window Front End, defect)

Description

[DutchGemini]

User-Agent:       Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; Avant Browser; Avant Browser; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET CLR 1.1.4322)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.17) Gecko/20110414 Lightning/1.0b2 Thunderbird/3.1.10

In message Viewer some characters do not display correctly and a small block with the ASCII code appears. In the Editor (after Reply) these also do not show correctly. The ASCII code also seems incorrect as well (offset -1)

In a memo I've received this is what I have:

Left quotation mark ‘  code 0145 &H91 - shown as 0090
Right quotation mark ’  code 0146 &H92 - shown as 0091

In the Message Source Viewer ([Ctrl]+[U]) the above characters show in the text as '.

After I save a draft of the reply, all characters turn out ok when viewing or editing the message.

Reproducible: Always

Steps to Reproduce:
1. Write a mail
2. In the message body, type Alt+0144, block will appear
3. Save as draft
4. View the message under Drafts

Actual Results:  
Characters display correctly ot display blocked question mark (if illegal character)

Expected Results:  
The right characters should appear in the viewer and editor and not after having saved to Drafts.

Can send mail with screenshots on demand

Comment 1

[DutchGemini]

Attachment: Reply Message showing ASCII codes (wrong codes)

Comment 2

[DutchGemini]

Attachment: Editing Message after saving as Draft now showing characters correctly

Comment 3

[Joe Sabash [:JoeS1]]

Testing with:
Mozilla/5.0 (Windows NT 5.0; rv:5.0) Gecko/20110602 Thunderbird/5.0b2pre ID:20110602000050 (The next release)

I am not able to duplicate the hex off by one symptom here.

Many times the proper character set is not stated correctly in incoming mail.
IE Content-type: text/plain; charset=UTF-8;

The default character set can be set on a per folder basis by right clicking on the folder and choosing "properties" 
Setting Default Character decoding to "Unicode (UTF-8)" may reduce the problem for you.

Comment 4

[DutchGemini]

(In reply to comment #3)
> Testing with:
Mozilla/5.0 (Windows NT 5.0; rv:5.0) Gecko/20110602
> Thunderbird/5.0b2pre ID:20110602000050 (The next release)

I am not able to
> duplicate the hex off by one symptom here.

Have ran a Repair. Hex offset not visible. Withdrawing comment and image.

Many times the proper character
> set is not stated correctly in incoming mail.
IE Content-type: text/plain;
> charset=UTF-8;

Encoding is explicit in message (see attachment).


The default character set can be set on a per folder basis
> by right clicking on the folder and choosing "properties" 
Setting Default
> Character decoding to "Unicode (UTF-8)" may reduce the problem for you.

INBOX is "Western (ISO-8859-1)" same as General encoding for in/outgoing mail. Setting of UTF-8 does not enhance reading, ASCII blocks continue to appear.

Comment 5

[DutchGemini]

Attachment: Message source (Ctrl+U) - Adapted

Comment 6

[DutchGemini]

Comment on attachment 539794 [details]
Reply Message showing ASCII codes (wrong codes)

Withdrawn, see Comment #3.

Comment 7

[Thomas D. (:thomas8)]

DutchGemini, do you still see this bug in TB12 or Trunk?

On Windows XP/TB12, Alt+0144 in composition (from STR) does nothing for me.

Comment 8

[DutchGemini]

On XP/TB12.0.1, Alt+0144 in composition does nothing anymore. Tested Fixed fonts as well as Variable font. Memo's showing ASCII code "blocks" instead of characters now also display properly. For me the problem is resolved.

Comment 9

[Thomas D. (:thomas8)]

(In reply to DutchGemini from comment #8)
> On XP/TB12.0.1, Alt+0144 in composition does nothing anymore. Tested Fixed
> fonts as well as Variable font. Memo's showing ASCII code "blocks" instead
> of characters now also display properly. For me the problem is resolved.

Resolved wfm per reporter's comment 8.

Not sure why we no longer accept Alt+NumberCode in composition: DutchGemini, is that a problem? Feel free to file a new bug if yes.

Comment 10

[DutchGemini]

We must continue to accept Alt+Numbercode otherwise we can't add characters physically not on our keyboard.

Invalid combinations leading to non-existing characters in the used font should not produce any character and for sure not the fancy ASCII-code block. Alt+0144 is such one.

No need to file another BR.

Comment 11

[Thomas D. (:thomas8)]

(In reply to DutchGemini from comment #10)
> We must continue to accept Alt+Numbercode otherwise we can't add characters
> physically not on our keyboard.
> 
> Invalid combinations leading to non-existing characters in the used font
> should not produce any character and for sure not the fancy ASCII-code
> block. Alt+0144 is such one.

Well, DutchGemini, you said in comment 8 that the main problem of this bug is "resolved", which means the problem described in comment 0 is gone and such characters now display correctly.
 
> No need to file another BR.

In your own interest, please DO file a new bugreport for new problems.
One issue per bug, one bug per issue. We'll make it much more difficult for developers to understand the remaining problem if we morph this bug. More difficult means less attention. Easy and clear from scratch means more attention, better chances for a fix.

We need new STR, Actual Behaviour, and Expected Behaviour, of which you seem to have a clear idea what should happen for valid and invalid combinations.