555693 - Incorrect non-ascii chars when reopening draft message without prerendering in message pane

Status: RESOLVED DUPLICATE of bug 597369

(Thunderbird :: Message Compose Window, defect)

Description

[hawran]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.8) Gecko/20100214 Ubuntu/9.10 (karmic) Firefox/3.5.8
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.9.1.8) Gecko/20100227 Lightning/1.0b1 Thunderbird/3.0.3

I wrote a message with no ascii characters. I didn't finish it so I just closed it, the message went to the Drafts folder.
When I tried to continue editing, the original characters were incorrect, there were a lot of ? characters (see attachments).



Reproducible: Always

Steps to Reproduce:
1. Write a message with no ascii characters.
2. Close the message (save it into drafts)
3. Look/Open the message for editing again.
Actual Results:  
No ascii characters are incorrect.

Expected Results:  
No ascii characters are correct.

The original language is Czech, characters encodings utf8 (see attachments).
Linux Ubuntu 9.10 (Karmic Koala)
Thunderbird Configuration:
Generated: Mon Mar 29 2010 14:27:47 GMT+0200 (CET)
User Agent: Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.9.1.8) Gecko/20100227 Lightning/1.0b1 Thunderbird/3.0.3
Build ID: 20100227141902

Enabled Extensions: [5]
- British English Dictionary 1.19: http://www.google.com/search?q=Thunderbird%20British%20English%20Dictionary
- Lightning 1.0b1: http://www.mozilla.org/projects/calendar/releases/lightning1.0b1.html
- MR Tech Toolkit 6.0.4: http://www.mrtech.com/extensions/
- Noia 2.0 eXtreme XT 3.21: http://pagesperso-orange.fr/thunderbird-noia2/
- České slovníky pro kontrolu pravopisu 1.0.1: http://www.google.com/search?q=Thunderbird%20%u010Cesk%E9%20slovn%EDky%20pro%20kontrolu%20pravopisu

Installed Themes: [3]
- Default: http://www.mozilla.org/
- Littlebird 1.8.56: https://addons.mozilla.org/en-US/thunderbird/addon/1493/developers
- Noia 2.0 eXtreme 3.21: http://pagesperso-orange.fr/thunderbird-noia2/

Installed Plugins: (8)
- Adobe Reader 9.3
- DivX® Web Player
- Java(TM) Plug-in 1.6.0_18-b07
- QuickTime Plug-in 7.2.0
- Shockwave Flash
- Software 602 Filler plugin
- VLC Multimedia Plugin (compatible Totem 2.28.2)
- Windows Media Player Plug-in 10 (compatible; Totem)

Comment 1

[hawran]

Attachment: An original message

Comment 2

[hawran]

Attachment: The message within the Drafts folder

Comment 3

[hawran]

Attachment: The message for editing

Comment 4

[hawran]

Attachment: Character encodings

Comment 5

[[:Aureliano Buendía]]

hawran:

1. select your draft folder (the folder where your "wrong" email are stored);
2. click on right mouse botton for show context menu and select "properties..."
item on it;
3. on "general information" tab uncheck option "Apply default to all message to
all messages in the folder..."

This resolve your issue?

ps: in general you should be set as uncheck this flag on all folders (Inbox, Sent, Draft, ...).

Comment 6

[hawran]

That option has already been unchecked, hence this is not a solution.

Additionally, I've noticed this ridiculous behaviour in the Sent folder.

Comment 7

[[:Aureliano Buendía]]

Could attache here a message that has this issue? (select and save it as *.eml from menu-->file)

Comment 8

[hawran]

Attachment: A couple of eml files

I've just attached a couple of eml files as requested by Aureliano Buendía.

I've had a look at the Base64 encoded part and I'd say there's something wrong during the encoding phase, the characters are wrong.

Btw, pay attention to the file 04-some_ascii_and_czech_characters_from_the_1st_line_on_a_keyboard_between_keys_+_and_=.eml , the non ascii characters are OK!!!

Comment 9

[hawran]

Hi,
any progress?

Can I help with whatever?

Comment 10

[hawran]

Hi,
Is anybody taking care of it?

It IS really annoying issue!

Got some more info on the matter:
I've created an e-mail, used non-ascii character within a subject
I've sent it.

When I've checked out the e-mail sen, I've noticed that, for some reason, there's a strange signature within the e-mail's header:
------------------------------------
Subject: =?iso-8859-2?Q?P=E1r=20link=F9=20ohledn=EC=20prepaidu?=
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
------------------------------------

iso-8859-2? WHY????
I've got utf-8 as a default for all.

Comment 11

[hawran]

I've just noticed an interesting thing (I'd sent one message to my other account and got it through pop3 then):

From "Sent" folder:
---------------------------------------------------------------
From: My Name <[e-mail]>
To: "Man, Other" <[e-mail]>
Date: Wed, 21 Jul 2010 08:34:42 +0200
Subject: =?utf-8?B?UMOhciBsaW5rxa8gb2hsZWRuxJsgcHJlcGFpZHU=?=
Accept-Language: cs-CZ, en-US
Content-Language: cs-CZ
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.9.1.10)
 Gecko/20100512 Lightning/1.0b1 Thunderbird/3.0.5
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64

WmRhcmVjIQ0KDQpQb3M/P2w/P20gcD8/ciBsaW5rPz8gcHJvIHNlYmV2emQ/P2w/P3Y/P24/Py4g
...
...
dC8wMCUyMFNoYXJlZCUyMERvY3VtZW50cy9Gb3Jtcy9EYXRhc2hlZXQuYXNweA0KDQpoYXcNCg0K
---------------------------------------------------------------


From "Inbox" folder:
---------------------------------------------------------------
From - Wed Jul 21 08:36:11 2010
Reply-To: =?us-ascii?Q?My=20Name?= <[e-mail]>
X-Spam-Checker-Version: szn-spamassassin 2009-06-26
X-Spam-Status: score=-2.6
X-Spamscore: 0
X-Esetresult: clean
From: =?us-ascii?Q?My=20Name?= <[e-mail]>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.9.1.10) Gecko/20100512 Lightning/1.0b1 Thunderbird/3.0.5
Mime-Version: 1.0
To: =?us-ascii?Q?Man=2C=20Other?= <[e-mail]>
Subject: =?iso-8859-2?Q?P=E1r=20link=F9=20ohledn=EC=20prepaidu?=
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Zdarec!

Pos??l??m p??r link?? pro sebevzd??l??v??n??. Nebo sebedestrukci, jak 
kdo chce:
...
...
---------------------------------------------------------------

Comment 12

[Jörg Wagner]

Same behaviour here, but I have an important bit of additional information to offer:

The wrong (or correct) display of non ASCII chars after reopening a draft email depends on the character set of the last inbox email that has been displayed just before the draft is reopened. Here some examples:

Last viewed inbox email is
Content-Type: text/plain; charset="iso-8859-1"
--> WRONG display of accented characters in reopened draft email

Last viewed inbox email has multiple parts with
Content-Type: Text/Plain; charset="iso-8859-1"
Content-Type: Text/HTML; charset="iso-8859-1"
--> WRONG display of accented characters in reopened draft email

Last viewed inbox email is
Content-Type: text/plain; charset=UTF-8
--> CORRECT display of accented characters in reopened draft email

Last viewed inbox email has multiple parts with
Content-Type: text/plain; charset=UTF-8
Content-Type: text/html; charset=UTF-8
--> CORRECT display of accented characters in reopened draft email


System is:
  TB 3.1.2
  Windows XP
  Draft folder: "Apply default to all message in the folder..." is UNCHECKED


Hope that helps to nail down this annoying problem. I experience it every day since I switched to TB3.

Cheers, Jörg.

Comment 13

[Jörg Wagner]

Still the same with TB 3.1.3 - extremely annoying.
To make sure it is only TB I switched off all Add-ons - result is still the same.

To best reproduce make sure you have no other emails open in TB and switch of the preview window (F8). Close and restart TB. Then view an inbox email with Latin-1 (ISO-8859-1) encoding, close it again and open a UTF-8 draft email. 
Result: All UTF-8 chars will display incorrectly.

Comment 14

[33wq78ggfdgfd462q70]

Same problem for me with TB 3.1.6 on Windows 7 x64, it's really extremely annoying.

Comment 15

[Jörg Wagner]

I think I found another bit of information to reproduce this bug and also some kind of workaround:

I always disable the Message Pane when using TB. This seems necessary to reproduce the problem!

To work around it:  Reenable the message pane by pressing F8. As soon as you click on a msg in the drafts folder, it is shown in the message pane - and there it is always showing the CORRECT encoding! 

Looks like the problem only happens when opening msgs without message pane preview. With the pane active you can also double click any draft msg to open it in its own tab with correct encoding (probably because just before opening the tab the encoding has been correctly set for the message pane preview).

Comment 16

[Wayne Mery (:wsmwk)]

Do you still see this issue when using version 5 or 6 (due out in a week)?
- If you no longer see the problem, please set status to RESOLVED, and set resolution to WORKSFORME. (or perhaps INVALID, if you determined the problem is not Thunderbird)
- If you still see the problem, please provide updated details, and any additional steps needed to reproduce the problem.

Comment 17

[Jörg Wagner]

This issue was still very much alive and kicking with the latest 3.1 of Thunderbird, but doing some tests with 5.0 I could not reproduce it within the last 10 minutes.
Considering the fact that the bug always appeared quite erratically (probably depending on the encoding of the last emails that had been opened before reopening a draft email) and was never easy to reproduce, I would prefer to leave this bug report open for some more weeks. If I drop again on the issue I will leave a note here.

Comment 18

[Jörg Wagner]

It just happened again. Unfortunatelly no easy way to reproduce it. As I said above the message pane must be switched OFF. An active msg pane clearly has the side effect of initiating the correct character decoding. Without msg pane the character encoding when reopening a draft email seems to depend on the encoding of emails that have been opened before. But I am not able to create a clearly reproducable setup to provoke the error. It probably depends on multiple factors and on the usage history before reopening a draft email.

Comment 19

[Jörg Wagner]

Here is an example screenshot from Thunderbird 6.0:
http://www.digilog.de/fremdbilder/thunderbird/Thunderbird%20Bug%20555693%20-%20Incorrect%20no%20ascii%20chars.png
It is an email reply that I had saved to the Drafts folder and reopened later.
The corrupted display vanished when I reopened the draft with msg pane activated.
It also vanished when I restarted Thunderbird and reopened the draft before opening any other email.

Comment 20

[Jörg Wagner]

And still the same problem in TB 16.0.2.

Comment 21

[Carlos Martin Ugalde]

And still the same problem in TB 24.3.0

Comment 23

[Jörg Wagner]

Complete compilation of infos about this bug:

Problem:
Character encoding of emails stored in Drafts folder is lost when they are reopened while the message pane is switched OFF.

To reproduce:
+ Make sure you have a UTF-8 encoded email with accented characters (e.g. äöü) saved in Drafts folder (write UTF-8 encoded email, hit Ctrl-S to save, close email without sending).
+ Switch OFF message pane using F8 key toggle.
+ Reopen email saved in Drafts folder by double-clicking it. Check whether accented characters appear correctly. This step does not always reproduce the problem, so keep trying. Open other emails from Inbox in-between trials - preferably emails that are NOT UTF-8 encoded (e.g. Latin-1).

Work around:
+ Open message pane using F8 key before reopening emails from Drafts folder.
+ Double clicking emails in Drafts folder will then open them correctly as they are first loaded into the msg pane, which eliminates the problem.

Affected:
+ Any versions of TB from 3.0 to current 24.3. 
+ Reproducible on Linux, Win XP, Win 7 and Win 7 64. So probably not OS dependent.

Comment 24

[Magnus Melin [:mkmelin]]

Confirming on trunk, thanks for the STR!

Comment 25

[pizza_rulez]

And the bug still exists more than 5 years after its initial description... unbelievable. Confirmed for version 38.3.0

Comment 26

[Nicolas Dietrich [:nidi]]

I also have this issue in Icedove (Debian stable) 31.8.0, a collegue has it on a current Thunderbird on Windows 7.

Comment 27

[Nicolas Dietrich [:nidi]]

Still happens in Icedove 38.4.0 on Debian Jessie (stable).

Note that having the message pane closed is not strictly necessary to trigger the issue. Another (also not fully deterministic way) is to save an pgp-encrypted mail to draft, restart thunderbird
only one way to reproduce

Comment 28

[Nicolas Dietrich [:nidi]]

(Sorry for the incomplete comment due to my wrong assumption on the bugzilla UI, starting over)

Still happens in Icedove 38.4.0 on Debian Jessie (stable).

I've changed OS and hardware as apparently this issue is OS-independent (happens on Windows and Linux) and also hardware independent (happens on x86 and x86_64).

Note that having the message pane closed is not strictly necessary to trigger the issue. Another (also not always reproducible) way with open message pane is to save an pgp-encrypted mail to draft, restart thunderbird and open the message without previously entering the decryption passphrase. The effect is the same - the message isn't prerendered in the message pane and the editor then uses the wrong encoding. I've corrected the title slightly in order to reflect this.

Comment 29

[Jorg K (CEST = GMT+2)]

Fixed by bug 597369 or bug 1239658.

Comment 30

[M. Beil]

In my case the above solution did not work, but the following steps:
Right click on "Entwuerfe" (drafts) and select a fix  text coding (Unicode (UTF-8)).
Now I can safe and reopen drafts with the right non-ISO characters.