1309711 - After recovery with new selected email Message Pane will show that email with Text Encoding of message which was selected before

Status: RESOLVED DUPLICATE of bug 1287336

(SeaMonkey :: MailNews: Message Display, defect)

Description

[Mark]

Attachment: SampleMailFolders.zip

User Agent: Mozilla/5.0 (Windows NT 6.0; rv:43.0) Gecko/20100101 Firefox/43.0 SeaMonkey/2.40
Build ID: 20160120202951

Steps to reproduce:

SeaMonkey 2.40 on Windows Vista
User agent: Mozilla/5.0 (Windows NT 6.0; rv:43.0) Gecko/20100101 Firefox/43.0 SeaMonkey/2.40
Build identifier: 20160120202951

Extensions:
  ChatZilla 0.9.92
  DOM Inspector 2.0.16
  SeaMonkey Default Theme is selected

In case BugZilla doesn't display the strings correctly, in the following description:
- "Iñtërnâtiônàlizætiøn" should look similar "Internationalization" but with accented characters
- "Iñtërnâtiônà lizætiøn" has each accented character replaced with two other characters


Setup:
1. Create and switch to a new profile (Tools > Switch Profile > Manage Profiles); take note of where the profile is stored
2. Open SeaMonkey Mail & News
3. Set up an account with dummy details to initialise the mail folders (don't need to be able to connect to a server)
4. Set the account to store draft messages in the "Drafts" folder on "Local Folders" (Edit > Mail & Newsgroups Account Settings > (Account name) > Copies & Folders)
5. Close SeaMonkey
6. Extract* the "Drafts" and "Inbox" files from the attached "SampleMailFolders.zip" to $PROFILE_PATH\Mail\Local Folders\
7. Open SeaMonkey Mail & News using the clean profile

Steps to reproduce - Draft Corruption:
1. Select the Drafts folder under Local Folders
2. Select the "Test UTF-8 Draft" message
3. Press F8 to open the message pane (depending on previous actions, it may or may not display correctly, but that's not the concern at this step)
4. Press F8 to close the message pane
5. Double-click the "Test UTF-8 Draft" message to edit it
6. Observe the correct display of the string "Iñtërnâtiônàlizætiøn" in the second line of the message
7. Select the Inbox folder under Local Folders
8. Select the message "Test windows-1252"
9. Press F8 to open the message pane (it will probably display incorrectly, but that's not the concern at this step)
10. Press F8 to close the message pane
11. Select the Drafts folder under Local Folders
12. Double-click the "Test UTF-8 Draft" message to edit it
13. Expected Result: The message should look as it did in step 6.
    Actual Result: The second line of the message is "Iñtërnâtiônà lizætiøn" instead, having been decoded using an incorrect encoding

If this corruption is not noticed, the message may be further edited and re-saved or sent with corrupt characters. With the preference mailnews.send_default_charset set to "UTF-8", saving the corrupted draft and repeating this sequence increases the corruption. (UTF-8 doesn't seem to be available through the GUI, at Edit > Preferences > Mail & Newsgroups > Text Encoding, but I'm sure it used to be an option in previous versions as I've got it set in my normal profile; I'm not sure if other default charsets could lead to similarly increasing corruption).

Steps to reproduce - Incorrect Display:
1. Select the Inbox folder under Local Folders
2. Select the message "Test windows-1252"
3. Press F8 to open the message pane (depending on previous actions, it may or may not display correctly, but that's not the concern at this step)
4. Press F8 to close the message pane
5. Select the message "Test UTF-8"
6. Press F8 to open the message pane
   Expected Result: The second line of the message should be "Iñtërnâtiônàlizætiøn"
   Actual Result: The second line of the message is "Iñtërnâtiônà lizætiøn" instead, having been decoded using an incorrect encoding
7. Press F8 to close the message pane
8. Press F8 to open the message pane
   Result: The second line of the message is now correct "Iñtërnâtiônàlizætiøn"
9. Press F8 to open the message pane


* Sample messages are attached as a convenience, but can be created from scratch if desired, using a working email account. For the Inbox messages:
1. Compose a message, select Options > Text Encoding > Unicode, include "UTF-8" in the subject and the string "Iñtërnâtiônàlizætiøn" in the message, address the message to yourself, send the message
2. Compose a message, select Options > Text Encoding > Western, include "windows-1252" in the subject and the string "Iñtërnâtiônàlizætiøn" in the message, address the message to yourself, send the message
3. Wait for the messages to arrive in your Inbox
For the Drafts messages:
1. Compose a message, select Options > Text Encoding > Unicode, include "UTF-8" in the subject and the string "Iñtërnâtiônàlizætiøn" in the message, save draft
2. Compose a message, select Options > Text Encoding > Western, include "windows-1252" in the subject and the string "Iñtërnâtiônàlizætiøn" in the message, save draft

Comment 1

[Rainer Bielefeld]

@reporter:
Can you simply send such an email to me?

Comment 2

[David E. Ross]

Is this a duplicate of bug #715823?

Comment 3

[Mark]

@Rainer: Done.

@David: I don't think it's the same. #715823 looks like it occurs when forwarding a multi-part messages, where different parts use different encodings. This one happens with messages with only a single part.

I should have mentioned that in reproducing this I used plain-text-only messages - no HTML or alternative part.

Comment 4

[Rainer Bielefeld]

For me currently NOT reproducible with unofficial en-US SeaMonkey 2.49a1  (NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0 Build 20160930004545  (Default Classic Theme) on German WIN7 64bit

I've received 2 test-emails by PM, did some tests. 
currently I do not understand the core of the problem.

My first simple test:
11. In Email client / Folder Pane ˋselect Inbox with test mail 
   "Test Bug 1309711 - UTF-8" → Click test email in Thread Paneˊ
   » In Message Pane I see the word "Iñtërnâtiônàlizætiøn"
     That seems ok, 
     Message header tells:
     "Content-Type: text/plain; charset=UTF-8; format=flowed"
    Menu ˋView → Text Endocing shows "Unicode"ˊ
12. For selected test email change ˋMenu View → Text Encoding → Westernˊ
   » I see "Iñtërnâtiônà lizætiøn "
     Well, if I select wrong encoding ...

New Test:
21. In Email client / Folder Pane ˋselect Inbox with test mail 
   "Test Bug 1309711 - Windows-1252" → Click test email in Thread Paneˊ
   » In Message Pane I see the word "Iñtërnâtiônàlizætiøn"
     That seems ok, 
     Message header tells:
     "Content-Type: text/plain; charset=windows-1252; format=flowed"
    Menu ˋView → Text Endocing shows "Western"ˊ
22. For selected eest email change ˋMenu View → Text Encoding → Unicode
   » I see "I�t�rn�ti�n�liz�ti�n  "
     Well, if I select wrong encoding ...

@Reporter, please tell:
a) What exactly is your REAL problem? 
You get a Message with "charset=UTF-8", but SeaMonkey displays corrupted text in Message Pane? Because Menu ˋView → Text Endocing shows wrong encoding "Western"ˊ?
Or shows Corrupt text after having opened the email in email composer?
c) add information 
c1) concerning your Operating System (Distribution, Language)
c2) concerning your SM localization (UI language, Locale setting)
c3) SM settings that might be related to your problems 
    May be you should attach a copy of menu 
   'Help → Trouble shooting information”' as a text document
c4) how you launch SM (Double click on document? console? …)
c5) Whether problem persists in safe mode without add-ons
c6) Whether problem persists with blank new profile
    (you really did the test with new profild due to your steps?)
c7) everything else crossing your mind after you read linked texts
d) POP3 or IMAP account?

Comment 5

[Rainer Bielefeld]

(In reply to Rainer Bielefeld from comment #4)
NOT reproducible  with  en-US SeaMonkey 2.40 final Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) from official download page,  Gecko/20100101  Firefox/ 43.0  Build 20160120202951, (Default Classic Theme) on German WIN7 64bit

I again did my steps 11 ... 22 without reproducing reporter's problem, no "Iñtërnâtiônà lizætiøn", neither in Message pane not if opened in Message composer.

I forgot to mention that my steps 21,22 have been done for copies of the received test messages, created by <ctrk+drag-drop> of the 2 higlighted messages in inbox to LocalFolders-Drafts.

Comment 6

[Mark]

Attachment: Output from Help > Troubleshooting Information

@Rainer:

I am not manually selecting the encoding:
- Select the message encoded in Windows-1252 (Western).
- Press F8 to show the message pane (if it is not already shown)
- Press F8 again to hide the message pane
- Select (i.e. single-click in the thread pane; do not open it) the message encoded in UTF-8
- Press F8 to show the message pane

At this point, the message is displayed incorrectly - it appears to have been decoded using Windows-1252 rather than UTF-8. This is not a manual selection, it is SeaMonkey's automatic selection. The messages have the correct charset in the Content-Type header, indicating the encoding which should be used.

- Press F8 to hide the message pane
- Press F8 to show the message pane

The message now displays correctly.

a) The problem is that the message is shown using the wrong encoding when the message pane is first opened, if the previous message shown in the message pane used a different encoding.

The same effect can lead to the corruption of draft messages, which is worse as it can be difficult to correct if the message is saved again before the corruption is noticed:
- Save a draft using UTF-8 encoding
- View a message encoded in Windows-1252 (Western) in the message pane
- Close the message pane
- Double-click the draft message to open it for further editing; the text is corrupted in the same way; under some conditions, which I haven't fully determined, saving the draft in this corrupted state and repeating these steps can lead to further corruption of the affected characters.

c1) OS is Windows Vista, 32-bit Business.
c2) UI language is "English (United Kingdom)"; the clean test profile used for testing this issue has no spell-check dictionary installed; I don't see any other options related to locale.
c3) SM settings are mostly the defaults for a new profile, with minimal changes I described in the initial report. I've attach the output from Help > Troubleshooting Information.
c4) SM is launched using a Windows shortcut with target ["C:\Program Files\SeaMonkey\seamonkey.exe" -mail], but the issue also occurs if the browser is launched via the default shortcut (without "-mail") and then Mail & News is opened.
c5) Yes, the problem still occurs after restarting with with add-ons disabled.
c6) Yes, I created a new profile to test this issue just before posting the report.
c7) I can't think of anything else in the configuration which would affect this.
d) My main account is POP3, and that's how I received the test emails I sent to myself.

Comment 7

[Rainer Bielefeld]

(In reply to Mark from comment #6)
Thank you, I was completely wrong. Now I can reproduce the problem, will do some additional investigation tomorrow.

Comment 8

[Rainer Bielefeld]

Steps how to create test mail for users in West Europa and how to reproduce the problem:
30. Email client: Preferences “HTML” for composing and viewing 
31. go to <https://ja.wikipedia.org/wiki/%E6%9D%B1%E4%BA%AC>
32. MenuˋFile → Send Pageˊ
   » Email will be opened in composer
33. copy/paste text below second horizontal line below article eheading
   (starts with "東京") into email body below link
34. Check email subject line: should show “東京 - Wikipedia”
35. Send Email to one of your POP3 accounts
36. Check received email in inbox, body in Message Pane will show text with
   Japanese looking characters, all looks very similar to Wikipedia article.
37. Now in Thread Pane select a “normal” email in your inbox in a West European
    language: looks “normal”
38. <f8> to hide Message pane
39. Select your "東京" test email in Thread pane
38. <f8> to restore Message pane
   » body looks strange, “дЄ≠еЫљи™ЮпЉИгБ°гВЕгБЖгБФгБПгБФпЉЙгБѓгАБ ...”
     instead of Japanese looking characters

If these STR does not work for you please ask reporter or me for 2 test emails.

a): not related to Draft. The problem is: if you select a new email in Thread
    Pane while Message Pane is hidden, after recovery the Message Pane will 
    show the newly selected email with text encoding of the email which was
    selected before.
    Not related or limited to "Drafts" 
d): Problem affects POP3 and IMAP (as expected)

Comment 9

[Rainer Bielefeld]

e) no obvious DUPs found in <https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=DUPs1309711&sharer_id=41036>, but 
  "Bug 749537 - Forwarding creates message with wrong charset encoding " 
   looks similar.    And
e1) "Bug 1287336 - Thunderbird sometimes loads messages without specifying encoding"
  probably is very similar.

Comment 10

[Rainer Bielefeld]

Already reproducible with en-US SeaMonkey 2.33.1 Gecko/20100101 Build 20150321194901 (Classic Theme) on German WIN7 64bit

Comment 11

[Rainer Bielefeld]

f) Already reproducible with:
   Spanish SeaMonkey 2.26  (Portable) Gecko/20120604 Firefox/13.0 
   Build 20140428215944 (Default Classic Theme) on German WIN7 64bit 

g) Still worked fine with:
   en-US  SeaMonkey 2.19  (Windows NT 6.1; WOW64; rv:22.0)  Gecko/20100101
     Firefox/rv:22.0  Build 20130630011339  (Classic Theme) on German WIN7 64bit
   SeaMonkey 2.10   Mozilla/5.0 (Windows NT 6.1; WOW64; rv:13.0) Gecko/20120604
     Firefox/13.0 SeaMonkey/2.10

So an old regression!

Comment 12

[Rainer Bielefeld]

e) I still think that this one has common roots with Bug 1287336.        but:
h) NOT Reproducible with Thunderbird Daily 52.0a1 (2016-10-10) (64-bit)

Nevertheless, I mark this one NEW for now, we will see whether there is a relation to TB-bugs

Comment 13

[Mark]

(In reply to Rainer Bielefeld from comment #9)

Thanks for taking the time to reproduce and check earlier versions.

Bug 1287336 may have the same root cause as this one, but it seems to have missed that the visible effect is dependant on having previously viewed a message in a different encoding. Without that information, it might be difficult to reliably reproduce the issue (and therefore to confirm a fix).

Without more detail in bug 749537, it's not clear whether it could be the same as this issue (incorrect encoding used after viewing a message with a different encoding) or bug 715823 (incorrect encoding used when forwarding a multi-part message with different parts in different encodings) which was mentioned here in comments 2 and 3. Or, it could be another issue unrelated to either.

Comment 14

[bblack]

Clearly a duplicate of Bug 1287336. 
Both have the same cause but need to be patched in different places.

Comment 15

[bblack]

(In reply to Rainer Bielefeld from comment #8)

> 37. Now in Thread Pane select a “normal” email in your inbox in a West
> European language

This is not necessary.
You get exactly the same behavior when you click on a copy of the same email.
It is only a change between any two emails necessary to solve the error.

> 38. <f8> to hide Message pane
> 39. Select your test email in Thread pane
> 38. <f8> to restore Message pane
>    » body looks strange