Open Bug 488282 Opened 11 years ago Updated 4 months ago

Problem with copies placed in exchange sent message with special characters (é, à,..)

Categories

(MailNews Core :: Networking: IMAP, defect)

1.8 Branch
x86
macOS
defect
Not set

Tracking

(Not tracked)

People

(Reporter: kakeez, Unassigned)

References

Details

(Keywords: intl)

Attachments

(3 files)

User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b3) Gecko/20090305 Firefox/3.1b3
Build Identifier: version 2.0.0.6 (20070807)

If you copy-paste a text written in MS-Word onto a new mail and proceed to sending the mail, the copy of the message placed in the 'Sent' folder does not display correctly. The received mail is displayed correctly.

The problem seems to occur because MS-Word uses windows-1252 encoding. The received mail is correctly identified as using windows-1252 encoding and correctly displayed. The mail saved in the 'Sent' folder is identified as windows-1252 but does not display correctly.

For example:

J'espère will display J?esp?re in the sent folder.

When I tried reproducing the same thing but this time changing the character encoding in the new mail window from the default ISO-8859-1 to windows-1252, both emails (received one and the one copied into sent folder) display correctly.

Nonetheless this is a bug because that change should happen automatically. Thunderbid should detect and set the new encoding to windows-1252 and not try to save the mail into 'Sent' using ISO-8859-1, which I am assuming is happening.





Reproducible: Always

Steps to Reproduce:
1.Write J'espère in MS-WORD
2.Copy-paste into a new mail
3.Send email to yourself
Actual Results:  
Received mail displays correctly.
Copy placed in 'Sent' does not display correctly

Expected Results:  
Both received and copy should display correctly
Is the sent folder a local folder (like mostly in pop3) or a remote folder (like in imap generally) ?
It is a remote folder.
Kakeez can you follow the instrcutions at https://wiki.mozilla.org/MailNews:Logging and log what is going on imap wise (et si tu comprends pas écris moi en privé je t'aiderais), and attach the imap log file to this bug ?
Component: General → Networking: IMAP
Product: Thunderbird → MailNews Core
QA Contact: general → networking.imap
Version: unspecified → 1.8 Branch
Attached file Requested IMAP log
-replaced the actual domains by private.com
-replace my name (first and last) my private_firstname(lastname)
Hi,

Followed the instructions and attached the log file 

Also, my bug report should be updated. It appears that you do not need to copy paste from word for the bug to occur. If you simply type in french and send, the same problem occurs: received mail displays correctly and sent mail does not. The only exception to this is when you type in french and include an attachement! In this case both received and sent display correctly.So this is not linked to windows-1252 encoding at all as I thought since the default encoding when typing is ISO.

Also what I wrote :
 ''When I tried reproducing the same thing but this time changing the character
encoding in the new mail window from the default ISO-8859-1 to windows-1252,
both emails (received one and the one copied into sent folder) display
correctly.''

is not correct. This does not work. What works is copy-pasting the text from the received email (which is displayed correctly) and sending it again, this time with windows-1252. In this case, the sent mail displays correctly.

Thank you.

Karim
Severity: normal → major
Summary: Problem with copies placed in sent folder of messages encoded with windows-1252 → Problem with copies placed in sent folder of messages with special characters (é, à,..)
Is this an Exchange IMAP sent folder? 
IIRC, that's simply an Exchange bug... (and a dupe)
Attachment #372918 - Attachment mime type: application/octet-stream → text/plain
Yes this is an Exchange IMAP folder. This problem does not occur when using another mail client such as Mail.app (OS X).
From the log:

> -1340583936[18c64770]: 1d81800:ewa.private.com:S-Sent:CreateNewLineFromSocket: J'esp?re que tu vas bien cette ann?e.

... so the server is sending the question marks to us.

Have you tried setting the Saved folder to be on the Local Folders account? When you do, is the copy in the sent folder displayed correctly from there?
Summary: Problem with copies placed in sent folder of messages with special characters (é, à,..) → Problem with copies placed in exchange sent folder of messages with special characters (é, à,..)
Trying to set to a Local Folder, the copy is displayed correctly. Setting back the sent folder to be on IMAP excahnge, copies are displayed correctly until Thunderbird is restarted, at which point the bad behavior returns.
Duplicate of this bug: 489760
So, I have the same bug here (Windows XP, Exchange 2007). But, I'm not sure about comment #9, my experience seems different : sometimes accents are displayed correctly, and sometimes (often) they don't, without any change in the configuration.
All encoding changes seems to have no effect.

I've read somewhere users have same problem with Gmail on IMAP. I'm not sure it is an Exchange bug. Because a manual copy of the same message in the same IMAP folder works perfectly.
(In reply to comment #4)
> Requested IMAP log

The MS Exchange server apparently reformats/alters mail data.
  - Header sequence is changed, some headers are added.
  - format=flowed, Content-Transfer-Encoding: 8bit
    => no format=flowed, Content-Transfer-Encoding: quoted-printable
And, 8bit byte code of iso-8859-1 is wrongly altered by the MS Exchange server.

(In reply to comment #9)
> Setting back the sent folder to be on IMAP excahnge, copies are displayed correctly until Thunderbird is restarted, at which point the bad behavior returns.

Reformating/alteration is executed while Sent folder is not opened?
Mail source is changed after restart of Tb? UID is changed at server after restart of Tb? (Check value of "Order Received" column of IMAP folder)

[Log data(viewed in iso-8859-1 and pasted here)]

(1) APPENDed mail data to Sent folder by Tb (==copy of mail data passed to SMTP server) 
> ewa.private.com:A:SendData: 3 append "Sent" (\Seen) {408+}
> ewa.private.com:A:SendData: Message-ID: <49E623CA.7010407@private.com>
> Date: Wed, 15 Apr 2009 20:13:30 +0200
> From: private_firstname private_lastname <private_firstname.private_lastname@private.com>
> User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070807)
> MIME-Version: 1.0
> To: private_firstname private_lastname <private_firstname.private_lastname@private.com>
> Subject: Bonjour: =?ISO-8859-1?Q?=E7a_va=3F?=
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
> Content-Transfer-Encoding: 8bit
>
> J'espère que tu vas bien cette année.
>         (8bit byte code of iso-88590-1. If log is viewed in UTF-8, dislayed as)
>              J'esp�re que tu vas bien cette ann�e.
>
> ewa.private.com:A:SendData: 
> ewa.private.com:A:CreateNewLineFromSocket: 3 OK APPEND completed.

(2) FETCHed mail data from Sent folder
> ewa.private.com:S-Sent:CreateNewLineFromSocket: * 727 FETCH (UID 763 RFC822.SIZE 518 BODY[] {549}
> ewa.private.com:S-Sent:STREAM:OPEN Size: 518: Begin Message Download Stream
> ewa.private.com:S-Sent:CreateNewLineFromSocket: From: private_firstname private_lastname <private_firstname.private_lastname@private.com>
> ewa.private.com:S-Sent:CreateNewLineFromSocket: To: private_lastname private_firstname <private_firstname.private_lastname@private.com>
> ewa.private.com:S-Sent:CreateNewLineFromSocket: Date: Wed, 15 Apr 2009 20:13:30 +0200
> ewa.private.com:S-Sent:CreateNewLineFromSocket: Subject: =?iso-8859-1?Q?Bonjour:_=E7a_va=3F?=
> ewa.private.com:S-Sent:CreateNewLineFromSocket: Thread-Topic: =?iso-8859-1?Q?Bonjour:_=E7a_va=3F?=
> ewa.private.com:S-Sent:CreateNewLineFromSocket: Message-ID: <49E623CA.7010407@private.com>
> ewa.private.com:S-Sent:CreateNewLineFromSocket: Accept-Language: en-US
> ewa.private.com:S-Sent:CreateNewLineFromSocket: Content-Language: en-US
> ewa.private.com:S-Sent:CreateNewLineFromSocket: X-MS-Has-Attach:
> ewa.private.com:S-Sent:CreateNewLineFromSocket: X-MS-TNEF-Correlator:
> ewa.private.com:S-Sent:CreateNewLineFromSocket: user-agent: Thunderbird 2.0.0.6 (Macintosh/20070807)
> ewa.private.com:S-Sent:CreateNewLineFromSocket: Content-Type: text/plain; charset="iso-8859-1"
> ewa.private.com:S-Sent:CreateNewLineFromSocket: Content-Transfer-Encoding: quoted-printable
> ewa.private.com:S-Sent:CreateNewLineFromSocket: MIME-Version: 1.0
> ewa.private.com:S-Sent:CreateNewLineFromSocket: 
> ewa.private.com:S-Sent:CreateNewLineFromSocket: J'esp?re que tu vas bien cette ann?e.
> ewa.private.com:S-Sent:CreateNewLineFromSocket: 
> ewa.private.com:S-Sent:CreateNewLineFromSocket:  FLAGS (\Seen \Recent))
> ewa.private.com:S-Sent:STREAM:CLOSE: Normal Message End Download Stream
> ewa.private.com:S-Sent:CreateNewLineFromSocket: 7 OK FETCH completed.
Status: UNCONFIRMED → NEW
Ever confirmed: true
(In reply to comment #0)
> Thunderbid should detect and set the new encoding to windows-1252 and not try
> to save the mail into 'Sent' using ISO-8859-1, (snip)

kakeez@hotmail.com(bug opener), check your character encoding related settings for outgoing mail.
 - Tools/Options/Display, Fonts&Encodings, Fonts
   (a) Character Encodings, Outgoing Mail:
   (b) [ ] use the default character encoding in replies
If (a)==iso-8859-1 && (b)==Checked, reply mail to mail with charset=windows-1252 is composed in iso-8859-1.

The MS Exchange server(or application on the server) possibly assumes 8bits-byte-code==windows-1252(or CP 850, or similar one).
Can (a)==windows-1252 be a workaround of your problem?
Status: NEW → UNCONFIRMED
Status: UNCONFIRMED → NEW
Very strange this bug : with the same configuration on both Thunderbird and Exchange 2007 (I'm the Exchange admin here - and I use TB !), the results are differents. Mails I've sent today :

11:04 -> internal (Exchange recipient) (new) mail : bad copy
11:07 -> external (Re:) mail : good copy
11:19 -> internal (Re:) mail : good copy
11:24 -> external (Re:) mail : good copy
11:48 -> internal (new) mail : bad copy
12:04 -> internal (Re:) mail : bad copy
13:40 -> mixed recipients in bcc (new) mail : good copy
13:48 -> internal (new) mail : good copy
13:49 -> internal (Re:) mail : good copy

...
> Reformating/alteration is executed while Sent folder is not opened?
> Mail source is changed after restart of Tb? UID is changed at server after
> restart of Tb? (Check value of "Order Received" column of IMAP folder)
> 

I am sorry but I don't understand your questions...
> The MS Exchange server(or application on the server) possibly assumes
> 8bits-byte-code==windows-1252(or CP 850, or similar one).
> Can (a)==windows-1252 be a workaround of your problem?

It is not a workaround. Same problem as before.
(In reply to comment #14)
> Very strange this bug : with the same configuration on both Thunderbird and
> Exchange 2007 (I'm the Exchange admin here - and I use TB !), the results are
> differents. Mails I've sent today :
> 
> 11:04 -> internal (Exchange recipient) (new) mail : bad copy
> 11:07 -> external (Re:) mail : good copy
> 11:19 -> internal (Re:) mail : good copy
> 11:24 -> external (Re:) mail : good copy
> 11:48 -> internal (new) mail : bad copy
> 12:04 -> internal (Re:) mail : bad copy
> 13:40 -> mixed recipients in bcc (new) mail : good copy
> 13:48 -> internal (new) mail : good copy
> 13:49 -> internal (Re:) mail : good copy
> 
> ...

This confirms the erratic behavior I have reported:

1-The fact that when sending a mail with attachment, this problem disappears.
2-The fact that after setting the sent mail folder to local and then back to IMAP, the problem disappears until there is a restart.
3-The initial confusion reported when copy-pasting from word and resetting the character encoding..

Basically, sometimes you have the problem and sometimes you don't. Good luck to whoever decides to debug this :)

K.
(In reply to comment #17)
> 2-The fact that after setting the sent mail folder to local and then back to IMAP,
> the problem disappears until there is a restart.

It indicates that reformatting/altering of mail data(then incorrect mail data) for TB is executed by MS Exchange server after termination of Tb/before restart of Tb. 

Please execute step by step check of mail source and UID of mail.
(0) Show "Order Received" column of IMAP folder (value is UID of mail),
    and sort by "Order Received" column, and view newest mail.   
(1) Copy a mail with correct 8bits char of iso-8859-1 to local mail flder
    (say Folder-1, call mail-1),
    or Compose a mail/Send Later, then move the mail from Unsent (Outbox if
    Tb 3) of "Local Folders" to local Folder-1.
(2) Upload the mail in Folder-1 to IMAP folder(UID=A, call mail-1X)
(3) Copy the mail from IMAP folder(UID=B, call mail-2) to local Folder-2
(4) Terminate Tb, and restart Tb after a while.
(5) Copy the mail from IMAP folder(UID=C, call mail-3) to local Folder-3
(6) Rebuild Index of IMAP folder(force re-download of mail data)
(7) Copy the mail from IMAP folder(UID=D, call mail-4) to local Folder-4

At which step is mail source altered? Same UID(A==B==C==D)? Or UID is changed?
(In reply to comment #18)
> At which step is mail source altered? Same UID(A==B==C==D)? Or UID is changed?

Note that the mail is never altered when it's copied manually in a folder (IMAP or local). The mail is altered only when TB place a sent copy in the imap sent folder defined in the options.
UID (order received) is changed all the times when a mail is copied, even if it has not be altered.
Note that this log is obtained with exactly the same configuration of TB and Exchange.
Attachment #381733 - Attachment mime type: application/octet-stream → text/plain
Attachment #381732 - Attachment mime type: application/octet-stream → text/plain
Keywords: intl
Summary: Problem with copies placed in exchange sent folder of messages with special characters (é, à,..) → Problem with copies placed in exchange sent message with special characters (é, à,..)
Severity: major → normal
You need to log in before you can comment on or make changes to this bug.