Return receipts not conform to RFC 2047

RESOLVED FIXED

Status

Thunderbird
General
RESOLVED FIXED
12 years ago
10 years ago

People

(Reporter: Jean-Claude REPETTO, Assigned: Bienvenu)

Tracking

({fixed1.8.1.3})

Firefox Tracking Flags

(Not tracked)

Details

(URL)

Attachments

(2 attachments, 1 obsolete attachment)

(Reporter)

Description

12 years ago
User-Agent:       Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.0.2) Gecko/20060421 Firefox/1.5.0.2
Build Identifier: Thunderbird 1.5.0.2 FR downloaded from http://www.mozilla.com/products/download.html?product=thunderbird-1.5.0.2&os=win&lang=fr

When TB sends a return receipt in French, it sends non-ASCII characters in the subject header.

When the recipient of the return receipt has configured amavisd-new to reject bad headers, I get error messages such as :

INVALID HEADER (INVALID CHARACTERS OR SPACE GAP)
Non-encoded 8-bit data (char E9 hex) in message header 'Subject': Subject: Accus\351 de r\351ception (...

and the recipient doesn't get the return receipt.

The problem comes from some localized messages are using non-ASCII characters .
For example :
MdnDisplayedReceipt=Accus\u00e9 de r\u00e9ception (affich\00e0)

These messages should be encoded by TB (RFC 2047).
Same problem for the following messages :
MdnDispatchedReceipt,MdnProcessedReceipt,MdnDeletedReceipt,MdnDeniedReceipt,MdnFailedReceipt.

This problem certainly occurs for several other languages.


Reproducible: Always

Steps to Reproduce:
1. Install a french version of Thunderbird 1.5.0.2
2. Send a message to Thunderbird from another place, asking for a return receipt
3. When the message arrives, accept to send the return receipt
4. At the other place, look at the headers of the return receipt message sent by Thunderbird

Actual Results:  
The header of the return receipt contains :
Subject: Accusé de réception (affiché) -  Essai AR jcr



Expected Results:  
The header od the return should contain someting like :
Subject: =?iso8859-1?Q?Accus=E9_de_r=E9ception_(affich=E9)?= -  Essai AR jcr
(Reporter)

Comment 1

12 years ago
Same problem in version 1.5.0.4

Comment 2

12 years ago
Yes, this is a Thunderbird bug for the return receipt function. The bug is reported for the French version of TBird but I suspect it will be a problem for all languages that use 8bit characters.

For example, 

1. If someone sends a message header: le ministère des Finances

TBird/French version returns the following subject header in French.

Subject: Accusé de réception (affiché) -  le =?ISO-8859-1?Q?minist=E8re_des_Finances?=

The canned header part "Accusé de réception (affiché)" does not go through MIME encoding.

2. The body part that acknowledges the receipt a message always looks like this:

--------------mdn080702080101020103080400
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Ceci est un accusé de réception pour le courrier éléctronique envoyé à user_id@yahoo.com.

Note : Cet accusé de réception indique seulement que le message a été affiché sur l'ordinateur du destinataire. Il n'y a aucune garantie que le destinataire ait lu ou appréhendé le contenu du message.

======================

What's wrong is that if the MIME encoding is OFF in the preference, Tbird always adds: 

Content-Transfer-Encoding: 7bit 

even if the body contains 8bit data. 

The return receipt part quotes the original headers -- which is always supposed to be in 7bit (ASCII) but what TBird is adding are 8bit character data.

Is there a way make the return receipt go though the normal message formatting process so that the MIME requirements are met properly?



Status: UNCONFIRMED → NEW
Ever confirmed: true

Comment 3

12 years ago
(In reply to comment #2)
> 
> 1. If someone sends a message header: le ministère des Finances
> 
> TBird/French version returns the following subject header in French.
> 
> Subject: Accusé de réception (affiché) -  le
> =?ISO-8859-1?Q?minist=E8re_des_Finances?=
> 
> The canned header part "Accusé de réception (affiché)" does not go through
> MIME encoding.

Momoi-san, I'll ask an ignorant question. What if the subject is in a different charset encoding than the canned text? i.e. you are using a french build (Accusé de réception (affiché)) but the subject in the message we are generating a return receipt for us is encoded in say a japanese character set? Should we be encoding each of these separately in the subject? Or should the entire subject be encoded once? 

FYI, this should be trivial to fix, the code is right here:

http://lxr.mozilla.org/mozilla/source/mailnews/extensions/mdn/src/nsMsgMdnGenerator.cpp#574

We just need to figure out if both strings should be encoded together for the subject or if the canned text should be encoded separately. 

Comment 4

12 years ago
it seems prett obvious that my question was ignorant :). We'll want to encode each part of the subject separately, RFC 2047 talks about that. So this should be easy if we can get a string that maps to the charset of the current locale.
(Assignee)

Comment 5

12 years ago
see also bug 139615 

I'll attach a patch in a minute that addresses these issues, by using utf-8
(Assignee)

Comment 6

12 years ago
Created attachment 257405 [details] [diff] [review]
proposed fix for subject and body encoding

This builds on Scott's initial patch to encode the return receipt part of the subject as utf8, and also generate 8 bit body, encoded as utf8.

I changed the "structured" arg to nsMsgI18NEncodeMimePartIIStr to FALSE. I believe "structured" is used to indicate that it's an address list, and passing in TRUE was causing things before '(' to get trimmed off.

I removed the localization comment about not translating the mdn body strings, since I think they can now be localized. I don't have to check that into 2.0 if you think that's a bad idea, but it's not going to break anything, most likely :-)
Assignee: mscott → bienvenu
Status: NEW → ASSIGNED
Attachment #257405 - Flags: superreview?(mscott)

Comment 7

12 years ago
A question & comment:

I thought the spec was to return a localized return receipt only if the original sender's first accept language choice matched that of the user's localized UI language. That is, the French Thunderbird UI language user will send out the French version of the return receipt only if the message's X-Accept language header lists "fr" at the top. Otherwise, it falls back to the ASCII version of the return msg. Is this not true?

If the above assumption is right, I don't understand the need to use UTF-8 in encoding body or the headers. In cases where the original contains non-Latin 1 headers such as Chinese characters, you can re-use the original MIME encoded words. Otherwise, simply use the encoding that matches the UI language.

It is still true that in some parts of the world, e.g East Asia, UTF-8 causes garbling problem. If we use UTF-8, we could end up inconveniencing the requester of the receipt.
(Assignee)

Comment 8

12 years ago
Kat, I think currently we're always returning the localized return receipt, except we tell the localizers not to localize the strings.

How do we find out the current localization charset?

Comment 9

12 years ago
Comment on attachment 257405 [details] [diff] [review]
proposed fix for subject and body encoding

thanks David for doing this. I think it's fine to remove the comment on the branch.
Attachment #257405 - Flags: superreview?(mscott) → superreview+
(Assignee)

Comment 10

12 years ago
fixed on trunk and branch

We should spin Kat's issue into a new bug...
Status: ASSIGNED → RESOLVED
Last Resolved: 12 years ago
Keywords: fixed1.8.1.3
Resolution: --- → FIXED

Comment 11

12 years ago
> Kat, I think currently we're always returning the localized return receipt,
> except we tell the localizers not to localize the strings.

Thanks for the update. 

> How do we find out the current localization charset?

The language of the TB build is in the X-user agent string. And there should be a file that maps from the language to default mail_charset.

> We should spin Kat's issue into a new bug...

I think this bug will be inevitable given the acceptance of UTF-8 in some locales.

Comment 12

12 years ago
Hah, I think this is one of the rare cases where a TB patch "fixed" a SeaMonkey file but not the TB one - even if it's just a L10n note ;-)

You should probably remove the localization note from http://lxr.mozilla.org/mozilla/source/mail/locales/en-US/chrome/messenger/msgmdn.properties as well :)
(Assignee)

Updated

12 years ago
Duplicate of this bug: 139615

Comment 14

12 years ago
Jean-Claude, can you help verify this bug with a branch nightly:

http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-mozilla1.8-l10n/

Thanks!
(Reporter)

Comment 15

12 years ago
I have tested the windows version in french (thunderbird-2.0pre.fr.win32.installer.exe).

I have sent myself a message with the following subject :
Subject: Essai =?ISO-8859-1?Q?accus=E9_de_r=E9ception?=

And I have received a mdn with that subject:
Subject: =?UTF-8?B?QWNjdXPDqSBkZSByw6ljZXB0aW9uIChhZmZpY2jDqSkgLSA=?=Essai =?ISO-8859-1?Q?accus=E9_de_r=E9ception?=

The subject is correctly displayed in Thunderbird :
Accusé de réception (affiché) - Essai accusé de réception

So, I think it is correct. Thanks to all the people that worked to fix this bug !

Jean-Claude

Comment 16

12 years ago
Dear all,
I would like to bring to your attention the following bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=254149

I think it's related to this one. In this case, amavis complains that the subject header is not correctly encoded.

My test bench are several mail servers (debian etch and sarge) with amavis as content filter, while on the client side I've got TB up to version 2.0pre (20070319) (french).

Comment 17

11 years ago
In Seamonkey/1.1.1 there is still a bug. The Subject is coded as follows (German locale):

Subject: ?ISO-8859-15?Q?Empfangsbest=E4tigung_=28angezeigt=29?=

But it should be

Subject: =?ISO-8859-15?Q?Empfangsbest=E4tigung_=28angezeigt=29?=

The fix for Thunderbird is probably not ported to Seamonkey. The file mozilla/mailnews/extensions/mdn/src/nsMsgMdnGenerator.cpp differs from Seamonkey to Thunderbird (see attachment). 

Do we need a separate bug report for Seamonkey?

Comment 18

11 years ago
Created attachment 260854 [details] [diff] [review]
Diff between Seamonkey and Thunderbird

Comment 19

11 years ago
Bjoern, I don't understand your diff. There is only one nsMsgMdnGenerator file in the tree. It's shared by seamonkey and thunderbird, as are most of the files in mailnews.

Comment 20

11 years ago
Created attachment 260858 [details] [diff] [review]
 Diff between Seamonkey and Thunderbird
Attachment #260854 - Attachment is obsolete: true

Comment 21

11 years ago
Yes, normally the file mozilla/mailnews/extensions/mdn/src/nsMsgMdnGenerator.cpp should be shared between Seamonkey and Thunderbird. But for the current versions (Seamonkey 1.1.1 and Thunderbird 2.0.0) this is not the case and therefore this bug is still present in Seamonkey. I wanted to show this with the attached patch.

Currently I'm testing, if Seamonkey 1.1.1 works right with nsMsgMdnGenerator.cpp from Thunderbird 2.0.0. 

Comment 22

11 years ago
Bjoern:
I believe the bug you are seeing is actually in the language pack - and additionally, SeaMonkey 1.1.1 is much older source than Thunderbird 2.0, this will be fixed in SeaMonkey 1.1.2 which should get released about the same time as Thunderbird 2.0 final.

Comment 23

11 years ago
My tests with nsMsgMdnGenerator.cpp from Thunderbird 2.0.0 copied to Seamonkey 1.1.1 were successful:
- the German language pack seems to be ok
- Subject is correctly encoded

Is it possible to verify if Seamonkey 1.1.2 will have the corrected version of nsMsgMdnGenerator.cpp? What is the correct CVS branch for Seamonkey 1.1.2 development?

Comment 24

11 years ago
The CVS branch for SeaMonkey 1.1.2 is the same as for Thunderbird 2.0, i.e. MOZILLA_1_8_BRANCH
(Assignee)

Updated

11 years ago
Duplicate of this bug: 267702

Comment 26

11 years ago
Return receipts look right when sent from 2.0.0.5-pre (lt langpack).
Status: RESOLVED → VERIFIED

Comment 27

11 years ago
Oops, I guess I was too early.
Here's what a "Subject" header of the Return Receipt looks like:

Subject: Gavimo pažyma (laiškas atvertas) - =?windows-1257?Q?=FEi=FBrim_=FEi=FBrim=2E=2E=2E?=

That is clearly RFC-incompliant.

One question: am I supposed to put a QP-string in my translation for this case, or should I leave this responsibility to the application (Thunbderbird)?
Status: VERIFIED → REOPENED
Resolution: FIXED → ---

Updated

11 years ago
Blocks: 386993

Updated

10 years ago
Flags: wanted-thunderbird3?

Updated

10 years ago
Duplicate of this bug: 386542

Comment 29

10 years ago
Could anyone please take a look at comment #27 ?
OS: Windows 2000 → All
Hardware: PC → All

Updated

10 years ago
Depends on: 462725
(In reply to comment #12)
> You should probably remove the localization note from
> http://lxr.mozilla.org/mozilla/source/mail/locales/en-US/chrome/messenger/msgmdn.properties
> as well :)

This bug is pretty much the poster child for "don't comment on closed bugs unless you want to be ignored," isn't it? Pushed in http://hg.mozilla.org/comm-central/rev/442f53864c91 and thanks for catching it.

(In reply to comment #27)
> Here's what a "Subject" header of the Return Receipt looks like:
> 
> Subject: Gavimo pažyma (laiškas atvertas) -
> =?windows-1257?Q?=FEi=FBrim_=FEi=FBrim=2E=2E=2E?=

Nice catch - you're just missing one step, where you say "here's what a Subject header looks like _unless you have already sent an email in the same session_." Filed (and to my astonishment, fixed) bug 462725 for that case.

> One question: am I supposed to put a QP-string in my translation for this case,
> or should I leave this responsibility to the application (Thunbderbird)?

No, you shouldn't cheat your way around the bug like de and fr did, but since this is after all the poster child for "don't comment on closed bugs etc." you apparently should have filed a new bug saying "...still don't conform..."
Status: REOPENED → RESOLVED
Last Resolved: 12 years ago10 years ago
Resolution: --- → FIXED

Comment 31

10 years ago
(In reply to comment #30)
I'll keep that in mind, thanks. ;) Also thanks for finally fixing.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

Updated

10 years ago
Status: REOPENED → RESOLVED
Last Resolved: 10 years ago10 years ago
Resolution: --- → FIXED
Flags: wanted-thunderbird3?
You need to log in before you can comment on or make changes to this bug.