335534 - Return receipts not conform to RFC 2047

Status: RESOLVED FIXED

(Thunderbird :: General, defect)

Description

[Jean-Claude REPETTO]

User-Agent:       Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.0.2) Gecko/20060421 Firefox/1.5.0.2
Build Identifier: Thunderbird 1.5.0.2 FR downloaded from http://www.mozilla.com/products/download.html?product=thunderbird-1.5.0.2&os=win&lang=fr

When TB sends a return receipt in French, it sends non-ASCII characters in the subject header.

When the recipient of the return receipt has configured amavisd-new to reject bad headers, I get error messages such as :

INVALID HEADER (INVALID CHARACTERS OR SPACE GAP)
Non-encoded 8-bit data (char E9 hex) in message header 'Subject': Subject: Accus\351 de r\351ception (...

and the recipient doesn't get the return receipt.

The problem comes from some localized messages are using non-ASCII characters .
For example :
MdnDisplayedReceipt=Accus\u00e9 de r\u00e9ception (affich\00e0)

These messages should be encoded by TB (RFC 2047).
Same problem for the following messages :
MdnDispatchedReceipt,MdnProcessedReceipt,MdnDeletedReceipt,MdnDeniedReceipt,MdnFailedReceipt.

This problem certainly occurs for several other languages.


Reproducible: Always

Steps to Reproduce:
1. Install a french version of Thunderbird 1.5.0.2
2. Send a message to Thunderbird from another place, asking for a return receipt
3. When the message arrives, accept to send the return receipt
4. At the other place, look at the headers of the return receipt message sent by Thunderbird

Actual Results:  
The header of the return receipt contains :
Subject: Accusé de réception (affiché) -  Essai AR jcr



Expected Results:  
The header od the return should contain someting like :
Subject: =?iso8859-1?Q?Accus=E9_de_r=E9ception_(affich=E9)?= -  Essai AR jcr

Comment 1

[Jean-Claude REPETTO]

Same problem in version 1.5.0.4

Comment 2

[Katsuhiko Momoi]

Yes, this is a Thunderbird bug for the return receipt function. The bug is reported for the French version of TBird but I suspect it will be a problem for all languages that use 8bit characters.

For example, 

1. If someone sends a message header: le ministère des Finances

TBird/French version returns the following subject header in French.

Subject: Accusé de réception (affiché) -  le =?ISO-8859-1?Q?minist=E8re_des_Finances?=

The canned header part "Accusé de réception (affiché)" does not go through MIME encoding.

2. The body part that acknowledges the receipt a message always looks like this:

--------------mdn080702080101020103080400
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Ceci est un accusé de réception pour le courrier éléctronique envoyé à [e-mail].

Note : Cet accusé de réception indique seulement que le message a été affiché sur l'ordinateur du destinataire. Il n'y a aucune garantie que le destinataire ait lu ou appréhendé le contenu du message.

======================

What's wrong is that if the MIME encoding is OFF in the preference, Tbird always adds: 

Content-Transfer-Encoding: 7bit 

even if the body contains 8bit data. 

The return receipt part quotes the original headers -- which is always supposed to be in 7bit (ASCII) but what TBird is adding are 8bit character data.

Is there a way make the return receipt go though the normal message formatting process so that the MIME requirements are met properly?

Comment 3

[Scott MacGregor]

(In reply to comment #2)
> 
> 1. If someone sends a message header: le ministère des Finances
> 
> TBird/French version returns the following subject header in French.
> 
> Subject: Accusé de réception (affiché) -  le
> =?ISO-8859-1?Q?minist=E8re_des_Finances?=
> 
> The canned header part "Accusé de réception (affiché)" does not go through
> MIME encoding.

Momoi-san, I'll ask an ignorant question. What if the subject is in a different charset encoding than the canned text? i.e. you are using a french build (Accusé de réception (affiché)) but the subject in the message we are generating a return receipt for us is encoded in say a japanese character set? Should we be encoding each of these separately in the subject? Or should the entire subject be encoded once? 

FYI, this should be trivial to fix, the code is right here:

http://lxr.mozilla.org/mozilla/source/mailnews/extensions/mdn/src/nsMsgMdnGenerator.cpp#574

We just need to figure out if both strings should be encoded together for the subject or if the canned text should be encoded separately. 

Comment 4

[Scott MacGregor]

it seems prett obvious that my question was ignorant :). We'll want to encode each part of the subject separately, RFC 2047 talks about that. So this should be easy if we can get a string that maps to the charset of the current locale.

Comment 5

[David :Bienvenu]

see also bug 139615 

I'll attach a patch in a minute that addresses these issues, by using utf-8

Comment 6

[David :Bienvenu]

Attachment: proposed fix for subject and body encoding

This builds on Scott's initial patch to encode the return receipt part of the subject as utf8, and also generate 8 bit body, encoded as utf8.

I changed the "structured" arg to nsMsgI18NEncodeMimePartIIStr to FALSE. I believe "structured" is used to indicate that it's an address list, and passing in TRUE was causing things before '(' to get trimmed off.

I removed the localization comment about not translating the mdn body strings, since I think they can now be localized. I don't have to check that into 2.0 if you think that's a bad idea, but it's not going to break anything, most likely :-)

Comment 7

[Katsuhiko Momoi]

A question & comment:

I thought the spec was to return a localized return receipt only if the original sender's first accept language choice matched that of the user's localized UI language. That is, the French Thunderbird UI language user will send out the French version of the return receipt only if the message's X-Accept language header lists "fr" at the top. Otherwise, it falls back to the ASCII version of the return msg. Is this not true?

If the above assumption is right, I don't understand the need to use UTF-8 in encoding body or the headers. In cases where the original contains non-Latin 1 headers such as Chinese characters, you can re-use the original MIME encoded words. Otherwise, simply use the encoding that matches the UI language.

It is still true that in some parts of the world, e.g East Asia, UTF-8 causes garbling problem. If we use UTF-8, we could end up inconveniencing the requester of the receipt.

Comment 8

[David :Bienvenu]

Kat, I think currently we're always returning the localized return receipt, except we tell the localizers not to localize the strings.

How do we find out the current localization charset?

Comment 9

[Scott MacGregor]

Comment on attachment 257405 [details] [diff] [review]
proposed fix for subject and body encoding

thanks David for doing this. I think it's fine to remove the comment on the branch.

Comment 10

[David :Bienvenu]

fixed on trunk and branch

We should spin Kat's issue into a new bug...

Comment 11

[Katsuhiko Momoi]

> Kat, I think currently we're always returning the localized return receipt,
> except we tell the localizers not to localize the strings.

Thanks for the update. 

> How do we find out the current localization charset?

The language of the TB build is in the X-user agent string. And there should be a file that maps from the language to default mail_charset.

> We should spin Kat's issue into a new bug...

I think this bug will be inevitable given the acceptance of UTF-8 in some locales.

Comment 12

[Robert Kaiser]

Hah, I think this is one of the rare cases where a TB patch "fixed" a SeaMonkey file but not the TB one - even if it's just a L10n note ;-)

You should probably remove the localization note from http://lxr.mozilla.org/mozilla/source/mail/locales/en-US/chrome/messenger/msgmdn.properties as well :)

Comment 14

[Scott MacGregor]

Jean-Claude, can you help verify this bug with a branch nightly:

http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-mozilla1.8-l10n/

Thanks!

Comment 15

[Jean-Claude REPETTO]

I have tested the windows version in french (thunderbird-2.0pre.fr.win32.installer.exe).

I have sent myself a message with the following subject :
Subject: Essai =?ISO-8859-1?Q?accus=E9_de_r=E9ception?=

And I have received a mdn with that subject:
Subject: =?UTF-8?B?QWNjdXPDqSBkZSByw6ljZXB0aW9uIChhZmZpY2jDqSkgLSA=?=Essai =?ISO-8859-1?Q?accus=E9_de_r=E9ception?=

The subject is correctly displayed in Thunderbird :
Accusé de réception (affiché) - Essai accusé de réception

So, I think it is correct. Thanks to all the people that worked to fix this bug !

Jean-Claude

Comment 16

[Alessandro]

Dear all,
I would like to bring to your attention the following bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=254149

I think it's related to this one. In this case, amavis complains that the subject header is not correctly encoded.

My test bench are several mail servers (debian etch and sarge) with amavis as content filter, while on the client side I've got TB up to version 2.0pre (20070319) (french).

Comment 17

[Bjoern Voigt]

In Seamonkey/1.1.1 there is still a bug. The Subject is coded as follows (German locale):

Subject: ?ISO-8859-15?Q?Empfangsbest=E4tigung_=28angezeigt=29?=

But it should be

Subject: =?ISO-8859-15?Q?Empfangsbest=E4tigung_=28angezeigt=29?=

The fix for Thunderbird is probably not ported to Seamonkey. The file mozilla/mailnews/extensions/mdn/src/nsMsgMdnGenerator.cpp differs from Seamonkey to Thunderbird (see attachment). 

Do we need a separate bug report for Seamonkey?

Comment 18

[Bjoern Voigt]

Attachment: Diff between Seamonkey and Thunderbird

Comment 19

[Scott MacGregor]

Bjoern, I don't understand your diff. There is only one nsMsgMdnGenerator file in the tree. It's shared by seamonkey and thunderbird, as are most of the files in mailnews.

Comment 20

[Bjoern Voigt]

Attachment: Diff between Seamonkey and Thunderbird

Comment 21

[Bjoern Voigt]

Yes, normally the file mozilla/mailnews/extensions/mdn/src/nsMsgMdnGenerator.cpp should be shared between Seamonkey and Thunderbird. But for the current versions (Seamonkey 1.1.1 and Thunderbird 2.0.0) this is not the case and therefore this bug is still present in Seamonkey. I wanted to show this with the attached patch.

Currently I'm testing, if Seamonkey 1.1.1 works right with nsMsgMdnGenerator.cpp from Thunderbird 2.0.0. 

Comment 22

[Robert Kaiser]

Bjoern:
I believe the bug you are seeing is actually in the language pack - and additionally, SeaMonkey 1.1.1 is much older source than Thunderbird 2.0, this will be fixed in SeaMonkey 1.1.2 which should get released about the same time as Thunderbird 2.0 final.

Comment 23

[Bjoern Voigt]

My tests with nsMsgMdnGenerator.cpp from Thunderbird 2.0.0 copied to Seamonkey 1.1.1 were successful:
- the German language pack seems to be ok
- Subject is correctly encoded

Is it possible to verify if Seamonkey 1.1.2 will have the corrected version of nsMsgMdnGenerator.cpp? What is the correct CVS branch for Seamonkey 1.1.2 development?

Comment 24

[Robert Kaiser]

The CVS branch for SeaMonkey 1.1.2 is the same as for Thunderbird 2.0, i.e. MOZILLA_1_8_BRANCH

Comment 26

[Rimas Kudelis [use needinfo to ping]]

Return receipts look right when sent from 2.0.0.5-pre (lt langpack).

Comment 27

[Rimas Kudelis [use needinfo to ping]]

Oops, I guess I was too early.
Here's what a "Subject" header of the Return Receipt looks like:

Subject: Gavimo pažyma (laiškas atvertas) - =?windows-1257?Q?=FEi=FBrim_=FEi=FBrim=2E=2E=2E?=

That is clearly RFC-incompliant.

One question: am I supposed to put a QP-string in my translation for this case, or should I leave this responsibility to the application (Thunbderbird)?

Comment 29

[Rimas Kudelis [use needinfo to ping]]

Could anyone please take a look at comment #27 ?

Comment 30

[Phil Ringnalda (:philor)]

(In reply to comment #12)
> You should probably remove the localization note from
> http://lxr.mozilla.org/mozilla/source/mail/locales/en-US/chrome/messenger/msgmdn.properties
> as well :)

This bug is pretty much the poster child for "don't comment on closed bugs unless you want to be ignored," isn't it? Pushed in http://hg.mozilla.org/comm-central/rev/442f53864c91 and thanks for catching it.

(In reply to comment #27)
> Here's what a "Subject" header of the Return Receipt looks like:
> 
> Subject: Gavimo pažyma (laiškas atvertas) -
> =?windows-1257?Q?=FEi=FBrim_=FEi=FBrim=2E=2E=2E?=

Nice catch - you're just missing one step, where you say "here's what a Subject header looks like _unless you have already sent an email in the same session_." Filed (and to my astonishment, fixed) bug 462725 for that case.

> One question: am I supposed to put a QP-string in my translation for this case,
> or should I leave this responsibility to the application (Thunbderbird)?

No, you shouldn't cheat your way around the bug like de and fr did, but since this is after all the poster child for "don't comment on closed bugs etc." you apparently should have filed a new bug saying "...still don't conform..."

Comment 31

[Rimas Kudelis [use needinfo to ping]]

(In reply to comment #30)
I'll keep that in mind, thanks. ;) Also thanks for finally fixing.