777685 - us-ascii character encoding used on HTML bugmails even with UTF-8 activated

Status: RESOLVED FIXED

(Bugzilla :: Email Notifications, defect)

Description

[Thorsten Schöning]

User Agent: Opera/9.80 (Windows NT 6.1; WOW64; U; de) Presto/2.10.289 Version/12.00

Steps to reproduce:

I upgraded Bugzilla from 2.22.x to 4.2.1, which activated HTML mails fro mail notifications by default. My installation has the UTF-8-parameter turned on. Afterwards I added a comment to one of my test bugs to verify everythign is working.


Actual results:

I received a mail notification with content of text/plain and text/html and german umlauts from my name as UTF-8 encoded characters. Those characters were not properly displayed in my mail client, The Bat! 5.1.6.2, as for example ö, but it shows ö instead.


Expected results:

I should have seen properly encoded german umlauts for my name.

Today I had time to look into this and I think I found the problem:
Bugzilla::BugMail sets a content-type for HTML mails, but without a
character encoding, which seem to result in a content type like the
following in the HTML mails I receive:

Content-Type: multipart/alternative; boundary="..."; charset="us-ascii"

This content-type is only for the mail container itself, the text/plain and
text/html content within this mail have its own proper content-type
with UTF-8 set:

--1343291439.3F77F23.3000
Date: Thu, 26 Jul 2012 10:30:39 +0200
MIME-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

--1343291439.3F77F23.3000
Date: Thu, 26 Jul 2012 10:30:39 +0200
MIME-Version: 1.0
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

The characters are all encoded properly, the problem seems to be the
character set of the outermost content-type statement, which let my
mail client to believe everything is encoded in US-ASCII. This may be
a problem of my client of course, but when I change BugMail.pm and
manually set a character encoding of UTF-8 everything is fine.

$email->content_type_set('multipart/alternative');
$email->charset_set('UTF-8');

As Bug 723944 introduced a workaround for encoding issues in plain text mails in 4.2.1, I would suspect the same character encoding is set for each part of the generated email.

Comment 1

[Frédéric Buclin]

Just to be sure I understand correctly: is your name displayed incorrectly...

a) in the HTML part of HTML+Text bugmails only?
b) in the plain-text part of HTML+Text bugmails only?
c) in both parts of HTML+Text bugmails?
d) in plain-text bugmails?
e) in the To: header of HTML+Text bugmails?
f) in the To: header of plain-text bugmails?

Of course, these questions are not exclusive, and so a yes/no for each would be welcome. :)

CC'ing Marc too in case he already saw this problem.


--- off-topic ---

And while I'm on it: thank you very much Thorsten for all your help in the support mailing-list. That's highly appreciated! :)

Comment 2

[Thorsten Schöning]

> a) in the HTML part of HTML+Text bugmails only?
> b) in the plain-text part of HTML+Text bugmails only?
> c) in both parts of HTML+Text bugmails?

In both, I will attach screenshots afterwards.

> d) in plain-text bugmails?

No, forgot to mention that: If I change my preferences to text only, everything is displayed correctly as only one encoding is given in the mail. I didn't test HTML only, but would expect the same with that.

> e) in the To: header of HTML+Text bugmails?
> f) in the To: header of plain-text bugmails?

Ny name is not mentioned in those headers, but the subject is encoded properly, so I would epext the same for To:

Subject: [Bug 299]=?UTF-8?Q?=20Headerpr=C3=BCfung=20auf=20Signaturdateien?=

> --- off-topic ---
> 
> And while I'm on it: thank you very much Thorsten for all your help in the
> support mailing-list. That's highly appreciated! :)

Thanks, it's a really good practice for my english knowledge. ;-)

Comment 3

[Thorsten Schöning]

Attachment: Screenshot for text-part of the mail

This screenshot shows what my mail client shows on the text tab for the default HTML messages Bugzilla sends. The subject has proper german umlauts, my name doesn't.

Comment 4

[Thorsten Schöning]

Attachment: Screenshot for text-part of the mail

Same as before, just with the HTML-tab.

Comment 5

[Thorsten Schöning]

Attachment: originally sent mail

This attachment is the mail my Bugzilla generated.

Comment 6

[Frédéric Buclin]

I tested with Thunderbird, and this is not an issue. It displays both parts correctly. I don't know what the RFC spec says about this, but I suppose the email client is supposed to look at each part encoding individually.

Comment 7

[:glob ✱]

right, this is a bug in the email client.  the main content-type header applies to the non-mime part of the body (in our case it's empty).

thorsten, what email client are you using?

that said .. it's a trivial fix and won't hurt to make the change.

Comment 8

[Thorsten Schöning]

(In reply to Byron Jones ‹:glob› from comment #7)
> thorsten, what email client are you using?

The Bat! 5.1.6.2, but they won't change the problem any time soon, therefore I hoped Bugzilla could.

Comment 9

[Marc Schumann [:Wurblzap]]

(In reply to Frédéric Buclin from comment #1)
> CC'ing Marc too in case he already saw this problem.

I haven't seen this problem yet. I'm using MS Outlook 2010 and the Google Mail client.

Comment 10

[Thorsten Schöning]

(In reply to Byron Jones ‹:glob› from comment #7)
> that said .. it's a trivial fix and won't hurt to make the change.

Did you come to any conclusion whether to include the workaround or not?

Comment 11

[Thorsten Schöning]

Attachment: set UTF-8 on multipart/alternative for broken mail clients

As of Comment 7, this patch enables UTF-8 for empty main content-type header to better support broken mail clients like mine.

Comment 12

[Frédéric Buclin]

Comment on attachment 684508 [details] [diff] [review]
set UTF-8 on multipart/alternative for broken mail clients

>         $email->content_type_set('multipart/alternative');
>+        $email->charset_set('UTF-8');

You cannot unconditionally set the charset to UTF-8. You should only do this if the utf8 parameter is turned on. Please ask :glob for review when uploading an updated patch.

Comment 13

[Thorsten Schöning]

Attachment: set UTF-8 on multipart/alternative for broken mail clients v2

UTF-8 is only set if it's an enabled param for Bugzilla and added one sentence as description.

Comment 14

[:glob ✱]

Comment on attachment 684605 [details] [diff] [review]
set UTF-8 on multipart/alternative for broken mail clients v2

r=glob

brackets around the condition should be removed on commit.

Comment 15

[:glob ✱]

Committing to: bzr+ssh://[e-mail]/bugzilla/trunk/
modified Bugzilla/BugMail.pm
Committed revision 8489.

Committing to: bzr+ssh://[e-mail]/bugzilla/4.4/
modified Bugzilla/BugMail.pm
Committed revision 8472.

Comment 17

[dl]

ver 4.4.1
bad default view. Look on 2 email before and after patch.

Comment 18

[dl]

Attachment: e-mail before patch

Comment 19

[dl]

Attachment: e-mail after patch

Comment 20

[dl]

patch for BugMail.pm file in #946597

Comment 21

[:glob ✱]

this patch has already been committed to 4.4.

Comment 22

[dl]

in  version 4.4.1 does not have

Comment 23

[Simon Green]

Committing to: bzr+ssh://[e-mail]/bugzilla/4.4/
modified Bugzilla/BugMail.pm
Committed revision 8472.