476488 - No-longer handles UTF-8 realnames correctly?

Status: RESOLVED INVALID

(Bugzilla :: Database, defect)

Description

[Christian Höltje]

Our Bugzilla installation was Bugzilla 3.0.6 and was using UTF-8 as the character set.

My profile's realname is Christian Höltje

When I upgraded to 3.2 everything looked fine, except now when I send an email, the realname displayed is as: "Christian H�ltje"

Looking in the database, I see that it has stored my name as: Christian H<F6>ltje
which is the iso-8859-1's "ö".

I suspect something is messed up, but ./checksetup.pl isn't telling me where it is.

Ciao!

Comment 1

[Max Kanat-Alexander]

Did you upgrade without the utf8 parameter turned on, or something? Is the utf8 parameter turned on in your 3.2 installation? Did you do a mysqldump and restore without preserving the character encoding?

Comment 2

[Christian Höltje]

(In reply to comment #1)
> Did you upgrade without the utf8 parameter turned on, or something?

No.  The utf8 value in data/params has been set since well before 3.0.6.  checksetup.pl said it was upgrading my tables to InnoSomething when I upgraded to 3.2.  I assume that's normal.

> Is the utf8 parameter turned on in your 3.2 installation?

Yes.

> Did you do a mysqldump and restore without preserving the character encoding?

No.  While this database is backed up, I haven't restored from it in a long long time.


Additional note:
The web interface shows my name correctly.  View-Info shows the page being loaded as UTF-8.

How do I check if the tables are correct UTF-8?  As I said, checksetup.pl says everything is copacetic.

Ciao!

Comment 3

[Max Kanat-Alexander]

(In reply to comment #2)
Hmm. Do you have custom templates, perhaps a custom email template?

In the sent emails, is the content-type set correctly with the utf-8 charset? Could you attach the source of one of these emails?

Comment 4

[Christian Höltje]

I do have a custom email template.  It's the same one from 3.0.6; I don't think I had to make any changes for 3.2.

The template creates an HTML and text version using MIME.

The text version is sent as:
----
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Christian Höltje
----

The Web version is sent as:
----
Content-Transfer-Encoding: base64
Content-Type: text/html; charset=utf-8

Christian H�ltje
----

The top one seems to be UTF-8 displayed as iso-8859.
The bottom one seems to be iso-8859 displayed as UTF-8.

The relevant code from the custom newchangedmail.txt.tmpl:
----
...
+MIME-Version: 1.0
+Content-Type: multipart/alternative;
+              boundary="[% boundary %]"
+
+
+--[% boundary %]
+Content-Type: text/plain; charset=utf-8
+Content-Transfer-Encoding: 7bit
+
+[%+ Param('urlbase') %]show_bug.cgi?id=[% bugid %]
+[%+ diffs %]
+
+[% PROCESS email/signature.txt.tmpl %]
+
+--[% boundary %]
+Content-Transfer-Encoding: base64
+Content-Type: text/html; charset=utf-8
+
+[%+ FILTER base64 %]
....
----

The original template doesn't include a Content-Type at all.

Ciao!

Comment 5

[Max Kanat-Alexander]

(In reply to comment #4)
> I do have a custom email template.

  Okay. In that case this is a support issue, and not a bug in Bugzilla, unless you can reproduce it with the default email template. Support questions go to one of the resources listed here:

  http://www.bugzilla.org/support/

Comment 6

[Christian Höltje]

You're right.  It's an encoding problem of somekind.  It's QP when using the default email template.

Comment 7

[Christian Höltje]

Though that still begs the question of why it is ISO-8859-1 in the database?

Comment 8

[Max Kanat-Alexander]

(In reply to comment #7)
> Though that still begs the question of why it is ISO-8859-1 in the database?

  Most likely just because of how you connected. The MySQL client doesn't use utf-8 by default.

Comment 9

[Christian Höltje]

(In reply to comment #8)
>   Most likely just because of how you connected. The MySQL client doesn't use
> utf-8 by default.

Bingo.

For future reference this is done by adding "default-character-set = utf8" to the [mysql] section.

Ciao!

Comment 11

[Christian Höltje]

More information about this:

My problem was that Bugzilla < 3.2 I had to force the Content-Types and encoding because Bugzilla didn't fully understand UTF-8 nor did it understand multipart MIME messages correctly.

In 3.2 it now does! Yay!  But that means my old stuff broke in the new bugzilla.

Here is the revised version of the newchangedmail.txt.tmpl above:
----
...
+Content-Type: multipart/alternative;
+              boundary="[% boundary %]"
+
+
+--[% boundary %]
+Content-Type: text/plain
+
+[%+ Param('urlbase') %]show_bug.cgi?id=[% bugid %]
+[%+ diffs %]
+
+[% PROCESS email/signature.txt.tmpl %]
+
+--[% boundary %]
+Content-Type: text/html
+
....
----

The Content-Type is still required, but it should not include the charset.  Content-Transfer-Encoding should not be included.
I also removed the explicit FILTER base64 since I removed the Content-Transfer-Encoding.
Finally, I removed the MIME version (this was probably harmless, though).

Ciao!