676880 - surrogate pairs mishandled by Bugzilla + MySQL backend

Status: RESOLVED DUPLICATE of bug 405011

(Bugzilla :: Database, defect)

Description

[Felix S. Klock II [:pnkfelix, :fklock]]

I attempted to create a bug that in its description text contained a unicode code point from beyond the Basic Multilingual Plane (BMP; aka plane 0), that is represented in UTF-16 via a surrogate pair.  (I made the text by cut-and-pasting from a terminal window where I had generated the desired text.

The bugzilla.mozilla.org Bugzilla instance cut off the text right at the point where the surrogate pair occurred, and dropped the remainder of my text.

After I asked about this in the #bugzilla chat room on mozilla's irc channels, LpSolit tried to replicate my problem and determined that he was only able to replicate it on Bugzilla hosts with a MySQL backend (which I guess is the same as what bugzilla.mozilla.org uses?).  The PostgresSQL and SQLite backends do not appear to have the same problem.

To see what I am talking about, I made some small sample bugs on the landfill bugzilla servers, in particular posting the same bug to the three bugzilla instances with each of the database backends.  Links:

WORKS:
Postgres: https://landfill.bugzilla.org/bugzilla-tip-pg/show_bug.cgi?id=3229
  SQLite: https://landfill.bugzilla.org/bugzilla-tip-sqlite/show_bug.cgi?id=13961

BROKEN:
   MySQL: https://landfill.bugzilla.org/bugzilla-tip/show_bug.cgi?id=15718

I don't know enough about MySQL to know if this is an inherent limitation of that database backend, or if its something wrong with how bugzilla is linking up to MySQL.

But either way, it seems to me that Bugzilla should be capable of scanning the input text and alerting the bug reporter when the input text contains code points that will disrupt the transmission to or from the database.

(For reference, the original bug that prompted me to file this: https://bugzilla.mozilla.org/show_bug.cgi?id=676235 ; note there I have put comments documenting my attempts to properly transmit the desired code point via the bugzilla service.)

Comment 1

[Felix S. Klock II [:pnkfelix, :fklock]]

Attachment: text containing a code point outside BMP (plane 0)

Comment 2

[Frédéric Buclin]

See:
http://bugs.mysql.com/bug.php?id=14052
http://dev.mysql.com/doc/refman/5.5/en/charset-unicode.html

Comment 3

[Max Kanat-Alexander]

Interesting. So are you pasting them in UTF-16 format? I believe there should be a UTF-8 equivalent of any of these characters. If you paste them in UTF-8, do they work?

Comment 4

[Felix S. Klock II [:pnkfelix, :fklock]]

(In reply to Max Kanat-Alexander from comment #3)
> Interesting. So are you pasting them in UTF-16 format? I believe there
> should be a UTF-8 equivalent of any of these characters. If you paste them
> in UTF-8, do they work?

When I paste the blackboard bold 0 out of the attached file, they do not work.  The attached file is UTF-8.

Comment 5

[Max Kanat-Alexander]

Okay. From the bugs listed, it sounds like this is a MySQL issue, and not something that we can fix. So I'm marking it INVALID, because that's our only available resolution for things like this, even though I do agree that your bug itself is a valid issue that should be fixed.

If you can think of some way for us to work around this that wouldn't add a lot of complexity to Bugzilla, feel free to reopen this, but I think that ultimately we won't be able to, and that it has to be resolved on the database server side.

Comment 6

[Frédéric Buclin]

This is actually a dupe.