477442 - "Edit as comment" mangles charset

Status: NEW

(Bugzilla :: Attachments & Requests, defect)

Description

[Phil Ringnalda (:philor)]

STR:

1. Add an attachment with a non-ISO-8859-1 character, like a Mozilla DTD with a UTF-8 ellipsis character
2. Go to the &action=edit URL
3. Note that the character is properly displayed
4. "Edit as comment"
5. Note that if your install is using MySQL, the character is mangled into the "â¦" that's the result of treating UTF-8 as ISO-8859-1

After seeing one on bmo, I added one to landfill's bugzilla-tip and bugzilla-3.2-branch, both of which failed, and added one to bugzilla-tip-pg which did not fail (the one I added to bugzilla-tip-oracle is, sadly, too unsafe to be viewed by ordinary mortals, so I don't know whether it worked).

Comment 1

[Frédéric Buclin]

... is also mangled in bugzilla-tip-oracle. I just turned on allow_attachment_display there.

Comment 2

[Max Kanat-Alexander]

Actually, all DBs should be doing this. If they're not, it's a sign that they're treating the raw bytes of attachments as always being utf8, which is not good.

The problem here is that we no longer get the advantage of the browser's charset detector, since we no longer use XMLSerializer to display the attachment.

We actually have no idea what character set the attachment is in, though we could use Encode::Guess (which isn't that reliable) on it and convert to utf8 on display for "edit attachment as comment", if the utf8 bit is on, for text/* attachments.

I'm going to leave the blocking flag as "?" until somebody does more investigation into a good way to fix this and we see how invasive it would be.

Comment 3

[Vitaly Fedrushkov]

The symptom is probably introduced by attachment 286357 [details] [diff] [review]

Comment 4

[Max Kanat-Alexander]

(In reply to comment #3)
> The symptom is probably introduced by attachment 286357 [details] [diff] [review]

  Sort of. That plus a combination of bug 38862 (more bug 38862 than the utf8 bug).

Comment 5

[Max Kanat-Alexander]

I can't make this a blocker, because I don't even know if it's actually even *fixable* in any reasonable way.

Comment 7

[Frédéric Buclin]

Bugzilla 3.2 is restricted to security bugs only. Mass-retargetting to 3.6.

Comment 10

[Frédéric Buclin]

This is not limited to MySQL. The problem is reproducible with PostgreSQL.

Comment 11

[Frédéric Buclin]

This is not limited to MySQL. The problem is reproducible with PostgreSQL.

Comment 12

[Frédéric Buclin]

Bugzilla 4.0 is restricted to security fixes only.

Comment 13

[Jesse Ruderman]

Also affects the commit message of https://bugzilla.mozilla.org/attachment.cgi?id=809337&action=edit

Comment 15

[Peiying Mo [:CocoMo]]

i couldn't search in regular attachment mode so in order to do so, I switched to edit as comment mode.