568503 - charset is not set for application/* types (XML, JSON, Atom, RDF, Javascript...)

Status: RESOLVED FIXED

(Bugzilla :: Bugzilla-General, defect)

Description

[Gervase Markham [:gerv]]

Even when Bugzilla is using UTF-8, config.cgi does not set a charset in the Content-Type header. This means that data will be by default interpreted by clients as ISO-8859-1, which is bad (dataloss).

We should make config.cgi send "charset=utf-8" in the Content-Type header if Bugzilla is using UTF-8.

[ However, note that it's only legal to set a charset in a Content-Type header if you have already set a content type. Therefore, if $format->{'ctype'} is "", i.e. Bugzilla does not know about the ctype being used, adding the charset parameter to the header won't work, and clients will still default to ISO-8859-1. Another reason why Bugzilla should know about all common ctypes. ]

Gerv

Comment 1

[Gervase Markham [:gerv]]

Hmm. I originally wrote a config.cgi-specific patch for this for 3.4 (and the bit in square brackets above makes sense for the 3.4 code). However, 3.6 now uses Bugzilla->cgi->header(). And our CGI.pm sets the charset to UTF-8 whenever the utf8 parameter is turned on. So why doesn't it work?

It turns out Perl's CGI.pm only puts its internal charset into the header for text types :-( Or at least, it doesn't do it for application/json or application/xml. That rather sucks.

I'm not sure what the correct fix is here. We already have a bit of "CGI.pm doesn't add the charset; add it ourselves" code in multipart_start(). So we could do that for all application/ types too... I don't really want to special-case XML and JSON in our CGI code!

Max?

Gerv

Comment 2

[Max Kanat-Alexander]

  Hmm. I'd suggest filing a bug against CGI.pm and see what the maintainer says. They've been very responsive lately.

Comment 3

[Gervase Markham [:gerv]]

Bug filed: https://rt.cpan.org/Public/Bug/Display.html?id=57945 

I've noticed that specifying the charset explicitly on the header() call does work, so we could work around this by hacking Bugzilla::CGI->header().

Gerv

Comment 4

[Gervase Markham [:gerv]]

Max: doesn't look like the CGI.pm team are going to fix this... what do we do now?

Gerv

Comment 5

[Max Kanat-Alexander]

Okay, that's annoying. Sounds like we should monkeypatch header() (which we already do anyway) to add the charset always--at least for certain types?

Comment 6

[Gervase Markham [:gerv]]

Sounds like a plan to me. 

Gerv

Comment 7

[Reed Loden [:reed]]

CGI.pm 3.51 fixes this.

Comment 8

[Matt Selsky [:selsky]]

Trunk requires CGI 3.51, so is this fixed?

Comment 9

[Frédéric Buclin]

(In reply to Matt Selsky [:selsky] from comment #8)
> Trunk requires CGI 3.51, so is this fixed?

CGI 3.51 is required since Bugzilla 3.2.10, see bug 591165, so yes, this problem is now fixed.