839023 - xmlrpc interface doesn't correctly escape data in response, resulting in invalid xml

Status: RESOLVED WONTFIX

(Bugzilla :: WebService, defect)

Description

[Benjamin Smedberg]

https://api-dev.bugzilla.mozilla.org/1.2/bug?id=4843&include_fields=_default,history consistently shows me a "Please come back later" message. Viewing the bug from b.m.o works, and most other bugs don't have this problem.

There may be a few other bugs which have a similar problem; I'm diagnosing errors retrieving about 60 bugs total.

Comment 1

[Gervase Markham [:gerv]]

Attachment: XML RPC test program showing bug (need to insert username and password)

Comment 2

[Gervase Markham [:gerv]]

This is a problem with Bugzilla's XML RPC "history" method returning invalid XML character data. Punting over to bugzilla.mozilla.org component for glob to take a look at, although we may finally find it's an upstream bug.

Gerv

Comment 3

[Benjamin Smedberg]

Also affected:

bug 13867
bug 140803
bug 148674
bug 15236
bug 188847
bug 191053
bug 23120
bug 298693
bug 30090
bug 313030
bug 51322
bug 67914

Comment 4

[:glob ✱]

Attachment: generated xml

there's the xml bugzilla generates as a response.
at char 421 there's an unescaped char(3).

Comment 5

[Gervase Markham [:gerv]]

So the issue here is that this char should have been replaced by an  or something like that? I'd expect our XML encoding library, whatever it is, to handle that stuff...

Gerv

Comment 6

[Gervase Markham [:gerv]]

U+0000-U+001F are illegal in HTML 4.0 and XML 1.0 (except the characters HR, LF and CR). And it's not permitted to use named character references such as  either (although it is permitted in XML 1.1, except for NUL): http://www.w3.org/International/questions/qa-controls

So what should we do?

1a) Silently filter out such characters whenever data is submitted?
1b) Throw an error whenever data is submitted with such characters?
2) Filter out such characters whenever data is displayed?
3) Both 1) and 2)?
4) Filter on submission, and do a one-time DB scan to remove any existing ones?
5) Nothing?

Gerv

Comment 7

[Gervase Markham [:gerv]]

Related: bug 538946.

Gerv

Comment 8

[Benjamin Smedberg]

Given that we already have such data in the DB, why not just filter on output and replace them with U+FFFD (unicode unknown character)?

Comment 9

[Gervase Markham [:gerv]]

LpSolit: if we wanted to do what Ben suggests in comment 8, where in the code do you think we could put such a filter? There may be quite a few fields and data items which need filtering...

Gerv

Comment 10

[Justin Fletcher]

I needed to fix this problem on a Bugzilla server (version 4.0.2) which I maintain where I don't care about the content in the XML-RPC response that contains the data. Inside Bugzilla/WebService/Server/XMLRPC.pm, in _strip_undefs, at the end of the function (around line 250):

    if (ref $initial eq '')
    {
     	$initial =~ s/([\x01-\x08\x0b\x0c\x0f-\x1f])/sprintf "\\x%02x", ord($1)/ge;
    }

This doesn't replace them with the 'unknown character', but with plain ascii in the form '\x##'. This may not be sufficient for some users, but it ensures that my xmlrpc client doesn't fail on the reading of the XML-RPC result.

Comment 12

[Andre Klapper]

(In reply to Justin Fletcher from comment #10)
> where I don't care about the content in the XML-RPC response that contains the data.
>     if (ref $initial eq '')
>     {
>      	$initial =~ s/([\x01-\x08\x0b\x0c\x0f-\x1f])/sprintf "\\x%02x",
> ord($1)/ge;
>     }

For anybody else trying this workaround: It works around the problem for comment text but obviously damages binary attachment content. Wondering if there's any way to only apply this to comment text.

Comment 15

[Ahmed Akhtar]

Now what happens. This bug is still persistent. Is it going to be fixed ever?

Comment 16

[Andre Klapper]

@Ahmed: No-one can predict the future. Generally speaking, bugs get fixed faster when someone interested in fixing provides a code patch for review. Would you fancy doing that? See https://wiki.mozilla.org/Bugzilla:Patches for more information if you are interested.

Comment 17

[Ahmed Akhtar]

Thanks a lot. I will look into that.

Comment 18

[Kohei Yoshino [:kohei]]

The JSON-RPC and XML-RPC APIs are deprecated and will no longer be updated. Feel free to file a new bug if the same issue can be reproduced with the REST API.

https://bugzilla.readthedocs.io/en/latest/integrating/apis.html