1055629 - Invalid byte sequence crash when accessing certain tickets via XML RPC API: "not well-formed (invalid token)"

Status: RESOLVED DUPLICATE of bug 839023

(Bugzilla :: WebService, defect)

Description

[Andre Klapper]

Attachment: Python script testcase

Upstreaming from https://bugzilla.wikimedia.org/show_bug.cgi?id=69747
Bug 315535 was probably the same issue.
Potential dup/related: Bug 538946 / bug 784308

Numerous bug reports in bugzilla.mozilla.org (and not only, also in our instance at bugzilla.wikimedia.org) include entertaining UTF-8 data. 

Two examples from this instance:
* https://bugzilla.mozilla.org/show_bug.cgi?id=771213#c0
* https://bugzilla.mozilla.org/show_bug.cgi?id=752627#c0

STEPS TO REPRODUCE:
Try to access these two example tickets via XML-RPC API by calling the attached Python script, via:
     time seq 771211 771215 | python minimal-bz.py
     time seq 752625 752629 | python minimal-bz.py


This will reproducibly trigger the following error:

body: '<?xml version="1.0" encoding="UTF-8"?><methodResponse><params><param><value><struct><member><name>bugs</name><value><struct><member><name>9444</name><value><struct><member><name>comments</name><value><array><data><value><struct><member><name>is_private</name><value><boolean>0</boolean></value></member><member><name>count</name><value><int>0</int></value></member><member><name>creator</name><value><string>[e-mail]</string></value></member><member><name>time</name><value><dateTime.iso8601>20070329T08:11:13</dateTime.iso8601></value></member><member><name>bug_id</name><value><int>9444</int></value></member><member><name>author</name><value><string>[e-mail]</string></value></member><member><name>text</name><value><string>A database error has occurred Query: SELECT\nmath_outputhash,math_html_conservativeness,math_html,math_mathml FROM math WHERE\nmath_inputhash = \'\xef\xbf\xbd\xef\xbf\xbd\xd7\xbe\xef\xbf\xbd\x1f\x11\xef\xbf\xbd\xef\xbf\xbd\x12@\x01\xcb\xb5\' LIMIT 1 Function: MathRenderer::_recall Error: 1\nERROR: invalid byte sequence for encoding "UTF8": 0xebc3d'
Traceback (most recent call last):
  File "minimal.py", line 64, in <module>
    fetch(i)
  File "minimal.py", line 49, in fetch
    com = server.Bug.comments(kwargs)['bugs'][bugid]['comments']
  File "/usr/lib/python2.7/xmlrpclib.py", line 1224, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1578, in __request
    verbose=self.__verbose
  File "/usr/lib/python2.7/xmlrpclib.py", line 1264, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1297, in single_request
    return self.parse_response(response)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1467, in parse_response
    p.feed(data)
  File "/usr/lib/python2.7/xmlrpclib.py", line 557, in feed
    self._parser.Parse(data, 0)
xml.parsers.expat.ExpatError: not well-formed (invalid token): line 3, column 22


body: 'edStreamException: Stream ended unexpectedly\n\nFirst tried on Win7 Firefox 12. Failed. First time noticing this problem. Previous Firefox version never had this problem. German locale.\n\nUsing Firefox 11 on Ubuntu this worked fine. After upgrading to Firefox 12 the upload failed. German locale.\n\nHere is excerpt from Firebug (network tab): \n\nRequest headers:\n\nAccept-Language\tde-de,de;q=0.8,en-us;q=0.5,en;q=0.3\nConnection\tkeep-alive\nCookie\tJSESSIONID=CC01257D51EDCE27EF70A143CBC6687D\nDNT\t1\nHost\tservice.guidedshop-saas.de\nReferer\thttp://service.****.de\nUser-Agent\tMozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20100101 Firefox/12.0\nRequest Headers From Upload Stream\nContent-Length\t39450\nContent-Type\tmultipart/form-data; \n\nPOST data:\nboundary=---------------------------114782935826962\n\nTeile           multipart/form-data\nuploadfile\tOggS\xef\xbf\xbd\x02\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd|2\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbdi\x19\xc3\xbe\x08\x01@fishead\xef\xbf\xbd\x03\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xc3\xa8\x03\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xc3\xa8\x03\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd'
Traceback (most recent call last):
  File "minimal-bz.py", line 65, in <module>
    fetch(i)
  File "minimal-bz.py", line 50, in fetch
    com = server.Bug.comments(kwargs)['bugs'][bugid]['comments']
  File "/usr/lib/python2.7/xmlrpclib.py", line 1224, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1578, in __request
    verbose=self.__verbose
  File "/usr/lib/python2.7/xmlrpclib.py", line 1264, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1297, in single_request
    return self.parse_response(response)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1467, in parse_response
    p.feed(data)
  File "/usr/lib/python2.7/xmlrpclib.py", line 557, in feed
    self._parser.Parse(data, 0)
xml.parsers.expat.ExpatError: not well-formed (invalid token): line 38, column 16

Comment 1

[Teemu Mannermaa (:wicked)]

Sounds like a problem in the WebService API itself, not Database as the normal view works fine. Especially if JSON-RPC works. Bug 868867 might be related if the Unicode chars are freakishly large.

Comment 2

[Andre Klapper]

Any ideas how to make the WebService more robust?

Like replacing or dropping problematic characters somewhere before
  text       => $self->type('string', $comment->body_full),
in
  http://bzr.mozilla.org/bugzilla/4.4/view/head:/Bugzilla/WebService/Bug.pm#L296
I guess?

Comment 3

[Frédéric Buclin]

(In reply to Andre Klapper from comment #2)
> Any ideas how to make the WebService more robust?

I guess Bugzilla::XMLRPC::Serializer.