415796 - All webservice methods fail if a string has multibyte characters in it

Status: RESOLVED FIXED

(Bugzilla :: WebService, defect, P1)

Description

[Frédéric Buclin]

I just tested the existing Bug.get and the new User.get webservice methods in 3.1.3+ (but I suspect the same problem exists in 3.0.3+), and they both fail if a string has non-ASCII characters in it (such as "Frédéric"). Despite the returned XML file starts with <?xml version="1.0" encoding="UTF-8"?>, such strings are mangled, generating the following error:

not well-formed (invalid token) at line 1, column 203, byte 203 at /usr/lib/perl5/vendor_perl/5.8.8/i386-linux/XML/Parser.pm line 187

So for some reason, strings we return are not valid UTF-8 strings. Did we forget to turn on a parameter somewhere?

Requesting blocking as non-US installations use a lot of non-ASCII characters, in bug summaries, product names, real names, etc..., making webservice unusable.

Comment 1

[Max Kanat-Alexander]

This is definitely a Bugzilla 3.2 blocker. I suspect the same problem does not exist in 3.0, since we changed the utf8 bit in 3.2. We would have to actually check 3.0 before making this a blocker or targeting it at 3.0.

Comment 2

[Frédéric Buclin]

OK, I just tested it on 3.0.3 and I cannot reproduce the problem there. So definitely a 3.2-problem only. Good.

Comment 3

[Frédéric Buclin]

Marc, any idea how to fix this problem? That's the last blocker we want to fix before releasing 3.1.4.

Comment 4

[Marc Schumann [:Wurblzap]]

I looked into this, but I can't seem to get a grip on this. I don't think I can help much here.

Comment 5

[Max Kanat-Alexander]

Okay, I suppose I'll have to take it then, unless himorin has any ideas.

Comment 6

[A. Shimono [:himorin]]

i'll look into this bug. (from now.)

Comment 7

[A. Shimono [:himorin]]

With cvs tip at Feb 7, i've got xml-rpc error message like
<?xml version="1.0" encoding="UTF-8"?>
<methodResponse><
fault><value><struct><member><name>faultString</name>
<value><string>Wide character in subroutine entry at /usr/lib/perl5/site_perl/5.8.5/XMLRPC/Lite.pm line 167.</string></value>
</member><member><name>faultCode</name>
<value><string>Client</string></value></member></struct></value>
</fault></methodResponse>.

I'm not sure that this is the same one which LpSolit describes.
Bug, it's not related with utf8 string.
For method Bug.get, this warning occured with the bug object 
>         $item{'internals'}        = $bug;
$bug is from new Bugzilla::Bug($bug_id).

Comment 8

[A. Shimono [:himorin]]

ah, i forget to comment about utf8-ed string.
when i commented out Bug.get internals, i could get the normal responce from the xml-rpc interface.

Comment 9

[A. Shimono [:himorin]]

as talking at irc, this is not a bug related on utf8=1.
so, my comment #7 is not related with this bug.

Comment 10

[Max Kanat-Alexander]

I've started looking into this. The problem is that internally SOAP is serializing data into Base64 and then deserializing it into byte strings, which turns off the utf8 bit incorrectly. Fixing this behavior is proving to be somewhat difficult.

Comment 11

[A. Shimono [:himorin]]

(In reply to comment #10)
> I've started looking into this. The problem is that internally SOAP is
> serializing data into Base64 and then deserializing it into byte strings, which
> turns off the utf8 bit incorrectly. Fixing this behavior is proving to be
> somewhat difficult.

mkanat, the problem you've described on comment #10 seems to be the same as my comment #7, but not the original LpSolit's.
i think you should try with utf8=0.

Comment 12

[Max Kanat-Alexander]

Attachment: WIP

Okay, this fixes the XMLRPC data type serializer (which is the problem, see the bug referenced in the code), but seems to cause some other problem that I haven't figured out yet.

Comment 13

[Max Kanat-Alexander]

Attachment: v1

This fixes it.

The problem was that some data ("internals" in our case) wasn't being passed through SOAP::Data->type, so it was getting auto-typed. This was turning our utf-8 strings into base64. Then, when they were deserialized on the other end (back into strings) they didn't have the utf-8 bit set on them. This was supposed to be fixed in SOAP::Lite 0.71, but it doesn't seem to be.

So, I had to subclass XMLRPC::Serializer to fix it. HOWEVER, there's another bug in SOAP::Lite that causes array datatypes to break if you subclass XMLRPC::Serializer, so I had to work around that:

http://rt.cpan.org/Ticket/Display.html?id=34515

Finally, this code now throws a warning, but seems to work correctly. I've looked at the raw XML and it's correct. The warning seems to be a bug in SOAP::Lite:

http://rt.cpan.org/Ticket/Display.html?id=34515

Comment 14

[Max Kanat-Alexander]

That first RT link should be:

http://rt.cpan.org/Ticket/Display.html?id=34514

Comment 15

[Frédéric Buclin]

In debug mode, I get (on my local installation):

SOAP::Parser::decode: ()
Can't call method "paramsall" on an undefined value at /usr/bin/XMLRPCsh.pl line 23, <> line 1.

Comment 16

[Max Kanat-Alexander]

(In reply to comment #15)
> In debug mode, I get (on my local installation):

  Hmm, but it should have printed out the XML before that. Could you attach the XML that it's getting that throws that error? (Also, what version of SOAP::Lite?)

Comment 17

[Frédéric Buclin]

Attachment: debug log

Using SOAP::Lite 0.69, XML::Parser 2.34.

Comment 18

[Max Kanat-Alexander]

Comment on attachment 312447 [details]
debug log

This attachment seems to contain characters that aren't valid UTF-8. 

Do you have the utf8 flag set on?

Any other idea as to how invalid UTF-8 could have gotten here? (Is the database correctly recoded?)

Comment 19

[Frédéric Buclin]

(In reply to comment #18)
> Do you have the utf8 flag set on?

Of course I have.


> Any other idea as to how invalid UTF-8 could have gotten here? (Is the database
> correctly recoded?)

AFAIK, the data is correctly encoded.

Comment 20

[Frédéric Buclin]

(In reply to comment #19)
> AFAIK, the data is correctly encoded.

If it wasn't correctly encoded, then é, à, etc... would be badly displayed in pages, as they use the UTF-8 encoding.

Comment 21

[Max Kanat-Alexander]

(In reply to comment #20)
> If it wasn't correctly encoded, then é, à, etc... would be badly displayed in
> pages, as they use the UTF-8 encoding.

  As they are. Open up the attachment in Firefox, set Firefox to UTF-8, and look at the XML. It shows up as misencoded for me.

Comment 22

[Frédéric Buclin]

(In reply to comment #21)
>   As they are. Open up the attachment in Firefox, set Firefox to UTF-8, and
> look at the XML. It shows up as misencoded for me.

That's what I got when using webservice. Viewing and saving the bug as XML displays all characters correctly with UTF-8. So the problem occurs somewhere between the server and the client, not in my DB itself.

Comment 23

[Frédéric Buclin]

Comment on attachment 312412 [details] [diff] [review]
v1

This fixes the problem with UTF-8 characters, but still doesn't fix the problem with upper-ASCII characters, such as é, à, ü. I will clone the bug to fix it separately and mark it as a blocker. r=LpSolit

Comment 24

[Frédéric Buclin]

(In reply to comment #23)
> with upper-ASCII characters, such as é, à, ü. I will clone the bug

I filed bug 426899. Please check in this patch, Max. :)

Comment 25

[Max Kanat-Alexander]

Yeah, I was going to check this in the other day but I had to wait for justdave to do his CVS changes, and then I haven't had the opportunity since then.

Checking in xmlrpc.cgi;
/cvsroot/mozilla/webtools/bugzilla/xmlrpc.cgi,v  <--  xmlrpc.cgi
new revision: 1.5; previous revision: 1.4
done
Checking in Bugzilla/WebService.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/WebService.pm,v  <--  WebService.pm
new revision: 1.7; previous revision: 1.6
done