Last Comment Bug 486306 - Truncated XML-RPC response (incorrect content-length header)
: Truncated XML-RPC response (incorrect content-length header)
Status: RESOLVED FIXED
:
Product: Bugzilla
Classification: Server Software
Component: WebService (show other bugs)
: 3.2.3
: All All
: -- major (vote)
: Bugzilla 3.4
Assigned To: Max Kanat-Alexander
: default-qa
Mentors:
: 499898 706276 (view as bug list)
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2009-04-01 05:40 PDT by nuabaranda
Modified: 2014-02-25 00:21 PST (History)
8 users (show)
LpSolit: approval+
LpSolit: approval3.4+
mkanat: blocking3.4.2+
mkanat: blocking3.4-
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
TCP dump of truncated HTTP request and response. (1.76 KB, application/octet-stream)
2009-04-01 05:41 PDT, nuabaranda
no flags Details
Patch for UTF-8 encoding bug. (1016 bytes, patch)
2009-05-07 10:11 PDT, nuabaranda
mkanat: review-
Details | Diff | Splinter Review
Patch, which fixes problem (924 bytes, patch)
2009-06-23 22:38 PDT, Jochen Wiedmann
no flags Details | Diff | Splinter Review
v1 (1.67 KB, patch)
2009-09-01 15:25 PDT, Max Kanat-Alexander
LpSolit: review+
Details | Diff | Splinter Review
v1 - 3.4 (1.71 KB, patch)
2009-09-01 15:28 PDT, Max Kanat-Alexander
LpSolit: review+
Details | Diff | Splinter Review

Description nuabaranda 2009-04-01 05:40:14 PDT
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 (.NET CLR 3.5.30729)
Build Identifier: Testopia 2.2 on Bugzilla 3.2.2

Longer XML-RPC responses of the Testopia XML-RPC interface are truncated by a few bytes leading to invalid XML. Seems to be an issue of the used SOAP:Lite Perl module, see this bug: https://sourceforge.net/tracker/index.php?func=detail&aid=2723971&group_id=66000&atid=513017



Reproducible: Always

Steps to Reproduce:
1. Request longer information items via XML-RPC.
Actual Results:  
Truncated thus invalid XML code in HTTP response.

Expected Results:  
Valid and complete XML response.

Bugzilla checksetup.pl output: 

* This is Bugzilla 3.2.2 on perl 5.8.8

Checking perl modules...
Checking for              CGI.pm (v3.21)   ok: found v3.42
Checking for            TimeDate (v2.21)   ok: found v2.22
Checking for           PathTools (v0.84)   ok: found v3.29
Checking for                 DBI (v1.41)   ok: found v1.53
Checking for    Template-Toolkit (v2.15)   ok: found v2.20
Checking for          Email-Send (v2.00)   ok: found v2.181
Checking for          Email-MIME (v1.861)  ok: found v1.863
Checking for Email-MIME-Modifier (v1.442)  ok: found v1.443
Checking for                JSON (v2.10)   ok: found v2.12
Checking for           Text-Diff (v0.35)   ok: found v0.35
Checking for          GD-Graph3d (v0.63)   ok: found v0.63

Checking available perl DBD modules...
Checking for              DBD-Pg (v1.45)    not found
Checking for           DBD-mysql (v4.00)   ok: found v4.010
Checking for          DBD-Oracle (v1.19)    not found

The following Perl modules are optional:
Checking for                  GD (v1.20)   ok: found v2.34
Checking for               Chart (v1.0)    ok: found v2.4.1
Checking for         Template-GD (any)     ok: found v1.56
Checking for          GDTextUtil (any)     ok: found v0.86
Checking for             GDGraph (any)     ok: found v1.4308
Checking for            XML-Twig (any)     ok: found v3.26
Checking for          MIME-tools (v5.406)  ok: found v5.420
Checking for         libwww-perl (any)     ok: found v2.033
Checking for         PatchReader (v0.9.4)  ok: found v0.9.5
Checking for          PerlMagick (any)     ok: found v6.2.4
Checking for           perl-ldap (any)     ok: found v0.33
Checking for         Authen-SASL (any)     ok: found v2.12
Checking for          RadiusPerl (any)     ok: found v0.13
Checking for           SOAP-Lite (any)     ok: found v0.710.08
Checking for         HTML-Parser (v3.40)   ok: found v3.55
Checking for       HTML-Scrubber (any)     ok: found v0.08
Checking for Email-MIME-Attachment-Stripper (any)     ok: found v1.314
Checking for         Email-Reply (any)     ok: found v1.202
Checking for            mod_perl (v1.999022) ok: found v2.000002
Checking for            Text-CSV (v1.06)   ok: found v1.10

See attached dump file.
Comment 1 nuabaranda 2009-04-01 05:41:31 PDT
Created attachment 370398 [details]
TCP dump of truncated HTTP request and response.
Comment 2 Nathan Parrish 2009-04-01 11:22:44 PDT
please see http://sourceforge.net/tracker/?func=detail&atid=513017&aid=2723971&group_id=66000 for a possible workaround involving converting UTF8-encoded non-breaking spaces to the text string  
Comment 3 Frédéric Buclin 2009-04-03 09:33:42 PDT
Why is this bug reported in the Bugzilla product?
Comment 4 nuabaranda 2009-04-03 09:48:34 PDT
I can also confirm it for the Bugzilla XML-RPC interface.
Comment 5 nuabaranda 2009-04-06 04:21:50 PDT
Tested both the Bugzilla and the Testopia XML-RPC interface. For both the following observations are true:

If the response contains at least one 3-byte UTF character the response is correctly encoded, valid and not truncated. If it only contains ASCII and 2-byte UTF characters it is handled as ASCII, thus double-encoded and finally the HTTP response is truncated resulting in invalid XML.
Comment 6 Max Kanat-Alexander 2009-04-06 19:25:27 PDT
I am fairly sure that the bug on SF is a Testopia problem, given what it says there.
Comment 7 nuabaranda 2009-04-07 00:02:36 PDT
(In reply to comment #6)
> I am fairly sure that the bug on SF is a Testopia problem, given what it says
> there.

The issue on SF just seems to be a part of the main problem which I described in comment #5. And as I said there I can confirm this issue for both Bugzilla and Testopia XML-RPC interfaces.
Comment 8 Greg Hendricks 2009-04-07 08:27:55 PDT
Since the Testopia XMLRPC interface is a copy of the Bugzilla interface, it would make sense that this issue appears in both. The real question is whether this is an issue with our implementation, or with SOAP::Lite itself.
Comment 9 nuabaranda 2009-05-07 10:09:39 PDT
Finally found the cause: There is a package Bugzilla::WebService::XMLRPC::Serializer in WebService.pm which says it was used to patch some SOAP::Lite bugs. Its method as_string() leads to the double encoding. Seems the bug was fixed in SOAP::Lite in the meantime. See the patch attached.

Maybe there are more such fixes around. For example I haven't tested anything base64-related up to now...
Comment 10 nuabaranda 2009-05-07 10:11:21 PDT
Created attachment 376266 [details] [diff] [review]
Patch for UTF-8 encoding bug.

Commented out method causing the bug.
Comment 11 Max Kanat-Alexander 2009-05-07 13:03:57 PDT
Comment on attachment 376266 [details] [diff] [review]
Patch for UTF-8 encoding bug.

Can you show me where in the Change history of SOAP::Lite this was fixed? If it's fixed, we need to remove this function entirely and require the higher version of SOAP::Lite.
Comment 12 nuabaranda 2009-05-08 00:20:23 PDT
(In reply to comment #11)
> Can you show me where in the Change history of SOAP::Lite this was fixed? If
> it's fixed, we need to remove this function entirely and require the higher
> version of SOAP::Lite.

As there was no SOAP::Lite bug referenced in the comments to this method (as for a couple of other methods) I could not really retrace it in SOAP::Lite. But I wrote a request to the SOAP::Lite maintainer to have a look here and give us some hint.
Comment 13 nuabaranda 2009-05-08 00:32:20 PDT
(In reply to comment #11)
> Can you show me where in the Change history of SOAP::Lite this was fixed? If
> it's fixed, we need to remove this function entirely and require the higher
> version of SOAP::Lite.

Could be this one here: http://rt.cpan.org/Public/Bug/Display.html?id=35041
Comment 14 nuabaranda 2009-05-25 00:05:01 PDT
(In reply to comment #11)
> Can you show me where in the Change history of SOAP::Lite this was fixed?

Here is Martin Kutters answer, he is the SOAP::Lite maintainer: 

this bug is fixed since 0.71.04 released on CPAN on 21-Apr-2008 (since
rev238 in SOAP::Lite's subversion - see
http://soaplite.svn.sourceforge.net/viewvc/soaplite?view=rev&revision=238 for details).

The description of the Bugzilla bug entry sounds like the double
encoding is due to the this fix: Since 0.71.04 SOAP::Lite encodes
UTF8-Characters (no matter how many bytes long) correctly by calling

encode($encoding, $response)

before sending the content out on the wire.

encode($encoding, $characters) returns a sequence of octets (bytes) in
the encoding specified, so SOAP::Lite now sends out a byte-stream in the
encoding specified.

SOAP::Lite actually determines the content length before performing the
encoding (by asking for the bytelength of the string) which may lead to
invalid results when double encoded characters are contained in the
response. This could be construed as a bug, too.
Comment 15 Max Kanat-Alexander 2009-06-02 02:12:55 PDT
Actually, I cannot reproduce. I put several two-byte characters into a bug. In particular, this string as the summary:

ƀƁƂƃƄƅƆƇƈƉƊƋƌƍƎƏƐƑƒƓ

And Bug.get returns the correct XML. SOAP::Lite's XMLRPCsh.pl script correctly decodes the result. I see no truncation whatsoever, and the returned string is correct (XMLRPCsh.pl displays it in Perl Unicode notation):

\x{180}\x{181}\x{182}\x{183}\x{184}\x{185}\x{186}\x{187}\x{188}\x{189}\x{18a}\x{18b}\x{18c}\x{18d}\x{18e}\x{18f}\x{190}\x{191}\x{192}\x{193}

There could not be any double-encoding by us, because the Serializer code that we wrote checks utf8::is_utf8 first before encoding, so if it was already encoded, it would not encode it again.

I tested this all with SOAP::Lite 0.71.07.
Comment 16 nuabaranda 2009-06-02 02:20:56 PDT
Are you sure that there is no other 3-byte-character in the response? If this is the case, the encoding is correct. Could you try a single german umlaut as ä or ü or ö?
Comment 17 nuabaranda 2009-06-02 02:24:01 PDT
Did you have a look at my TCP dump which clearly shows the truncation? Could you attach a dump of your test?
Comment 18 nuabaranda 2009-06-02 02:40:47 PDT
(In reply to comment #15)
> There could not be any double-encoding by us, because the Serializer code that
> we wrote checks utf8::is_utf8 first before encoding, so if it was already
> encoded, it would not encode it again.

It's the other way around: you get it from the Bugzilla database and the fix I commented out in my patch encodes it the first time. Then the response is passed to SOAP::Lite and there the second encoding happens. In previous SOAP::Lite releases this (second) encoding was missing but was added in between time. And this leads to the bug. 

In general, as I understand it, Bugzilla should not care about encoding issues. The underlying communication libraries such as SOAP::Lite should do it (or pass it on). So when you add bugfixes for problems in underlying libraries you should carefully regression-test for them being fixed there.
Comment 19 Max Kanat-Alexander 2009-06-23 17:48:55 PDT
*** Bug 499898 has been marked as a duplicate of this bug. ***
Comment 20 Jochen Wiedmann 2009-06-23 22:38:10 PDT
Created attachment 384817 [details] [diff] [review]
Patch, which fixes problem

Please reopen this bug. I can clearly reproduce it and, what's more, I am able to resolve it with the attached patch. The patch seems basically to do the same than the previous proposed patch, just that the location has changed. Indeed, removing the as_string method in XMLRPC.pm seems to fix my problem.
Comment 21 Jochen Wiedmann 2009-06-23 22:39:34 PDT
(In reply to comment #20)
> Created an attachment (id=384817) [details]
> Patch, which fixes problem
> 
> Please reopen this bug. I can clearly reproduce it and, what's more, I am able
> to resolve it with the attached patch. The patch seems basically to do the same
> than the previous proposed patch, just that the location has changed. Indeed,
> removing the as_string method in XMLRPC.pm seems to fix my problem.

Forgot to note: My patch is against Bugzilla 3.3.4.
Comment 22 Jochen Wiedmann 2009-06-23 22:45:11 PDT
Max, regarding your comment in bug 499898:

I am ready to pass the contents of my current database (just about 261K) to you privately. (Customer data, I cannot tell how sensitive it is) I can also pass the complete client code, which is nothing but the test suite of a proposed Perl client for the Bugzilla webservices API.
Comment 23 Max Kanat-Alexander 2009-06-24 00:08:04 PDT
Hey Jochen. Okay, could you send everything to my email address?
Comment 24 Max Kanat-Alexander 2009-09-01 15:15:24 PDT
Okay, I can reproduce this now, and it's definitely a problem. Anything containing non-ASCII characters doesn't work.
Comment 25 Max Kanat-Alexander 2009-09-01 15:25:40 PDT
Created attachment 397988 [details] [diff] [review]
v1

Okay, here we go. It should be OK to require 0.710.06--it was released in 2008.
Comment 26 Max Kanat-Alexander 2009-09-01 15:28:34 PDT
Created attachment 397989 [details] [diff] [review]
v1 - 3.4

And here's the same patch for the branch.
Comment 27 Frédéric Buclin 2009-09-02 06:39:13 PDT
Comment on attachment 397988 [details] [diff] [review]
v1

>Index: Bugzilla/Install/Requirements.pm

>+        # 0.710.04 is required for correct UTF-8 handling

Nit: It's 0.710.03 or better, as far as I know.

Otherwise, this indeed fixes the problem. r=LpSolit
Comment 28 Max Kanat-Alexander 2009-09-04 14:26:00 PDT
(In reply to comment #27)
> Nit: It's 0.710.03 or better, as far as I know.

  I got 0.710.04 from comment 14, which has info from the SOAP::Lite maintainer.
Comment 29 Max Kanat-Alexander 2009-09-04 14:30:12 PDT
tip:

Checking in Bugzilla/Install/Requirements.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/Install/Requirements.pm,v  <--  Requirements.pm
new revision: 1.69; previous revision: 1.68
done
Checking in Bugzilla/WebService/Server/XMLRPC.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/WebService/Server/XMLRPC.pm,v  <--  XMLRPC.pm
new revision: 1.6; previous revision: 1.5
done


3.4:

Checking in Bugzilla/Install/Requirements.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/Install/Requirements.pm,v  <--  Requirements.pm
new revision: 1.60.2.5; previous revision: 1.60.2.4
done
Checking in Bugzilla/WebService/Server/XMLRPC.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/WebService/Server/XMLRPC.pm,v  <--  XMLRPC.pm
new revision: 1.1.2.3; previous revision: 1.1.2.2
done
Comment 30 m 2012-05-09 16:49:35 PDT Comment hidden (spam)
Comment 31 m 2012-05-09 16:49:35 PDT Comment hidden (spam)
Comment 32 m 2012-05-09 16:49:35 PDT Comment hidden (spam)
Comment 33 m 2012-05-09 16:49:35 PDT Comment hidden (spam)
Comment 34 m 2012-05-09 16:49:36 PDT Comment hidden (spam)
Comment 35 Jochen Wiedmann 2012-05-09 23:41:23 PDT
Just dor a suggestion: It might make sense to always set content-length=-1. That way the client wouldn't expect it to terminate at a given point.
Comment 36 Adrián 2012-06-08 02:13:47 PDT
Hi, I think this bug should be reopened. I am having still this bug in version 4.2.1.

Bug #706276 should be marked as duplicated with this one.

As a workaround, I have removed the "Content-Length" header in the XMLRPC.pm file, and server automatically uses Chunked transfer encoding.

The patch for version 4.2.1:
--- bugzilla-4.2.1/Bugzilla/WebService/Server/XMLRPC.pm.orig    2012-04-18 22:32:47.000000000 +0200
+++ bugzilla-4.2.1/Bugzilla/WebService/Server/XMLRPC.pm 2012-06-08 10:57:50.827070643 +0200
@@ -64,6 +64,7 @@
     foreach (@{Bugzilla->cgi->{'Bugzilla_cookie_list'}}) {
         $self->response->headers->push_header('Set-Cookie', $_);
     }
+    $self->response->headers->remove_header("Content-Length");
 }

 sub handle_login {


This patch is also correct for the last XMLRPC.pm file in trunk (version 8259), except for the numbers of lines (the new version has less license lines in the beginning of the file).
Comment 37 Wayne 2013-04-07 13:15:46 PDT
*** Bug 706276 has been marked as a duplicate of this bug. ***
Comment 38 Wayne 2014-01-15 09:09:24 PST
After upgrading SOAP::Lite to 1.08 and XMLRPC::Lite 0.717 on Windows I can confirm that this still exists in Bugzilla 4.2.7 and 4.4.1.
It does seem to be fixed in Bugzilla 4.2.6 with SOAP::Lite 1.06 on Ubuntu 12.04.
Comment 39 Thomas Stegh 2014-02-24 04:44:05 PST
Hi, I also think this bug should be reopened. 

I still get this bug in Bugzilla version 4.4.2 with SOAP-Lite 1.10, XMLRPC-Lite 0.717 on Windows

The workaround in Comment 36 did the trick for me.
Comment 40 Frédéric Buclin 2014-02-24 06:02:29 PST
We are not going to reopen this bug. The patch has been checked in a long time ago. File a separate bug if you have exact steps to reproduce the issue.
Comment 41 Jochen Wiedmann 2014-02-25 00:21:32 PST
Wayne, Thomas: We'll definitely need more input. See, for example, the attachment from comment #1. It is unlikely, that we'll be able to reproduce this bug, but input like this should be able to provide a starting point.

Note You need to log in before you can comment on or make changes to this bug.