Closed
Bug 486306
Opened 15 years ago
Closed 15 years ago
Truncated XML-RPC response (incorrect content-length header)
Categories
(Bugzilla :: WebService, defect)
Tracking
()
RESOLVED
FIXED
Bugzilla 3.4
People
(Reporter: nuabaranda, Assigned: mkanat)
References
Details
Attachments
(2 files, 3 obsolete files)
1.67 KB,
patch
|
LpSolit
:
review+
|
Details | Diff | Splinter Review |
1.71 KB,
patch
|
LpSolit
:
review+
|
Details | Diff | Splinter Review |
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 (.NET CLR 3.5.30729) Build Identifier: Testopia 2.2 on Bugzilla 3.2.2 Longer XML-RPC responses of the Testopia XML-RPC interface are truncated by a few bytes leading to invalid XML. Seems to be an issue of the used SOAP:Lite Perl module, see this bug: https://sourceforge.net/tracker/index.php?func=detail&aid=2723971&group_id=66000&atid=513017 Reproducible: Always Steps to Reproduce: 1. Request longer information items via XML-RPC. Actual Results: Truncated thus invalid XML code in HTTP response. Expected Results: Valid and complete XML response. Bugzilla checksetup.pl output: * This is Bugzilla 3.2.2 on perl 5.8.8 Checking perl modules... Checking for CGI.pm (v3.21) ok: found v3.42 Checking for TimeDate (v2.21) ok: found v2.22 Checking for PathTools (v0.84) ok: found v3.29 Checking for DBI (v1.41) ok: found v1.53 Checking for Template-Toolkit (v2.15) ok: found v2.20 Checking for Email-Send (v2.00) ok: found v2.181 Checking for Email-MIME (v1.861) ok: found v1.863 Checking for Email-MIME-Modifier (v1.442) ok: found v1.443 Checking for JSON (v2.10) ok: found v2.12 Checking for Text-Diff (v0.35) ok: found v0.35 Checking for GD-Graph3d (v0.63) ok: found v0.63 Checking available perl DBD modules... Checking for DBD-Pg (v1.45) not found Checking for DBD-mysql (v4.00) ok: found v4.010 Checking for DBD-Oracle (v1.19) not found The following Perl modules are optional: Checking for GD (v1.20) ok: found v2.34 Checking for Chart (v1.0) ok: found v2.4.1 Checking for Template-GD (any) ok: found v1.56 Checking for GDTextUtil (any) ok: found v0.86 Checking for GDGraph (any) ok: found v1.4308 Checking for XML-Twig (any) ok: found v3.26 Checking for MIME-tools (v5.406) ok: found v5.420 Checking for libwww-perl (any) ok: found v2.033 Checking for PatchReader (v0.9.4) ok: found v0.9.5 Checking for PerlMagick (any) ok: found v6.2.4 Checking for perl-ldap (any) ok: found v0.33 Checking for Authen-SASL (any) ok: found v2.12 Checking for RadiusPerl (any) ok: found v0.13 Checking for SOAP-Lite (any) ok: found v0.710.08 Checking for HTML-Parser (v3.40) ok: found v3.55 Checking for HTML-Scrubber (any) ok: found v0.08 Checking for Email-MIME-Attachment-Stripper (any) ok: found v1.314 Checking for Email-Reply (any) ok: found v1.202 Checking for mod_perl (v1.999022) ok: found v2.000002 Checking for Text-CSV (v1.06) ok: found v1.10 See attached dump file.
Reporter | ||
Comment 1•15 years ago
|
||
Reporter | ||
Updated•15 years ago
|
Version: unspecified → 2.2
Comment 2•15 years ago
|
||
please see http://sourceforge.net/tracker/?func=detail&atid=513017&aid=2723971&group_id=66000 for a possible workaround involving converting UTF8-encoded non-breaking spaces to the text string
Reporter | ||
Updated•15 years ago
|
Component: API → WebService
Product: Testopia → Bugzilla
Version: 2.2 → 3.2.2
Comment 3•15 years ago
|
||
Why is this bug reported in the Bugzilla product?
Reporter | ||
Comment 4•15 years ago
|
||
I can also confirm it for the Bugzilla XML-RPC interface.
Reporter | ||
Comment 5•15 years ago
|
||
Tested both the Bugzilla and the Testopia XML-RPC interface. For both the following observations are true: If the response contains at least one 3-byte UTF character the response is correctly encoded, valid and not truncated. If it only contains ASCII and 2-byte UTF characters it is handled as ASCII, thus double-encoded and finally the HTTP response is truncated resulting in invalid XML.
Assignee | ||
Comment 6•15 years ago
|
||
I am fairly sure that the bug on SF is a Testopia problem, given what it says there.
Component: WebService → API
Product: Bugzilla → Testopia
Version: 3.2.2 → 2.2
Reporter | ||
Comment 7•15 years ago
|
||
(In reply to comment #6) > I am fairly sure that the bug on SF is a Testopia problem, given what it says > there. The issue on SF just seems to be a part of the main problem which I described in comment #5. And as I said there I can confirm this issue for both Bugzilla and Testopia XML-RPC interfaces.
Comment 8•15 years ago
|
||
Since the Testopia XMLRPC interface is a copy of the Bugzilla interface, it would make sense that this issue appears in both. The real question is whether this is an issue with our implementation, or with SOAP::Lite itself.
Reporter | ||
Comment 9•15 years ago
|
||
Finally found the cause: There is a package Bugzilla::WebService::XMLRPC::Serializer in WebService.pm which says it was used to patch some SOAP::Lite bugs. Its method as_string() leads to the double encoding. Seems the bug was fixed in SOAP::Lite in the meantime. See the patch attached. Maybe there are more such fixes around. For example I haven't tested anything base64-related up to now...
Component: API → WebService
OS: Linux → All
Product: Testopia → Bugzilla
Version: 2.2 → 3.2.3
Reporter | ||
Comment 10•15 years ago
|
||
Commented out method causing the bug.
Assignee | ||
Updated•15 years ago
|
Attachment #376266 -
Flags: review-
Assignee | ||
Comment 11•15 years ago
|
||
Comment on attachment 376266 [details] [diff] [review] Patch for UTF-8 encoding bug. Can you show me where in the Change history of SOAP::Lite this was fixed? If it's fixed, we need to remove this function entirely and require the higher version of SOAP::Lite.
Assignee | ||
Updated•15 years ago
|
Hardware: x86 → All
Reporter | ||
Comment 12•15 years ago
|
||
(In reply to comment #11) > Can you show me where in the Change history of SOAP::Lite this was fixed? If > it's fixed, we need to remove this function entirely and require the higher > version of SOAP::Lite. As there was no SOAP::Lite bug referenced in the comments to this method (as for a couple of other methods) I could not really retrace it in SOAP::Lite. But I wrote a request to the SOAP::Lite maintainer to have a look here and give us some hint.
Reporter | ||
Comment 13•15 years ago
|
||
(In reply to comment #11) > Can you show me where in the Change history of SOAP::Lite this was fixed? If > it's fixed, we need to remove this function entirely and require the higher > version of SOAP::Lite. Could be this one here: http://rt.cpan.org/Public/Bug/Display.html?id=35041
Reporter | ||
Comment 14•15 years ago
|
||
(In reply to comment #11) > Can you show me where in the Change history of SOAP::Lite this was fixed? Here is Martin Kutters answer, he is the SOAP::Lite maintainer: this bug is fixed since 0.71.04 released on CPAN on 21-Apr-2008 (since rev238 in SOAP::Lite's subversion - see http://soaplite.svn.sourceforge.net/viewvc/soaplite?view=rev&revision=238 for details). The description of the Bugzilla bug entry sounds like the double encoding is due to the this fix: Since 0.71.04 SOAP::Lite encodes UTF8-Characters (no matter how many bytes long) correctly by calling encode($encoding, $response) before sending the content out on the wire. encode($encoding, $characters) returns a sequence of octets (bytes) in the encoding specified, so SOAP::Lite now sends out a byte-stream in the encoding specified. SOAP::Lite actually determines the content length before performing the encoding (by asking for the bytelength of the string) which may lead to invalid results when double encoded characters are contained in the response. This could be construed as a bug, too.
Assignee | ||
Updated•15 years ago
|
Flags: blocking3.4+
Target Milestone: --- → Bugzilla 3.2
Assignee | ||
Updated•15 years ago
|
Assignee: ghendricks → webservice
QA Contact: default-qa → default-qa
Assignee | ||
Comment 15•15 years ago
|
||
Actually, I cannot reproduce. I put several two-byte characters into a bug. In particular, this string as the summary: ƀƁƂƃƄƅƆƇƈƉƊƋƌƍƎƏƐƑƒƓ And Bug.get returns the correct XML. SOAP::Lite's XMLRPCsh.pl script correctly decodes the result. I see no truncation whatsoever, and the returned string is correct (XMLRPCsh.pl displays it in Perl Unicode notation): \x{180}\x{181}\x{182}\x{183}\x{184}\x{185}\x{186}\x{187}\x{188}\x{189}\x{18a}\x{18b}\x{18c}\x{18d}\x{18e}\x{18f}\x{190}\x{191}\x{192}\x{193} There could not be any double-encoding by us, because the Serializer code that we wrote checks utf8::is_utf8 first before encoding, so if it was already encoded, it would not encode it again. I tested this all with SOAP::Lite 0.71.07.
Status: UNCONFIRMED → RESOLVED
Closed: 15 years ago
Flags: blocking3.4+ → blocking3.4-
Resolution: --- → WORKSFORME
Reporter | ||
Comment 16•15 years ago
|
||
Are you sure that there is no other 3-byte-character in the response? If this is the case, the encoding is correct. Could you try a single german umlaut as ä or ü or ö?
Reporter | ||
Comment 17•15 years ago
|
||
Did you have a look at my TCP dump which clearly shows the truncation? Could you attach a dump of your test?
Reporter | ||
Comment 18•15 years ago
|
||
(In reply to comment #15) > There could not be any double-encoding by us, because the Serializer code that > we wrote checks utf8::is_utf8 first before encoding, so if it was already > encoded, it would not encode it again. It's the other way around: you get it from the Bugzilla database and the fix I commented out in my patch encodes it the first time. Then the response is passed to SOAP::Lite and there the second encoding happens. In previous SOAP::Lite releases this (second) encoding was missing but was added in between time. And this leads to the bug. In general, as I understand it, Bugzilla should not care about encoding issues. The underlying communication libraries such as SOAP::Lite should do it (or pass it on). So when you add bugfixes for problems in underlying libraries you should carefully regression-test for them being fixed there.
Updated•15 years ago
|
Target Milestone: Bugzilla 3.2 → ---
Assignee | ||
Updated•15 years ago
|
Summary: Truncated XML-RPC response → Truncated XML-RPC response (incorrect content-length header)
Comment 20•15 years ago
|
||
Please reopen this bug. I can clearly reproduce it and, what's more, I am able to resolve it with the attached patch. The patch seems basically to do the same than the previous proposed patch, just that the location has changed. Indeed, removing the as_string method in XMLRPC.pm seems to fix my problem.
Comment 21•15 years ago
|
||
(In reply to comment #20) > Created an attachment (id=384817) [details] > Patch, which fixes problem > > Please reopen this bug. I can clearly reproduce it and, what's more, I am able > to resolve it with the attached patch. The patch seems basically to do the same > than the previous proposed patch, just that the location has changed. Indeed, > removing the as_string method in XMLRPC.pm seems to fix my problem. Forgot to note: My patch is against Bugzilla 3.3.4.
Comment 22•15 years ago
|
||
Max, regarding your comment in bug 499898: I am ready to pass the contents of my current database (just about 261K) to you privately. (Customer data, I cannot tell how sensitive it is) I can also pass the complete client code, which is nothing but the test suite of a proposed Perl client for the Bugzilla webservices API.
Assignee | ||
Comment 23•15 years ago
|
||
Hey Jochen. Okay, could you send everything to my email address?
Status: RESOLVED → UNCONFIRMED
Resolution: WORKSFORME → ---
Assignee | ||
Comment 24•15 years ago
|
||
Okay, I can reproduce this now, and it's definitely a problem. Anything containing non-ASCII characters doesn't work.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Flags: blocking3.4.2+
Target Milestone: --- → Bugzilla 3.4
Assignee | ||
Comment 25•15 years ago
|
||
Okay, here we go. It should be OK to require 0.710.06--it was released in 2008.
Assignee: webservice → mkanat
Attachment #370398 -
Attachment is obsolete: true
Attachment #376266 -
Attachment is obsolete: true
Attachment #384817 -
Attachment is obsolete: true
Status: NEW → ASSIGNED
Attachment #397988 -
Flags: review?(LpSolit)
Assignee | ||
Comment 26•15 years ago
|
||
And here's the same patch for the branch.
Attachment #397989 -
Flags: review?(LpSolit)
Updated•15 years ago
|
Attachment #397988 -
Flags: review?(LpSolit) → review+
Comment 27•15 years ago
|
||
Comment on attachment 397988 [details] [diff] [review] v1 >Index: Bugzilla/Install/Requirements.pm >+ # 0.710.04 is required for correct UTF-8 handling Nit: It's 0.710.03 or better, as far as I know. Otherwise, this indeed fixes the problem. r=LpSolit
Updated•15 years ago
|
Attachment #397989 -
Flags: review?(LpSolit) → review+
Updated•15 years ago
|
Flags: approval3.4+
Flags: approval+
Assignee | ||
Comment 28•15 years ago
|
||
(In reply to comment #27) > Nit: It's 0.710.03 or better, as far as I know. I got 0.710.04 from comment 14, which has info from the SOAP::Lite maintainer.
Assignee | ||
Comment 29•15 years ago
|
||
tip: Checking in Bugzilla/Install/Requirements.pm; /cvsroot/mozilla/webtools/bugzilla/Bugzilla/Install/Requirements.pm,v <-- Requirements.pm new revision: 1.69; previous revision: 1.68 done Checking in Bugzilla/WebService/Server/XMLRPC.pm; /cvsroot/mozilla/webtools/bugzilla/Bugzilla/WebService/Server/XMLRPC.pm,v <-- XMLRPC.pm new revision: 1.6; previous revision: 1.5 done 3.4: Checking in Bugzilla/Install/Requirements.pm; /cvsroot/mozilla/webtools/bugzilla/Bugzilla/Install/Requirements.pm,v <-- Requirements.pm new revision: 1.60.2.5; previous revision: 1.60.2.4 done Checking in Bugzilla/WebService/Server/XMLRPC.pm; /cvsroot/mozilla/webtools/bugzilla/Bugzilla/WebService/Server/XMLRPC.pm,v <-- XMLRPC.pm new revision: 1.1.2.3; previous revision: 1.1.2.2 done
Status: ASSIGNED → RESOLVED
Closed: 15 years ago → 15 years ago
Resolution: --- → FIXED
Comment hidden (spam) |
Comment hidden (spam) |
Comment hidden (spam) |
Comment hidden (spam) |
Comment hidden (spam) |
Comment 35•12 years ago
|
||
Just dor a suggestion: It might make sense to always set content-length=-1. That way the client wouldn't expect it to terminate at a given point.
Comment 36•12 years ago
|
||
Hi, I think this bug should be reopened. I am having still this bug in version 4.2.1. Bug #706276 should be marked as duplicated with this one. As a workaround, I have removed the "Content-Length" header in the XMLRPC.pm file, and server automatically uses Chunked transfer encoding. The patch for version 4.2.1: --- bugzilla-4.2.1/Bugzilla/WebService/Server/XMLRPC.pm.orig 2012-04-18 22:32:47.000000000 +0200 +++ bugzilla-4.2.1/Bugzilla/WebService/Server/XMLRPC.pm 2012-06-08 10:57:50.827070643 +0200 @@ -64,6 +64,7 @@ foreach (@{Bugzilla->cgi->{'Bugzilla_cookie_list'}}) { $self->response->headers->push_header('Set-Cookie', $_); } + $self->response->headers->remove_header("Content-Length"); } sub handle_login { This patch is also correct for the last XMLRPC.pm file in trunk (version 8259), except for the numbers of lines (the new version has less license lines in the beginning of the file).
Comment 38•10 years ago
|
||
After upgrading SOAP::Lite to 1.08 and XMLRPC::Lite 0.717 on Windows I can confirm that this still exists in Bugzilla 4.2.7 and 4.4.1. It does seem to be fixed in Bugzilla 4.2.6 with SOAP::Lite 1.06 on Ubuntu 12.04.
Comment 39•10 years ago
|
||
Hi, I also think this bug should be reopened. I still get this bug in Bugzilla version 4.4.2 with SOAP-Lite 1.10, XMLRPC-Lite 0.717 on Windows The workaround in Comment 36 did the trick for me.
Comment 40•10 years ago
|
||
We are not going to reopen this bug. The patch has been checked in a long time ago. File a separate bug if you have exact steps to reproduce the issue.
Comment 41•10 years ago
|
||
Wayne, Thomas: We'll definitely need more input. See, for example, the attachment from comment #1. It is unlikely, that we'll be able to reproduce this bug, but input like this should be able to provide a starting point.
You need to log in
before you can comment on or make changes to this bug.
Description
•